Finding Nothing Killed The Server In Germany
A safety trap I wrote to kill hung processes looked up a process by name, got nothing back, and killed process 0 instead. The server in Germany went down close to midnight on a Friday, Australian time, just after everyone in the German office had left early for the weekend.
One of my biggest mistakes happened many years ago, on a night when we were all staying back to work late.
We had pizza. We were locked in. It was late, close to midnight in Australia, when the server in Germany went down.
Nobody Left To Answer The Phone
At first we waited, assuming it would come back. It did not.
We tried to get access and got nothing. No response at all.
So we tried calling people in the German office. By then it was just after 3pm there. It was Friday, and everyone had left at 3pm to go home early.
There was nothing more we could do from where we were. The server stayed down for the whole weekend.
Monday, And A Reboot
On Monday we finally got hold of someone in the German office. They went and looked at the machine for us.
The screen did not show much. They gave it a reboot and it came back up.
That fixed the outage, but it did not tell us anything. Now it was a case of working out what had actually gone wrong, and the only place left to look was the logs.
Something Killed It
Going through the logs, it looked like a process had been killed. The question was why, and by what.
Then I found it. There was a bug in my code.
The thing that took the server down was a safety feature I had written.
How Dazuko Fit In
The server ran BSD, and we used a kernel module called Dazuko to detect file changes. Dazuko sits in the kernel and watches file access, so a program can be told whenever a file is touched.
When it detected changes, our system would fire off a number of scripts.
Scripts can hang, and scripts can run for far longer than they should. So the process had a safety trap built in to kill any of them that hung or ran too long. That was the part I wrote, and that was where the bug was.
Finding Nothing
The safety trap worked by looking for the process by name, getting its ID, then killing that ID.
That works fine when the process is there. The bug was what happened when it was not.
If you looked for the process name and it found nothing, the lookup returned 0. The code did not check for that. It took the 0 as an ID and passed it straight to kill.
Process 0 was not nothing. On BSD, the bottom of the process table is the system itself, with init, the core process that everything else comes from, right alongside it.
So it would find nothing, then kill the system.
Safety Features Are Code Too
The whole point of that trap was to stop runaway scripts from hurting the server. Instead it was the most dangerous thing running on it, because nobody looks at a safety feature as a thing that can break.
These days I test the code that handles failures harder than the code that handles success. That is the code that runs when things have already gone wrong.