Backgrounding tasks via Capistrano... or, why I hate shells
Posted by Andrew
Say you want to run a rake task in the background after a capistrano deploy. You'd think it'd be something like
run "rake blah &"
You would be wrong. Your capistrano task would never finish (if your rake task was supposed to just sit there and wait for something). If you do thought back to your Operating Systems class you'd remember something about standard input, standard output and standard error and you'd think, oh yeah let's just close those.
run "rake blah 2>&1 >/dev/null < /dev/null &"
You'd still be wrong because now even though it finishes, nothing's actually running on the server. What to do. Well one of the most horrible parts of dealing with UNIX is dealing with signals. Signals are a really primitive method of doing interprocess communication by sending a PID a notification of some predefined type. Processes can even register to listen to those signals and take appropriate Action.
For example. Say you wanted to make a server that you could tell to reload its configuration without having to bring down the server. The most common way of doing this is sending the server a HUP (Hang-up) signal. Of course it's obvious which program you'd use to send signals, why, it's called "kill" (that's sarcasm for anyone paying attention). Ok fine, so you can signal your process by figuring out its PID and giving it a signal
kill -HUP 12345
In the process a signal handler would fire and you could Do Things. Of course there's some signals which you can't respond to like -KILL
kill -KILL 12345
kills process 12345 immediately without giving that process the ability to Do Things. This is a useful safety feature in case your handler gets stuck. Then there's ABRT which usually means your process got a CONTROL-C. I could go on, but I'm getting bored.
So apparently the default behaviour for when you disconnect from a session is a HUP (Hang up) which for most processes means death. So what you need to do is tell your rake task to ignore HUP signals. An easy way to do this is to launch rake using the aptly named (for real this time) nohup program which will dutifully prevent hangup signals from reaching your beloved rake process. And VOILLA:
run "nohup rake blah 2>&1 >/dev/null < /dev/null &"
Let's all pause and recognize that there are more characters telling the shell to run something in the background than there are characters for invoking the actual thing running. Now, I get it, UNIX has to be robust and so on, but really? This seems like a fairly common occurrence and while there are deamonizing toolkits out there, I really don't want to have to deal with someone's toolkit in order to add these simple options around a task that I already know how to run.
What's the better solution? Well in my not-so-humble opinion I think that this kind of stuff should be part of POSIX (not LSB) and I should just be able to run
deamonize rake blah
On any system and it does the above. Maybe have an option that says redirect output to somewhere else. There's something similar in /etc/init.d/functions on SYSV systems but it's inconsistent between systems and is about 200 lines of terse BASH script to do something fairly simple.
The next step, of course, is being able to kill the long-running process from capistrano as well. Sure, you could figure out how fork works, and then do that little pattern of "AM I THE CHILD PROCESS OR THE PARENT PROCESS" which is oh-so-much-fun.
Or you could just do this:
In whatever task you're running spit out the Process.pid to a file somewhere. When you want to kill the long-running process just read the file and send it an ABRT. That easy. If you're invoking a command in the background using an ampersand (&) you can get the PID of the child process you just created by reading the $! (the ! means "wow! this is obvious!") environment variable.
nohup something 2>&1 >/dev/null < /dev/null &
echo $! > path/to/pid
And to kill it do
kill $(cat path/to/pid)
Again, no Toolkit Required. Going back to my little deamonize utility it'd be a nice feature to optionally tell it where you want the pid file to go.
daemonize -p path/to/pid -o path/to/log something
And that's the news from EberTech. Join us next time when we discuss something less irritating.