Resque. Two things to think about.
We started using the redis-backed Resque for background jobs. So far it’s been pretty great to work with. The syntax is clear and the whole thing just works. There are very good articles describing how to actually code Resque. Here’s one. I won’t bother with yet-another-walk-through.
However, I do want to talk about one gotcha and one thing you should think about.
The Gotcha: Possible Message Failure
Messages in Resque are not necessarily going to run. There is an, admittedly edge, case where a worker can pop a message off of a queue and then die. The message never gets put in a fail queue and it never actually runs. The message is gone. I have a fork that fixes this (makes it so messages can’t be lost unless Redis loses ‘em) https://github.com/tobowers/resque. However, that’s awaiting a merge.
Until my branch (or some other fix) gets merged, make sure you think about what’s going to happen if your job doesn’t run. For many apps this can be “ok” (since it won’t happen very often). Just make sure it’s OK for yours.
One Thing You Should Think About: Tracking Message State
Since you can’t reach into a Resque queue and say “talk to me about this job,” tracking message state is up to your application. We happen to do this in our database.
We put a “job” record into the database with body, result, state columns. The body is there so we can always re-create a message from the database if something goes wrong. The result keeps track of what happened and the state lets us track it along its merry way.
If you have to make absolutely sure that your messages get processed, you’ll have to think about how you’re using Resque.