At Flowdock we’re using Resque to process a bunch of background jobs: Campfire imports, analytics, payments, digest emails and others.

It works great, but I realized it’s not trivial to configure this combination to

• work with Bundler
• be reliable when servers are in inconsistent state
• restart gracefully without interrupting running jobs

So here’s how we’re doing it!

Configuring god

This is actually the part where existing documentation is correct. We have a fairly simple config/app.god, that just loads all the other files:

path = File.expand_path(File.join(File.dirname(__FILE__), 'god'))
God.load "#{path}/*.god"

Check out resque.god and resque_scheduler.god for the full configuration files, but here’s the beef:

God.watch do |w|
  w.dir      = "#{rails_root}"
  w.name     = "resque-#{resque_id}"
  w.group    = 'resque'
  w.interval = 30.seconds
  w.env      = {"QUEUES"=>resque_queues, "RAILS_ENV"=>rails_env, "BUNDLE_GEMFILE"=>"#{rails_root}/Gemfile"}
  w.start    = "bundle exec rake -f #{rails_root}/Rakefile environment resque:work"
  w.log      = "#{rails_root}/log/resque-#{resque_id}.log"
  …
end

Capistrano uses a different environment than your login shell, so it’s a good idea to define Bundler configuration here.

Configuring Capistrano

When deploying our app, we want to restart workers gracefully: they should be able to finish their on-going tasks first. Also, it needs to work whether god is up or not.

Here’s the configuration (to be included in config/deploy.rb) that I’ve found most reliable:

def try_killing_resque_workers
  run "pkill -3 -f resque"
rescue
  nil
end

desc "Restart God gracefully"
task "restart_god", :roles => :app do
  god_config_path = File.join(release_path, 'config', 'app.god')
  begin
    # Throws an exception if god is not running.
    run "cd #{release_path}; bundle exec god status && RAILS_ENV=#{rails_env} RAILS_ROOT=#{release_path} bundle exec god load #{god_config_path} && bundle exec god start resque"

    # Kill resque processes and have god restart them with the newly loaded config.
    try_killing_resque_workers
  rescue => ex
    # god is dead, workers should be as well, but who knows.
    try_killing_resque_workers

    # Start god.
    run "cd #{release_path}; RAILS_ENV=#{rails_env} bundle exec god -c #{god_config_path}"
  end
end

after :"deploy:restart", :"deploy:restart_god"

And that’s all folks!