Keep sidekiq running using monit

As an application grows it starts taking care of more and more work. At some point the first task comes that needs to be processed in background and one of the usual suspects for ruby is sidekiq.

Now, did it ever happen to you that you get it all up and running on the server and next thing you know it's down and you get error reports that stuff ain't working?

Meet monit, a tool that'll let you get ahead of trouble. It automatically restarts programs when down and can even send emails upon failures.

Install monit

Depending on your package manager this may differ:

apt-get install monit
yum install monit
brew install monit

Configure monit

Edit /etc/monit.conf or /etc/monitrc

Uncomment(or add) these lines:

set httpd port 2812 and
    use address localhost
    allow localhost

Add sidekiq entry

Create the file /etc/monit.d/sidekiq.monitrc with:

check process sidekiq_application_production0 with pidfile /path/to/shared/tmp/pids/sidekiq.pid 
  start program = "/bin/bash -c 'cd /path/to/app/current && source /home/user/.rvm/environments/ruby-2.0.0-p353@global && bundle exec sidekiq -e production -C /path/to/app/config/sidekiq.yml -P tmp/pids/sidekiq.pid -L log/sidekiq.log'"
  stop program = "/bin/bash -c 'kill -s INT `cat /path/to/shared/tmp/pids/sidekiq.pid`'" with timeout 90 seconds
  if totalmem is greater than 500 MB for 2 cycles then restart # eating up memory?
  group sidekiq

Let's not forget to start monit: /etc/init.d/monit start

Now a sudo monit status should show the sidekiq_application_production0 entry.

If you use capistrano you will notice the current and shared directories. If you don't use capistrano just make sure the pid is stored somewhere where the path doesn't change so monit can't find it anymore.

This configuration will also be useful if you use capistrano-sidekiq, just make sure the check process name in the config matches what is used during deploy.
This configuration includes a memory protection that will restart sidekiq before it eats up all the memory in your server. Yes, I've got a server down more than once due to this :(

Configure email server

Now, interested in receiving emails with monit's status? Easy, just add the following to /etc/monit.conf or /etc/monitrc
Add email server, e.g.: gmail (add this line)

set mailserver smtp.gmail.com port 587 username "notifier@gmail.com" password "********" using tlsv1 with timeout 30 seconds

Subscribe

Subscribe to absolutely all emails (includes non-error emails) (add this line)

set alert my_app_notifier@gmail.com

If everything is set up correctly, executing sudo monit reload should trigger an email.

If you want to get emails for specific events or have more questions regarding monit config check out http://mmonit.com/monit/documentation/monit.html

It's a huge page, just search for email, alert, whatever you need.

That's pretty much it. Now of course this doesn't take care of starting the processes if the server restarts. You'll have to look for another tool to do that.