It's all about timing baby!

Image: Suat Eman / FreeDigitalPhotos.net
Yeah, it's about timing.

There was a problem in one of our projects at moove-it related to slow processing. There is a daemon spawning new threads to process certain new entries to the database. The entries come from a different system, that's the reason for this program which processes each new entry. Sometimes at certain hours of a day there are peaks in the entries to the database and the process will fall behind by about 20.000 entries or more. So we started analyzing the code to understand what was happening and why it took so long. We noted that each new thread the daemon spawned took about 5 seconds to complete its task. As we narrowed the measurement we came up with some code that took 5 seconds to execute but it only involved access to the database. So thanks to Aaron Patterson's (@tenderlove) talk at RubyConf Uruguay about "Who makes the best asado" where he talked about rails and how it manages threads and database connections, we knew where to look.

What he explained is that each new thread requests its own database connection from the connection pool, and if there isn't a free connection, rails waits for about 5 seconds and if after that there is no free connection it iterates over all the threads to take back the connections of dead threads(more info). See the correlation with the 5 seconds I talked about in the previous paragraph? We immediately suspected that this was the problem. So we started searching the Rails API for a way to release the connection at the end of each thread's execution. Surprisingly, we didn't find an easy and understandable explanation anywhere at first googling ;), so we digged deeper and came up with the following line:
 ActiveRecord::Base.connection_handler.clear_active_connections!
The  ActiveRecord::Base.connection_handler method returns the connection handler for the current thread and the clear_active_connections method does what it looks like, or from the Rails doc: "Returns any connections in use by the current thread back to the pool, and also returns connections to the pool cached by threads that are no longer alive."

So this line returns the connections in use by a thread to the pool and enables the new threads spawned by the daemon to use the freed connections. This way we avoid the 5 second wait for rails to free the connections for us.

This one line picked up our performance from processing 1.000 entries in almost 2 hours to processing 10.000 in 5 minutes. Nice huh?!

That's it. I'm not sure if this is the best way of doing it since this method also "... returns connections to the pool cached by threads that are no longer alive." I guess this means it does the iteration over all the threads Aaron mentioned, but as you can see I'm happy with the performance improvement. We are using Rails 3.0.5, Aaron said that he will change the behavior, read more about it here.

Special thanks to @cheloeloelo who helped detecting the problems and digging through the Rails API finding the proper method to free the connections.

Popular posts from this blog

Export to Excel with spreadsheet gem

While using the spreadsheet gem to export data to a Microsoft Excel file I encountered various problems and lack of proper documentation forced me to dig pretty deep. So here are a few tips you might find useful. The gem works with a workbook object and worksheets in this workbook. To get started: workbook = Spreadsheet::Workbook.new sheet1 = workbook.create_worksheet name: worksheetname To write a row of data as usually is done in reports you can use: sheet1.row(1).replace [ 'col1', 'col2', 'col3'] Rows and Columns are 0 based, so col1 would be printed on A2 (Excel coords). Now to the beauty of it all, formatting, first of all, create a format: bold = Spreadsheet::Format.new :weight => :bold bold20 = Spreadsheet::Format.new :weight => :bold, size: 20 nobold20 = Spreadsheet::Format.new size: 20 bold_grey_bg = Spreadsheet::Format.new pattern_bg_color: :cyan, weight: :bold, pattern: 1, pattern_fg_color: :black Use :weight to set bold,
Read more

Caleidoscopio de dos naciones

Países hay muchos Pero prefiero a dos Copas hay muchas Cuatro, tienen los dos Norte y Sur Disciplina  y Corazón A veces glamour A veces pasión Blanco y Negro por el pasado Por la gloria es el dorado De Azul el cielo pintado Rojo pasión! O estoy errado? Así mi corazón está pintado Muchos creen que dividido Pero no esta separado Es como metal fundido
Read more

Keep sidekiq running using monit

Image
As an application grows it starts taking care of more and more work. At some point the first task comes that needs to be processed in background and one of the usual suspects for ruby is sidekiq. Now, did it ever happen to you that you get it all up and running on the server and next thing you know it's down and you get error reports that stuff ain't working? Meet monit, a tool that'll let you get ahead of trouble. It automatically restarts programs when down and can even send emails upon failures. Install monit Depending on your package manager this may differ: apt-get install monit yum install monit brew install monit Configure monit Edit  /etc/monit.conf or  /etc/monitrc Uncomment(or add) these lines: set httpd port 2812 and use address localhost allow localhost Add sidekiq entry Create the file /etc/monit.d/sidekiq.monitrc with: check process sidekiq_application_production0 with pidfile /path/to/shared/tmp/pids/side
Read more