I took care of a couple long standing issues this week and I wanted to quickly talk about them.

Tracking down Ruby/Rails memory leaks

Memory leaks are no fun. We sprouted a big leak about a month ago: Ruby VMs would get all bloated, often hitting 300-400 MB and sometimes consuming even more ridiculous amounts of memory.

I run about 20 instances on each virtual server and a bunch of misbehaving instances could quickly eat up all of the memory. Because restarting my mongrels (Thins, I mean) when they consume too much memory is pretty painless, I put off fixing it for a while.

This week, I tried using bleak_house to figure out what sorts of objects were responsible for my VM bloat. I found nothing.

Then I learned something really important about Ruby’s garbage collection and heap maintenance. I don’t know how I missed out on learning this earlier. I’m going to quote a bit from hacking on ruby’s garbage collector at lloydforge.org since he explains it well.

• First important thing: “When ruby runs out of heap space, it first does a GC run to try to free something up, and then allocates a new heap. the new heap is 1.8 times larger than the last.”
• Second important thing: “Because of the way ruby works, objects may _never_ be moved around in heaps. That means from the time they’re allocated to the time they’re freed they may not be moved to a new memory address”

You see where I’m going with this? Force ruby to build bigger and bigger heaps and you’ll probably never get that memory back. A single non-garbage object sitting on one of this heaps will keep it alive prevent you from getting that memory back. There is no heap compaction.

I read about some of the work that the Phusion guys had done on their ruby fork called “Ruby Enterprise Edition” and noticed that they had added a handy debugging tool: ObjectSpace.statistics.

I installed Ruby EE, added some ObjectSpace.statistics logging to each request and ran it on one server for a half-day. Once I had my results, it only took a few minutes to spot the problem requests and fix them. Phluid’s Ruby EE prints out the number of objects, size of objects, number of heaps, and size of heaps so you can spot when a request is causing the creation of a new (large) heap. The culprit was just a stupid coding error in a rarely-hit action that loaded way too much data via Activerecord. The heap would balloon to a gigantic size and although the objects loaded would all become garbage, something prevented the memory from being given back.

This was a heap-related problem but I think that these ObjectSpace statistics would be equally valuable for finding the reference-related leaks. You probably wouldn’t be able to eyeball the data like I did but you could add up the growth in # of objects and bytes across a large number of requests and come up with a few suspicious actions. The really nice thing is that I was able to set up this logging in production without impacting the site in any way.

Tracking down MySQL performance problems

If you are running MySQL, you should consider using one of the Percona patched versions: http://www.mysqlperformanceblog.com/2008/07/16/mysql-releases-with-percona-patches/

The Google SHOW INDEX_STATISTICS patch is great - you can use it as part of your index tuning and pruning because it will show you which indexes are being hit.

The microslow patch turned out to be an extremely valuable tool for finding problem queries. We have far too much query traffic to log all of it and MySQLs “slow query” log only works as whole second precision. In our case, we had some slow queries that were beating up the IO subsystem and they weren’t easy to spot until brought up the patched MySQL and set the threshold to 300 milliseconds (instead of 1 second).

I ended up breaking up a very large table into an InnoDB table with all of the important information and indexes and a MyISAM table containing the blobs (text columns). I changed a few lines of code on the application side to make things work with the new split table, and voila:

okay. I’ve run out of steam…

In other news - we hit a milestone of sorts by nearly making the Alexa 10,000 this month. We’re at 10,032