Well, I installed our new database server last night and so far things are looking good. The new box has way faster disks (4 SAS disks that are RAID10′d) and 32 GB of memory. You can’t scale up a database forever, but now I’ll actually be able to see through the smoke and tune my app more easily. We just didn’t have enough memory before and the database spent tons of time waiting on the disks as data was moved in and out of the InnoDB buffers. Now we’ve got 1 good database machine, so the next architectural upgrade might be moving the read-only slave to another good machine. (pssst - great scaling presentation over here)
There was a lot of shuffling involved in the move. I set up a brand new VM, dumped/restored the database, moved 2 application virtual machines from our smallest box to the space previously occupied by the database, moved the slave database and search engines (Ferret, Sphinx) to the small box, and then redistributed all of the memory and started it all back up. Have I mentioned that I love Xen? 
A few tools that made my life much easier:
- Maatkit - the parallel dump and restore tools are very fast and it is easy to monitor progress. The majority of the downtime during the move was going to be during backup/restore and Maatkit saved us precious minutes.
- dd over netcat - an easy low tech way to move my machine images between physical servers. I also made use of ReiserFS’s filesystem resize tools so that my disk images were as small as possible. Google for “dd netcat“
- mysqlbinlog - because I forgot to check to see where I wanted the slave database to start replicating from.
- …and thanks to Google and a blogger named Bruno for helping me get my Xen dom0 memory back without rebooting.
Also, I’m now running MySQL 5.1 (the newest release candidate) and the new InnoDB plugin. The plugin replaces the version of InnoDB that ships with 5.1 and offers additional features such as online index updates (woohoo!).
A huge thank you to all of our users - this machine was 100% paid for with donations. Thank you so much everyone! (..and also thanks for being patient during the last 4 weeks or so)
Comments (7)
Glad everything went well! And I love Ravelry. Thanks to you, Jess, and Bob!
Congrats on the move - awesome seeing all the details of how it happened.
I still find it amazing that you have the interest and appreciation for infrastructure and code. Well done on the move.
How are you doing the graphical dump of vmstat?
Hair in the new server!
I was wondering if you ever considered Postgresql an option back when you were starting Ravelry. Were there any particular reasons you chose MySQL?
Not proselytizing, just curious. I’ve actually never even used Postgresql for anything before, but I’m using it for a project of my own that I’m just starting, and want to know if there were any specific things that led you away from it. Development, scaling, etc…
Thanks!
By the way, my wife has been on Ravelry since almost the beginning, and got me to sign up a month or two ago. I’m amazed at how you’ve single-handedly managed to get so much done in such a short time. Congratulations on everything!
I’m having problems connecting to the Ravelry site. Is there anywhere that gives updates of server status?
I’ve been getting the 503 server unavailable message for about 20 minutes.
Trackback/Pingback (1)
[…] and 32 GB of memory. You can??t scale up a database forever, but now I??ll actually be able to shttp://codemonkey.ravelry.com/2008/04/21/new-database-server/Sun Microsystems Announces MySQL 5.1 BIOSThe new version of the world’s most popular open source […]