Well, I installed our new database server last night and so far things are looking good. The new box has way faster disks (4 SAS disks that are RAID10′d) and 32 GB of memory. You can’t scale up a database forever, but now I’ll actually be able to see through the smoke and tune my app more easily. We just didn’t have enough memory before and the database spent tons of time waiting on the disks as data was moved in and out of the InnoDB buffers. Now we’ve got 1 good database machine, so the next architectural upgrade might be moving the read-only slave to another good machine. (pssst – great scaling presentation over here)
There was a lot of shuffling involved in the move. I set up a brand new VM, dumped/restored the database, moved 2 application virtual machines from our smallest box to the space previously occupied by the database, moved the slave database and search engines (Ferret, Sphinx) to the small box, and then redistributed all of the memory and started it all back up. Have I mentioned that I love Xen?
A few tools that made my life much easier:
- Maatkit – the parallel dump and restore tools are very fast and it is easy to monitor progress. The majority of the downtime during the move was going to be during backup/restore and Maatkit saved us precious minutes.
- dd over netcat – an easy low tech way to move my machine images between physical servers. I also made use of ReiserFS’s filesystem resize tools so that my disk images were as small as possible. Google for “dd netcat“
- mysqlbinlog – because I forgot to check to see where I wanted the slave database to start replicating from.
- …and thanks to Google and a blogger named Bruno for helping me get my Xen dom0 memory back without rebooting.
Also, I’m now running MySQL 5.1 (the newest release candidate) and the new InnoDB plugin. The plugin replaces the version of InnoDB that ships with 5.1 and offers additional features such as online index updates (woohoo!).
A huge thank you to all of our users – this machine was 100% paid for with donations. Thank you so much everyone! (..and also thanks for being patient during the last 4 weeks or so)