New Database Server

Monday, April 21st, 2008

Well, I installed our new database server last night and so far things are looking good. The new box has way faster disks (4 SAS disks that are RAID10′d) and 32 GB of memory. You can’t scale up a database forever, but now I’ll actually be able to see through the smoke and tune my app more easily. We just didn’t have enough memory before and the database spent tons of time waiting on the disks as data was moved in and out of the InnoDB buffers. Now we’ve got 1 good database machine, so the next architectural upgrade might be moving the read-only slave to another good machine. (pssst - great scaling presentation over here)

There was a lot of shuffling involved in the move. I set up a brand new VM, dumped/restored the database, moved 2 application virtual machines from our smallest box to the space previously occupied by the database, moved the slave database and search engines (Ferret, Sphinx) to the small box, and then redistributed all of the memory and started it all back up. Have I mentioned that I love Xen? :)

A few tools that made my life much easier:

  • Maatkit - the parallel dump and restore tools are very fast and it is easy to monitor progress. The majority of the downtime during the move was going to be during backup/restore and Maatkit saved us precious minutes.
  • dd over netcat - an easy low tech way to move my machine images between physical servers. I also made use of ReiserFS’s filesystem resize tools so that my disk images were as small as possible. Google for “dd netcat“
  • mysqlbinlog - because I forgot to check to see where I wanted the slave database to start replicating from.
  • …and thanks to Google and a blogger named Bruno for helping me get my Xen dom0 memory back without rebooting.

Also, I’m now running MySQL 5.1 (the newest release candidate) and the new InnoDB plugin. The plugin replaces the version of InnoDB that ships with 5.1 and offers additional features such as online index updates (woohoo!).

A huge thank you to all of our users - this machine was 100% paid for with donations. Thank you so much everyone! (..and also thanks for being patient during the last 4 weeks or so)

Filed in Database, Hardware | Tagged , , , | Permalink | Comments (8)

MySQL 5.1 (finally?)

Saturday, April 12th, 2008

After years of development, it appears that MySQL 5.1 is going to be released at the MySQL conference next week.

The timing is perfect. Our new database server is arriving on Monday and there hasn’t been a release candidate release since January.

5.1 is supposed to be faster. I’m sure that the web will be awash with benchmarks but I’m still going to run some sysbenches on the new machine just for fun. I’ll post whatever I come up with. In addition to performance improvements and bug fixes, MySQL 5.1 has some new features (like partitioning). There is a short “What’s New?” page over at mysql.com: http://dev.mysql.com/doc/refman/5.1/en/mysql-nutshell.html.

update Doh. Looks like no GA release until June. At least there is a new release candidate.

Filed in Database | Tagged | Permalink | Comments (0)

Keeping up

Sunday, March 30th, 2008

Reading blogs (and forums, groups, etc) that are related to my work helps me stay interested, excited, and learning. I knew that it was time to leave Java-land when I stopped caring about reading Java-related blogs.

Here are my current favorites. I hope that some of you will find something useful in here. Maybe a good link, maybe an idea…

(After I started writing this I realized that my favorites folder is actually pretty thin. I’m subscribed to a zillion things but only a handful have made the hop from “tech” to “tech favorites”)

Things that aren’t listed here…

For most of the open source software that I really care about, I subscribe to the personal blogs of the creators, Trac feeds, Google groups, whatever. I’m using the latest code from many different projects and I like to know about new happenings, new ideas, all that.

Del.icio.us RSS

One great way to keep up with a specific topic is by subscribing to the del.icio.us/popular feeds for the tags that you are interested in. Del.icio.us is a social bookmarking service, which means that the “popular” links are things that are bookmarked by large numbers of people and the quality is usually pretty good.

Here are a few examples: ruby, rails, ui.

The feeds for things like “css”, “javascript”, etc sometimes turn up good links but they are pretty noisy. I find that most of the really good stuff that hits sites like Digg, Slashdot, etc will find their way to delicious’ popular page. I also often go to del.icio.us before I go to Google if I am researching technical topics.

Ruby or Rails centric

Like minds

  • I read LibraryThing’s main blog and Thingology blog. LibraryThing is the only web company (that I know of) that seems really similar to Ravelry in spirit and in execution. I find their blog to be really interesting (and sometimes, eerily familiar)

Web stuff

  • Ajaxian - high volume, but most of the interesting Javascript happenings hit their pages.
  • Douglas Crockford
  • High Scalability - jury’s still out on this one, but there have been a few really interesting posts

MySQL

General geeky stuff

  • Jeff Atwood’s Coding Horror - you’re gonna hear about his posts anyway so you might as well read them ;) Give it a try - I used to look forward to new posts but now I feel like he is swinging for the fences every time.
  • xkcd - you know, the stick figure guy

On a kinda related note - I’m not going to be able to make it to RubyFringe. Damn. Anyway - it’s a not-your-average Ruby conference in Toronto. Check out the list of speakers..

Filed in Blah | Tagged , | Permalink | Comments (4)

Load Balancing

Monday, March 10th, 2008

oops

I’ll get the embarrassing part out of the way first. So… Ravelry had over an hour’s worth of hiccups and short periods of downtime yesterday. The cause of the problem was just a stupid full disk - I missed rotating a log, I was too lazy to set up Nagios monitoring for all my disks, and I didn’t notice the steadily climbing graph. However, I changed a whole bunch of big stuff just before the the problem hit and ended up wasting time looking in all of the wrong places before this caught my eye:

I just wanted you to know that the new stuff that I’m about to gush about wasn’t the cause.

6 million Rails requests per day

Ravelry does just over 2 million page views each day. Once you add in all of the AJAX hits, RSS feeds, API calls, and a few other things it adds up to 6 million requests that actually hit the Rails app. (I just grepped one day’s master syslog for the “Completed in” lines)

We currently have 70 Thin instances that run the Rails application (previously using Mongrel - keep reading). Each can handle a single request at a time and I guess you can call them “app servers” for lack of a better term. Requests come in to the web server, things that aren’t static files like images are passed along to Rails, the response goes back to the client.

Our biggest challenges at this layer have been 1) making sure that the perceived speed of the site isn’t affected if one or some of the instances go crazy or go down and 2) deploying new versions of the site without interruption.

HTTP server to Rails - the progression

We are now on our 3rd real “version” of our setup when it comes to the front end web server and its connection to the Rails instances on the backend. There is a lot of people thinking (and coding) about better ways to run and deploy Rails apps and I’m sure that we’ll continue to change how we do things.

Version 0: Apache + FastCGI

Used this combo in an early development version. Kind of a crappy setup, but it was all I had since I was still on a shared host. If you’ve tried playing with Ruby on a shared host with this setup and you hate it, don’t worry. There is another way.

Version 1: Apache → mod_proxy_balancer → Mongrel

This is probably the simplest setup because most people already have a standard Apache installation. Start up your Mongrels (however many you want), configure your Apache, and Apache proxies the requests to your Mongrels using a simple balancing algorithm.

Our biggest problems? Both hot deploys and recovering problem Mongrels was made difficult by mod_proxy_balancer’s behavior. Restarting part or all of the cluster often resulted in Apache returning 500 errors anytime the affected servers were hit, even once they were back and up running properly. I often had to reload or restart Apache (which was a slow process when Apache was getting hit with lots of traffic) to bring things back to life.

Version 2 - nginx → Mongrel

I originally set up an Apache alternative because we were moving (finally) from a single server to our 3 machine, multiple virtual machine setup. I wanted to try nginx and see what all of the fuss was about. nginx uses less CPU than Apache, *far* less memory, and is simpler to configure. nginx is great - I highly recommend it. The biggest benefits that we got from this setup were more free memory, lighter load, and no more 500 errors.

I used Evented Mongrel for part of this time, due to the promise of extra stability under load.

Version 3: nginx + upstream_fair → Mongrel

We moved to the fair proxy balancer for nginx shortly after it came out - it was a great improvement. The fair module solves a major problem with connecting Mongrels to nginx the usual way. Because a Rails app running on Mongrel can only handle one request at a time, nginx’s round robin sharing of the load would often lead to requests waiting in line for a busy Mongrel when there are other free instances that could be used. Switching to this setup greatly improved the perceived speed because users were much less likely to be affected by a slow running requests initiated by others. If it weren’t for this module, I would have had to move some API calls, some searches, and administrative functions to background tasks or alternate clusters. It came out at just the right time :)

If you are starting something small and you aren’t dead set on using Apache, I recommend this configuration as a great and simple setup. nginx is excellent, upstream_fair makes the proxying smarter.

We ran this way for the last 4 months. I finally made some changes yesterday because my feature wish list had grown too large. There were 2 major things that I was hoping to correct. First, requests were still loaded on to the single-connection handling Rails app without regard to their current state. 2000 impatient concurrent users could easily turn a little slowness into a big pile-up by clicking madly and hitting refresh (as people tend to do if a site is unresponsive). This made it really hard to do releases without putting up a maintenance page, and since we were often doing daily releases, I *really* wanted to avoid taking the site down for maintenance all the time. Second, things would sometimes go funny with the balancing. I’d notice that an entire server’s worth of Mongrels was being ignored (by looking at my graphs) and I’d have to reload nginx to wake it up. I’m pretty sure that this only happened when I added and removed servers from the cluster during deploys but I’m not positive. In any case, the behavior was a little disconcerting.

Version 4: nginx → HAProxy (running now) → Thin

Disclaimer - I’ve only been running this for 2 days ;) and I’ll explain Thin after - the fact that I replaced Mongrel with Thin isn’t that important to my story.

HAProxy is a software load balancer. I love it. Instead of making nginx proxy requests to our set of 70 Mongrels (okay, Thins) it sends everything to HAProxy which does a much nicer job of balancing the load. HAProxy can handle HTTP and TCP traffic with some caveats - if you have *any* load balancing needs, I highly recommend that you take a peek at it.

Why I like HAProxy:

  • Understands per-server connection limits and configurable request queuing
  • Watches servers for up/down-ness (or single slow running requests, in the case of Rails/Mongrel) and routes requests appropriately
  • “abortonclose” tosses out useless aborted requests. From the HAProxy manual: “In presence of very high loads, the servers will take some time to respond… When clients will wait for more than a few seconds, they will often hit the “STOP” button on their browser, leaving a useless request in the queue, and slowing down other users, and the servers as well, because the request will eventually be served, then aborted at the first error encountered while delivering the response.”
  • Can lay off the back end and provide users with some sort of feedback (even if it is a 503 Not Available error) if things are going badly.
  • Great logging. It is so nice to be able to see (and analyze) how the balancing/proxying is going.
  • Fast and light! Low memory footprint, low CPU impact. nginx doesn’t put it to shame, which is saying something.
  • Cool and useful statistics page that shows up/down servers, session counts, request queue, server status and downtime, etc. Plenty of good stuff for my Nagios monitoring and Munin graphs.
  • This one isn’t scientific. I can do my hot deploys. By its very nature, HAProxy deals really well with a rolling restart of all servers.

Like I said, it has only been 2 days, but I am loving this setup so far. It is *so nice* to have lots of configurability and lots of visibility when it comes to the connection between my HTTP server and the application servers.

The HAProxy stats page:

nginx, HAProxy, and Mongrel/Thin

I didn’t see many (any) examples on the web, so I just want to share a few configuration snippets…

nginx config 

  upstream haproxy {
    server 127.0.0.1:8000;
  }
  
  server {
    # blah blah do the same thing that you would with the usual Mongrel setup
  }

haproxy config This should get you started - check out the haproxy manual for all you ever wanted to know. Make sure to look into the things that I bolded. 

  global
    log 127.0.0.1 local0 warning
    daemon
    # and uid, gid, daemon, maxconn...
    
  defaults
    mode            http
    retries         3
    option          abortonclose
    option          redispatch

    maxconn         4096

    timeout connect 5000
    timeout client  50000
    timeout server  50000

  frontend rails *:8000
    default_backend mongrels

  backend mongrels
    option httpchk
    balance roundrobin
    server server-1 127.0.0.1:8001 maxconn 1 check
    server server-2 127.0.0.1:8002 maxconn 1 check

Thin

Earlier, I mentioned that I switched from Mongrel to Thin. Thin is basically a drop-in replacement for Mongrel - you don’t have to do anything special (other than “gem install”) to run your rails app under Thin.

Why? The reasons why I like Thin are pretty simple. Development is very active, it is nicely packaged and includes examples, rake tasks, and scripts for controlling the service, and Rack makes things more flexible. I’m happy with the change, but I don’t think that people need to switch their Rails apps from Mongrel (yet).

Tomorrow is Monday

Since we are still adding 800ish people to the site a day and Monday is our busiest day, we break a new traffic record every Monday. We’ll see how things go on day 3 of my new setup.

Filed in Ruby on Rails | Tagged , , , , , , | Permalink | Comments (10)

Running Rails

Saturday, February 2nd, 2008

Another dry sysadmin/deployment related post. If any of you Ravelry people want to hear about something in particular, please leave a comment :)

A few quick notes about how I’m actually running my app:

  • Web server : nginx. I ran Apache for a while before I switched to nginx. Nginx uses far less resources under load and is simpler to configure and custom compile/update.
  • App server proxying/load balancing: nginx’s proxy with the addition of Grzegorz Nosek’s upstream_fair module, which sends requests to the least busy app server. This makes a huge difference because requests won’t stack up as long as you’ve got plenty of cluster nodes… I’ve been having some small problems with the module ignoring hosts that it should be proxying requests to and I’ve needed to reload the nginx configuration from from time to time. (This module is still beta)
  • App server: Mongrel 1.1.x. It’s not perfect, but what else is there? (keep reading)
  • Other stuff: I was using Swiftcore’s Evented Mongrel patch but it has been unreliable with the latest Mongrel. I’m going to try out their latest updated version when I have a minute.

What else is out there?

A couple interesting Mongrel-replacing hopefuls recently popped up:

  • Thin : Thin is the Mongrel parser + EventMachine + rack. Most of the talk and announcements happen over in the Google group. I’ve been following and tinkering with releases as they drop. Thin updates have been coming quickly and the whole effort looks promising.
  • Ebb : Ebb is similar - Mongrel parser + libev + rack. It was announced a couple weeks ago.
Filed in Ruby on Rails, Sysadmin | Tagged , , , , , | Permalink | Comments (4)

Sphinx for search

Wednesday, January 9th, 2008

In November or December, I started looking into other full text search options that would be suitable for searching forums (1.1 million posts) and projects (430,000 and lots of important associations). At the time, I was using Ferret (a port of Apache Lucene) and acts_as_ferret for all of our search needs and I wasn’t happy with how things were going with large search indexes. Indexing took a lot of time and a lot of power, unnoticed gaps would occur in the index (due to slow reindexing while live), search speed for large frequently updated indexes was relatively slow, sorting and filtering inside the search engine was not as simple as I had hoped….

I gave Sphinx a try and I was quickly hooked. Search is a very important part of our site and I think that Sphinx is going to be a great help.

the good

  • Really fast indexing.
  • Simple and sensible configuration - just hook it up to your database and write a SQL query that gathers all of the data that you want (for each document). Just write a single SQL statement that joins in all of the data that you are interested in searching on - one for each document type. Once that is done, you only have to define the columns that you want to use for filtering/sorting/grouping…
  • Fast filtering and sorting by user-defined attributes
  • Grouping - this is really handy. You can have your search results clustered by some common attribute and then sort/filter on top of that. Saves the usual waste of bringing back tons of results and then making the database do the work.
  • Pagination that works with both of the above.
  • Support for boolean, phrase, field search, proximity searches
  • Handy text related features - case folding, encoding/int’l character issues, HTML filtering, stemming
  • Runs as a standalone service which makes it easier to work with all around (easier deployment, scaling, monitoring, use from various languages)
  • APIs for PHP, Python, Java, Ruby, Perl, C++

The bad

  • No fuzzy or wildcard searches. We rely on autocomplete boxes and fuzzy suggestions a lot because we need to allow users to easily link to yarns and patterns that they’ve used in projects without forcing them to do normal search. However, Sphinx does offer prefix indexing and soundex matching, which may allow me to approximate some of what I want. I haven’t tried those options out yet…
  • No real-time updating. I wish I could tell Sphinx “document #19392 in the ‘forum’ index has been updated”, but I can’t. At the moment, I am getting around this missing feature by using two indexes for every search. I have a single “main” index that is very large and updated daily and a small “delta” index that is updated every 3 minutes and contains records that have changed since the last nightly run. This is certainly good enough for forum posts and the other things that I am searching with Sphinx.

The developer is working on both of these areas, which is great because I’d love to use Sphinx for everything.

Snippet

Just a little example of a real indexing and search run. The database query itself is probably a bottleneck here:

./indexer --rotate forum_posts

Sphinx 0.9.8-dev (r909)
Copyright (c) 2001-2007, Andrew Aksyonoff

using config file '/opt/sphinx/etc/sphinx.conf'...
indexing index 'forum_posts'...
collected 1126764 docs, 332.8 MB
sorted 63.6 Mhits, 100.0% done
total 1126764 docs, 332788731 bytes
total 218.109 sec, 1525791.18 bytes/sec, 5166.06 docs/sec
rotating indices: succesfully sent SIGHUP to searchd (pid=18897).

And a command line search…

./search --index forum_posts --phrase "i love cake" -l 1

Sphinx 0.9.8-dev (r909)
Copyright (c) 2001-2007, Andrew Aksyonoff

using config file '/opt/sphinx/etc/sphinx.conf'...
index 'forum_posts': query 'i love cake ': returned 23 matches of 23 total in 0.043 sec

displaying matches:
(snipped)

words:
1. 'i': 820193 documents, 2963422 hits
2. 'love': 109871 documents, 131778 hits
3. 'cake': 7191 documents, 9629 hits

I guess that’s it! If you’ve got search needs, you may want to give Sphinx a try …and don’t forget to take a look at the API for your favorite language.

Filed in Database | Tagged , , | Permalink | Comments (1)

Pretty pretty pages

Wednesday, January 2nd, 2008

Like a lot of programmers, I really enjoy writing code that looks nice on the page. No matter what language you choose to work in, you can always employ blocks, whitespace, and names to create something that pleases your eye. Some languages (like Ruby) may provide you with attractive and useful idioms and synonyms that can help you achieve that perfectly worded method.

If you are a web application developer, you’ll probably follow up your tiny masterpiece with a hideous blob of HTML (mixed with your language of choice), CSS and Javascript that places the interface on top of your application.

Prettier (dynamic) HTML pages

Witness - one hideous blob (taken from the popular phpBB). I just randomly chose this - but I think that it makes a good example. phpBB uses a templating system so the page looks sort of like a generic dynamically generated page and not PHP in particular. It is also an example of code that could use a little CSS love but I’m not going there :)

For Ravelry, I used the fabulous Haml. With Ham will rescue your pages from drowning in angle brackets, closing tags, and unnecessary verbosity. I’m too lazy to bother with posting syntax highlighted code here, so I give you a screenshot instead. Haml vs regular HTML:

Your view (HTML, etc) is probably a huge part of your code base - don’t you want it to be as clean and maintainable as the rest of your application? Haml may not be the single best way to author pages, but it demonstrates that there are alternatives to the same old mush. If you are coding something with Ruby on Rails - I highly recommend it. You’ll have more fun - I promise. If you are coding in some other language, things like Mako and Smarty can help you tidy up a little but you are still stuck with HTML and all of its warts. Of course - you might like the warts :)

Oh, and PHP folks may want to take a peek at PHAML.

Prettier Javascript

Javascript is sneaky - if you don’t keep an eye on it, you’ll find that it will metastasize into a giant meandering file.
We don’t let our other source file get huge - why should Javascript get a break?

Here are a couple quick tips for improving and organizing your Javascript code:

Organize your code into “packages”.

I’m not using one of the Javascript frameworks that provides a notion of packages, but it is easy to roll your own. I follow this Javascript module pattern.

Here is a tiny example - this is all the code you need:


var R = {};
R.messages = function() {
  var showMessageIndicator = function() {
    // this is a private method that shows a little animated activity indicator icon 
  };

  return {
    saveSelectedMessages: function() {
      showMessageIndicator();
      // save the messages... etc...
    }
  };
}();

Then in my page, I have an onclick event that calls “R.messages.saveSelectedMessages();” Once all of your Javascript is safely bundled up in neat little packages, you can get even fancier.

Separate source files for development

There are several ways to allow many individual JS source files in development without adding complexity or inefficiencies to your site. I chose to have my individual files packaged into a single versioned .js file as part of the release process. This allows me to keep my pages simple (the same 1 js file for every page) and set cache-related settings to the limit so that browsers don’t repeatedly download the same unchanging Javascript file. asset_packager does this work for me.

try a javascript framework

Beautification is one of the many reasons why a Javascript framework is useful. Most will provide shorter, simpler ways of accomplishing common tasks while adding some consistency to your app’s UI code. You’ll usually get useful effects, controls, and other UI goodies as part of the package. Don’t waste too much time coding commonly used GUI elements - the work may have been done for you already.

People endlessly fight about which is “best”. Take your pick - see what fits your taste, and consider whether your webapp framework has a preference.

A few popular choices:

Prettier CSS

CSS - I love you and I hate you. Stylesheets too quickly become large and unwieldy and full of small bits of repetition (trees of long and longer selectors and so on…). I don’t know how I’d even manage to make sense of my CSS without Firebug’s help.

The Haml folks bring us Sass. I’ve really have to check it out - by the time it was released, I was already saddled with way too much icky CSS. With Sass, nested elements are actually expressed that way (imagine that!) and lots of repetitive tag/id/class names are wiped away. Unfortunately, I can’t just run css2sass and be done with it - there are some unintentionally sneaky things going on with overriding and precedence in my CSS and I have to fix those up first.

No idea what the heck I am talking about? I swiped an image from Ajaxian that illustrates how Sass nices up CSS. This is better than typing selectors over and over all the way down to “#menu li a:hover”, no?

I’d also recommend that you split your CSS into separate files early on. Because of the overriding etc that I just mentioned, it can become much harder to split and organize a single CSS file once it has grown too large.

ugh.

PS - because Ravelry’s Javascript and CSS is packaged and minified, you won’t be able to peruse it for examples. phew! ;) The files don’t at all resemble the code that I look at while developing.

Filed in UI | Tagged , , , , | Permalink | Comments (6)

My Sysadmin Toolbox

Thursday, December 20th, 2007

Writing the software to run a site is only part of the work - you also got to make sure that everything keeps running smoothly. Keeps your systems tuned and happy doesn’t have to be drudgery - I’ve found it to be fun and interesting work as long as you simplify and streamline by gathering the right tools.

Deployment

At the moment, we are releasing new versions of Ravelry daily. Capistrano 2 makes it easy - I hit a few keys and watch it go. Here is what Capistrano does for me during a typical release:

  • Checks out the latest version of the application code from Subversion on each virtual server (actually, it updates a working copy)
  • Runs asset_packager which minifies, combines, and versions the site’s CSS and Javascript so that we have smaller files and we can ask browsers to cache like crazy
  • Updates the database to the new schema by running any Rails migrations that have been written since the last update. The schema version is stored inthe database. Easy peasy database releases!
  • Removes and stops half of the app server cluster before swapping to the new version of code and bringing them back online
  • Does the same with the other half
  • Voila! New Ravelry, and users probably didn’t even notice that it was happening

All this is done over SSH - no weird client software is required. If you work with Linux (even if you don’t work with Ruby or Rails) I highly recommend that you check out Capistrano.

Sometimes we can’t do a hot deploy (for whatever code-change related reason). In that case, Cap puts up a maintenance page while it fiddles with the app servers.

Monitoring

Nagios and Munin - I love these two tools.

Nagios

Nagios is an excellent monitoring tool. It can watch all of your services and email alerts when certain criteria are met. Ravelry has lots of moving parts - web server, app servers, master and slave database, mail, DNS, two different types of search servers, memcached cache servers, screengrabber, feed aggregator… If something goes wrong with one of these services or with a system itself (CPU, disk, etc) it is very nice to be alerted immediately. Plus, the Nagios configuration itself serves as a handy organizational tool. Best of all - it is free and flexible open source software with a feature set that beats many commercial monitoring packages.

Although it has a handy web interface, I usually interact with Nagios via the Firefox extension, the “nsc” command line utility, and email.

Munin

Munin is a really flexible and simple graphing tool. Using Munin, I can have a really handy at-a-glance dashboard that shows me the health of all of my different systems and software. I graph pretty much everything that is graphable :)

I find these types of graphs very valuable because I can monitor resource utilization over time, take a look at the effects of code changes on resources/performances, and easily spot spikes and other oddities. Here are two example graphs. The first is the query traffic hitting our master MySQL server. The second is the load average on the VM that grabs RSS feeds and takes screenshots of people’s blogs and other websites. The spike on the graph is Firefox freaking out about something while trying to grab a screencap. You can also see that the load has been stepping up little by little - something I’ll have to look into.

Reporting

I have a few other data sources that I periodically review: web usage stats, MySQL query logs, and Rails logs

Webalizer

We use Google Analytics for more advanced web stats and I hardly ever look at it. For the basic stats, I use plain old Webalizer (actually the Stone Steps version). Webalizer provides most of the information that I care about from a sys/network admin perspective.

Here is some output from the Ravelry API stats. We currently have several hundred users who are using a JSON API to show works in progress on their blogs. It shows hits, bytes, etc and breaks it down by day. Good enough for me - I can get a rough idea of where our bandwidth is going and I can see trends in the data.

Rails Logs

The rails application logs include great timing information that helps me find bottlenecks in our application. I use SyslogLogger to funnel all of the rails logs (and other useful logs) to a central syslog-ng server. The centralized logs are compressed and rotated raily, and whenever I want to take a look at performance information for 1 day of logs (which is plenty of data) I run a log file through pl_analyze.

MySQL

MySQL can be configured to output a very useful slow query log. Unfortunately, you can’t set the definition of “slow query” to anything less than 1 second, but it is still pretty helpful. Once you’ve got your slow query log, you can use the handy MySQL Statement Log Analyzer (mysqlsla) to summarize the data into easily digestible statistics.

Hackmysql.com has several other useful tools, including mysqlsniffer. Sometimes, I want to grab a snapshot of ALL MySQL activity so that I can roll it up into a summary and look for waste and opportunities for caching or more refined queries. To do this, I just run the sniffer for a while and dump the output to a file.

Bits and bobs

  • You probably want to monitor your app from outside your network as well. Pingdom is good and cheap.
  • Don’t waste your time looking for exceptions and errors in your logfiles. Add exception_notifier to your Rails app.
  • Munin plugins are really easy to write, but check out the plugin library on MuninExchange before you start rolling your own.
Filed in Network, Sysadmin | Tagged , , , , | Permalink | Comments (8)

Why Ruby on Rails?

Monday, December 17th, 2007

In this post: why Ruby? why Rails? why not? It sort of fizzles out at the end.

First - a quick definition. What is Ruby on Rails? Ruby is a programming language. Ruby on Rails is a framework for developing modern, database driven web-based apps that is written in Ruby. To grossly generalize - the framework takes care of the common tasks that are part of developing a working webapp - responding to URLs, rendering pages, usually communicating with a database.

People are often curious about why I chose Ruby on Rails for Ravelry. The truthful answer? Prior to Ravelry, I spent 6ish years developing web applications with Java (specifically Struts and then Struts 2) and the fun was gone. Java as a language had become old hat and the Java web application development community felt fractured, tired, and underenthused. I just wasn’t having any fun. I started working on Ravelry because I thought that it would be an interesting and enjoyable project and Ruby on Rails looked like it would take away drudgery and give me some of excitement. It definitely did - I love working with both Ruby and Rails and I’m glad that I made that decision.

There are lots of reasons to choose Ruby and Rails for your next web project. The short response? Productivity and Fun.

Ruby is beautiful

The Ruby site includes the caption: “a programmer’s best friend”. I couldn’t agree more. Working with Ruby has made programming fun again. Ruby code is fun to write - it is terse, flexible, pretty, and natural. If I need to write a bit of code to solve a general problem (outside of any work that I am doing) I always pick up Ruby first.

Matz, creator of Ruby, often speaks about Ruby and feeling:

“Instead of emphasizing the what, I want to emphasize the how part: how we feel while programming. That’s Ruby’s main difference from other language designs. I emphasize the feeling, in particular, how I feel using Ruby.”

Check out why’s (poignant) guide to ruby for a fun introduction to the language.

Rails is smart

Ruby on Rails understands the needs of web application developers and this understanding has saved me massive amounts of time. Without Rails I could have wasted a lot of time with unnecessary SQL, overly verbose and complex views/pages, ugly XML configuration files, clunky code… Rails is designed to feel like a specialized language for building software for the web and it shows. Common tasks are made easy and Ruby (the language) picks up the slack if you need to move outside of the real of the usual.

There is no way that I would have been able to develop Ravelry with the same speed and ease without Rails. There are now many Rails-like frameworks for other languages - a testament to how many things Rails got right.

thriving community

The Ruby/Rails communities are very active and very engaged. It’s a great place to be - there are so many interesting things going on that it’s hard to keep up. Very active projects, tons of great Ruby blogs, all kinds of books and other documentation. The huge community of enthused Rubyists is a very valuable thing.

I check http://del.icio.us/popular/ruby and http://del.icio.us/popular/rails daily so that I can keep up on all the cool Ruby and Rails related stuff that is going on in the world.

scaling is easy

Ravelry moved from a slice of a server to a full server and then on to 8 virtual servers with no architectural changes. Ruby on Rails doesn’t scale for you, it just leads you into building a scalable application from the beginning.

…and of course, there are downsides. These aren’t issues for me, but they may be for you:

more resource intensive (downside)

Your Ruby on Rails app is going to need more CPU power and more memory than an application written with PHP, Java, etc. You’ll need the CPU for Ruby - it is not as efficient as other languages (yet) and will suck up more processing power. The memory is for your Mongrels. Mongrel is one of the best ways to run your app (I guess you can call it an “application server”) and you typically have to run many Mongrel instances because each can only handle one request at a time. 10 Mongrels could easily cost you 1 GB of memory.

always on edge (downside)

If you don’t feel comfortable running the latest and greatest code (think brand spanking new code checked out from Subversion) and keeping up with what is going on inside the community, you may miss out on many of the benefits. Things change very quickly in Rubyland and you’ll get the best experience and the best information and documentation if you come along for the ride.

Filed in Ruby on Rails | Tagged , , | Permalink | Comments (6)

Moving out of the living room

Sunday, December 16th, 2007

First post. It seems like a little overview of Ravelry’s network, hardware, and other systems information would be a sensible way to start things off.

In this post: VPSes, Linux, Virtualization, hosting and bandwidth, Amazon S3

a starter home

If you are thinking about embarking on a web development project, I’d highly recommend that you find a host that can provide you with a VPS (Virtual Private Server). A small VPS can be had for as low as $10-$20. Unlike a traditional dirt cheap shared host, you have full control over an entire virtual server plus the flexibility to add memory and disk space when you are ready for it. I used (and am still using) a host called Rimuhosting - their pricing is competitive, their support people are very responsive, and they offer a pretty good menu of options including dedicated servers.

Jess and I started fooling around with ideas and code back in January. It would have been silly and expensive for us to build out our systems before we needed them, so we started with a small VPS. (Okay - so Ravelry wasn’t really in our living room when we started - it was sitting on a machine in Texas.

Eventually, we outgrew the VPS and moved up to an inexpensive leased dedicated server with the same host. When the dedicated server started to feel pokey because of limited resources, we knew that it was time to design and build our “real” setup. At this point, Ravelry was out of the starting gate and we weren’t afraid to invest money in infrastructure.

growing up

I knew that I wanted to own and manage our own servers and host them in a conveniently located datacenter. We live in Boston, so it wasn’t to hard to find a good datacenter nearby. We ended up getting a half cabinet with a company called Hosted Solutions because they were convenient, friendly, and not absurdly huge.

To get started, I decided that we’d need two big servers (ripe for virtualization, keep reading…), a firewall, and a gigabit switch. We overdid it a little - our budget was $10K and Dad had to give us a hand with the firewall purchase (thanks, Dad!). Still, I was very happy with what we put together. Here is what I was shopping for:

  • Inexpensive Linux servers with quality components and lots of options - I chose Silicon Mechanics.
  • Intel Xeons. I don’t know if AMD64 + Linux still comes with a few guaranteed headaches but I wasn’t about to find out. Plus, I really wanted to try out those dual quad core Xeons :)
  • Piles of memory. With many Mongrels and 2 MySQLs, we’ve got lots of mouths to feed. 16 GB is the current sweet spot.
  • Serial ATA RAID - 4 drives + RAID10 = speed and durability on a budget Also, plenty of disk space. Serial ATA is cheap and running out of space is no fun.

The other components were pretty basic - a Dell managed gigabit switch and a Cisco ASA firewall. Good, standard stuff. Little fuss.

bob <– Bob, stress testing the firewall and switch

avoiding those emergency trips to the datacenter

Do yourself a favor and splurge on IPMI or integrated KVM over IP. Sometimes this is built into an onboard card, sometimes it’s just a little extra add-on. It’s really nice to be able to have console access from home without bothering with more hardware, more wires, more crap.

Even better, my machines have this IPMI business which also provides remote access for power off/on, reset, and various sensors. Pretty much all of the management you need and served just the way you want it - over the network, no extra equipment, little extra expense.

It was especially convenient when I was setting the machines up in the datacenter. Just the ability to cable my laptop to a server for management without a keyboard and display was worth the added expense.

My IPMI console: not beautiful, but very useful: ipmi

which flavor of Linux?

Gentoo. Portage (Gentoo’s package system) rules. I don’t want your crummy binary packages. I want the “configure/install from the source” flexiblilty with the easy upgrading, administration, and dependency handling that a package manage provides. Also, Gentoo’s init and filesystem layouts are nice.

Gentoo is my favorite Linux and I’m very comfortable with it. I encourage you use your favorite Linux if that is what you are most comfortable with.

virtualization, baby!

My favorite part. I’m probably driving people nuts with the yammering about Xen and virtualization all the time. In short, I took my two big servers and turned them into 8 “just right” virtual servers. If you want to learn a little more about what virtualization actually is, check out the overview pages over at XenSource.

Okay, why virtualize? My own opinion:

  • Today’s servers are big - you will likely waste your resources if you run one “server” per machine.
  • An system installed directly on your bare hardware is a going to be less flexible, more brittle, and more of a headache to maintain and manage.
  • Easy migration and duplication - if your machines are all virtual you can easily shuffle them around, duplicate them, and so on

I can’t stress how happy I am that all of our server instances are virtual. Management and upgrading is actually fun. Do it. Really.

Pass the Xen, please.

Xen is the king of virtualization. It is a fantastic open source product and it costs you nothing. Three really important to think about before you start:

  • If you need to run any 64 bit virtual machines, Xen needs to be running on a 64 bit Linux and all of your VMs have to run on 64 bit kernels. Remember that you can conveniently run an otherwise all 32-bit linux with a 64 bit kernel. This is what we are doing - 64 bit kernels everywhere but only 64 bit software on the MySQL hosts that need them.

  • USE LVM FOR YOUR DISKS. If you are virtualizing but sticking with traditional disk partitions or image files for your disks, you are losing out on a lot of flexibility and performance. Use LVM2 to set up your disks - the HOWTO is here. With LVM, you’ll have flexible disks that you can create, delete, resize, and snapshot as needed. Want to be able to duplicate one of your virtual servers while it is running and start up a new copy for testing or as a base for a new machine? You need LVM.

  • If you are doing it all from scratch (including building the kernel) like I did, you’ll probably render your machine unbootable a couple times. Have a boot CD handy. The Gentoo Xen HOWTO is quite handy.

I can’t say enough good things about how freeing it is to have a large group of virtual machines that are riding on Xen and LVM. I can move servers to other machines, upgrade software, and do all kinds of other world-changing things with ease.

Xen’s “top” - this box is running 4 virtual machines: 1 web server, 2 app servers, 1 slave MySQL: xentop

bandwidth: our biggest expense

Here are a few tips for not wasting money on bandwidth:

  • Shop around! You’d be surprised how much prices can vary. Make sure to be aware of the reliability/redundancy behind the bandwidth that you are purchasing. In-datacenter bandwidth is often a “safer” blend of several different carriers and that security comes at an added expense.

  • Take full advantage of Amazon’s S3 storage service! If you are hosting any large images on your site, you should seriously consider moving them to Amazon. Take a look at the pricing and think about how it would work for you. It is really cheap.

  • Divide things up logically and don’t skimp on hostnames/virtual hosts. If you separate all of your static content hosts now (even if they are the names all point to the same machine) you’ll have an easier time fiddling with things later. Maybe we went overboard with avatars.ravelry, images.ravelry, images3.ravelry, assets.ravelry, creative.ravelry, etc… but the division has made things a lot easier for me. As an example, we’re looking at buying some cheaper in-datacenter bandwidth from a single carrier and if we decide to do it, it will be trivial for me to offload a bunch of those hostnames to the new connection. It also makes life easier when it comes to usage reporting on all of these different types of resources because the log files are already split out.

  • Later, I’ll write something about stylesheet and javascript minification, gzip compression, cache-related headers, etc etc. Without considering these things, you may be throwing bandwidth away (not to mention slowing down your users).

any questions or comments?

I guess that’s it for server/network related stuff! If you have any questions that are related to the things that I wrote about (curious Ravelry users - I’m talking to you) leave a comment and I’d love to answer in the next post.

useful links!

Next time: next time, I might talk about how I monitor and manage all of these virtual machines :) It’s not much work at all and little ecosystem of management software is pretty interesting.

Filed in Hardware, Network | Tagged , , , , , , , | Permalink | Comments (16)