Ravelry in Bullet Points

Inspired by Stack Exchange’s Architecture in Bullet Points. I have to admit that I was happy to see that we stack up pretty well, being a Ruby on Rails site. There’s no denying that Ruby is slower than C#. …and we could talk about Microsoft SQL Server vs MySQL as well.

It’s a little bit of an apples to oranges comparison. We’re a members only site that is mostly not indexed in Google and we’ve got a low unique visitor count and a high average time on the site. Also, we’re very, very budget conscious.

Traffic

  • 173 Million Page Views per Month (not including guests + sign in page views)
  • 1600 350 Rails requests per second (at peak times, requests for static files not included) whoa huge error there, 1600 reqs/sec just hitting Rails would be impressive. There were some static files included in that.
  • ? DNS requests per second. My TTLs are set wicked high for most hostnames anyway.
  • 180 Megabits per second at peak times (including static files)

Data Centers

  • 1/2 rack with Hosted Solutions (Windstream), 3 internet connections
  • 1U with RippleWeb for backup
  • Amazon S3 for backup

Production Servers

All are SuperMicro boxes running Gentoo Linux. Most are dual CPU Xeons and they range from 4 years old to new. SSDs have helped me make the oldest servers new again.

  • 4 application servers (Apache / Passenger / Ruby on Rails)
  • 3 database servers (MariaDB, 1 is a low-powered slave for backups)
  • 1 front end web server (nginx)
  • 1 misc utility server / front end web server
  • 1 caching proxy server for static files (nginx)
  • 3 router / firewalls (2 x Cisco, 1 x pfSense)

The most powerful server is the newest database server – a Dual Xeon 5530 with 8 x SAS drives and 48 GB of RAM (all in 1U!). Obviously, we can’t scale up forever but our database + indices is about 220 GB and there are some easy opportunities for splitting big tables out into other servers.

Software and Technologies Used

I usually do a yearly “Ravelry Runs On” post but here are the bullets.

  • Ruby on Rails
  • MariaDB (a better MySQL)
  • Phusion Passenger and REE
  • Gentoo Linux
  • nginx
  • haproxy for load balancing
  • Sphinx for search
  • memcached for caching
  • Redis where it helps
  • beanstalkd for a work queue
  • also using Java for a couple number crunching background jobs
  • Nagios for monitoring
  • MRTG and Munin for graphing
  • Git for source control

Developers and System Administrators

Here’s the part where I sound like an ass. It’s just me. I know that it will be healthier for the site (and for me, too!) to have a copilot. Maybe that’ll be in the cards for 2011.

Comments (9)

  1. Jeff Putz wrote:

    Great post as always. Have you looked into any cloud-based hosting and services? We recently deployed a new app at work (in Azure, naturally), and it was in a lot of ways a really freeing experience in terms of worrying about scale.

    Saturday, February 12, 2011 at 4:57 pm #
  2. casey wrote:

    Hi Jeff,

    I’ve used Amazon AWS on occasion when I needed a lot of power for a short time (for example, creating millions of thumbnails for all of our images when we added a higher resolution thumbnail)

    It’s just way too expensive for us.. and the advantages are few. Our half rack isn’t a sysadmin burden and our growth and traffic is pretty predictable and not very prone to spikes.

    Saturday, February 12, 2011 at 5:01 pm #
  3. Jeff Putz wrote:

    That’s what I was most curious about. There are certainly cost advantages at a certain scale, but it’s hard to know where that scale is.

    Saturday, February 12, 2011 at 5:04 pm #
  4. David Mann wrote:

    The growth from “it works on my development workstation” to “we’re serving up millions of pages a month” is fascinating. Interesting to see the evolution of the technology behind the site.

    I will send mojo your way if/when a copilot search is initiated. Finding someone on the same wavelength can really help when bouncing ideas around and I’m sure it would be nice to have a backup in case you wanted to try something novel like taking a week or two off :)

    Saturday, February 12, 2011 at 8:25 pm #
  5. Great post, I love these types of posts. Info is very interesting.

    I am curious why you chose Gentoo as your distro?

    Do you find any advantages of MariaDB over plain MySQL?

    Sunday, February 13, 2011 at 11:38 am #
  6. Audreygs wrote:

    Trust me when I say you will NEED a co-pilot once your “passenger” load increases.

    ~ Audrey from Burbank’s Unwind

    Tuesday, February 15, 2011 at 2:48 am #
  7. chief wrote:

    I am interested in how you learned to set all of this up. Can you recommend any books or blogs for scaling up a rails project like this. I just have a basic setup with a single nginx server and a single passenger application server, which is adequate for now at least.

    Monday, February 21, 2011 at 3:27 pm #
  8. chief wrote:

    ..and the Secret of Monkey Island is the best game ever!

    Monday, February 21, 2011 at 3:28 pm #
  9. Salman wrote:

    I love these kinds of posts :)

    Curious, when running your own rack, can you list all the hardware (and rough pricing) of things you need OTHER than the actual servers? i.e. switches, power supplies etc.

    Wednesday, February 23, 2011 at 2:37 pm #

Trackback/Pingback (1)

  1. [...] Ravelry in Bullet Points — Code Monkey Island codemonkey.ravelry.com/2011/02/12/ravelry-in-bullet-points/ – view page – cached Inspired by Stack Exchange’s Architecture in Bullet Points. I have to admit that I was happy to see that we stack up pretty well, being a Ruby on Rails site. There’s no denying that Ruby is slower than C#. …and we could talk about Microsoft SQL Server vs MySQL as well. Show influential only (2) $(‘#filter-infonly’).change(function() { var el = $(this); var url = document.location.href; var checked = el.attr(‘checked’); if (checked) { document.location.href = url + ((/?/.test(url)) ? ‘&’ : ‘?’) + ‘infonly=1′; } else { document.location.href = url.replace(/[?&]?infonly=1/,”); } }); [...]