A Look inside LiveJournal

Rarely does a company offer people the opportunity to see its infrastructural organization. Danga does in this rare
talk from LISA 2004.

Some takeaway lessons from the talk are:

• Separate services across hosts
• Scale each component separately
• Avoid large database replication by using internal pre-indexing of data across databases
• Let your services fit your needs (ie use both an IP and an HTTP filter for different problems)
• Storage is key, eg their use of both NetApp, mySQL replication and internally developed storage system MogileFS
• Cache is key, use Memcached
• Measure and monitor all aspects of the system for availability and planning

This talk is from 2004. We can only speculate to how things have changed since.

The Five Why’s

Joel Spolsky discusses Five Why’s, a method of determining what went wrong in a system and an approach to preventing it in the future.

In an occupation where outages are high profile and we never like to do the same thing twice, we’d all be wise to heed Joel’s story and advice.

Optimizing your database with Paul Tuckfield

Paul Tuckfield spoke at the mySQL conference in April about optimizing Youtube.

Toward the end of the talk, Paul shifts to a mainly system-oriented focus on optimization and presents a few tips:

• Increase the number of spindles
• Use software RAID striping for increased performance
• Eliminate non-database read-cache
• Use predictive selects to help optimize cache
• Use Memcached
• Reduce serialization by reducing/eliminating non-master nodes

You can listen to the talk here


or watch the video.

How Google Does Things

LISA is one of the best conferences relating to system administration and in ’06 Tom Limoncelli gave a frank and informative talk on how Google handles the largest of the large systems. Though few of us have access to the same tools and resources as Google, I think everyone can take home something from this talk, “My First Year At Google”