• Migrate from San Jose to Phoenix
• Implement fault tolerant infrastructure

During the maintenance period subversion will be unavailable for both reading and writing. Because we’re
switching to a newer version of subversion (and changing the data store to fsfs) the data migration will require a time consuming svnadmin dump / svnadmin load. We anticipate this step alone will take about 4 hours.

Time: January 28th 8pm-2am PST
Scheduled downtime: 6 hours
Estimated actual downtime: 4.5 hours
Impact: All subversion related services will be unavailable including viewvc

#1, No email for two days

First up on the plate was a major Zimbra outage. This was sparked by a RAID failure in one of our HP Storage Blades, but rapidly escalated into a data-loss situation due to what can only be described as poor backup planning.

The short version: backups were being made regularly, but were not being reliably shipped off of the server. It took a lot of effort from IT and patience from our users (that is: all of Mozilla’s paid staff… thank you!) to get back on track.

Remarkably, this had relatively minimal affect on development or release cadence- it didn’t delay the Rapid Release train or close the tree, and newsgroups were unaffected. It may have caused some features to develop more slowly than they otherwise would have, but the community and the company pulled together to get through the situation with more ease than anyone could have expected.

Fortunately, things on that front are much better now- our Zimbra infrastructure has improved, and the backup strategy has changed such that the same issue is largely negated. We’re better off now than we were 6 months ago, and we have plans to be significantly better yet in another 6 months.

 #2, addons.mozilla.org & versioncheck.addons.mozilla.org

In mid-December we began having major performance issues with sites behind our Phoenix Zeus (load balancer) cluster. This cluster supports a number of production sites, including addons.mozilla.org and versioncheck.addons.mozilla.org.

The issue was caused by an unexpected and extended increase of traffic from Firefox 3.6 users upgrading to Firefox 8 (and the Addons version checks).

This also highlighted 2 architectural design limitations that we were aware of but had not expected to be a problem for some time.

This Zeus cluster sits behind a redundant pair of Juniper SRX firewalls. These protect the Zeus Linux hosts and provide a point to monitor for unwanted activity (IDS). Like any device, these firewalls are limited by the number of concurrent sessions and new sessions per second that they can handle. The additional traffic put us over the threshold, and they started to drop connections.

Moving Zeus from behind the SRX solved one problem while exposing another bottleneck. This time, we learned that the traffic for versioncheck.addons.mozilla.org was overwhelming the 1GbE interfaces on the Zeus cluster and we had to quickly spin up 10GbE Zeus nodes.

These are really just general scaling issues that we were going to need to deal with sooner or later. Unfortunately, we hit a level of scale that didn’t fit the way we had planned (a lot of these improvements were things we had planned on doing in the early part of 2012).

Among the fixes were:

• TCP stack tuning
• Upgrading to 10-gigabit Ethernet on the Zeus hosts
• Routing / firewall changes (including removing the hardware firewall and switching to iptables)
• Reducing our use of multicast VIPs, opting for multiple “normal” VIPs using DNS to send traffic to all load balancers
• Segregating backend traffic onto separate load balancers in a different LB cluster
• Sending traffic to other datacenters (notably: versioncheck.addons.mozilla.org, which is highly cache-able)

There’s more work still to come on this. We are currently experimenting with ScaleArc iDB, a database/SQL aware load balancer, which would theoretically give us database query caching and query distribution. We presently do such load balancing through our main Zeus cluster’s, but they don’t understand any database-specific protocols… it’s just simple TCP-based proxying. A protocol-aware load balancer should provide the same performance benefits for database queries that an HTTP-caching-LB does for web content.

So, what’s the long term fix?

From these events we have drastically altered our approach to how we handle certain parts of our infrastructure. We are very excited to be working on what we are loosely dubbing the “Hyper-Critical Infrastructure” cluster, which is a completely standalone VMware / NetApp installation designed to be as autonomous as we can reasonably make it. For starters this will house Zimbra & LDAP. Sometime after that we will also likely migrate Mana – our internal documentation system based on Confluence – and intranet.mozilla.org. Other extremely critical apps are also fair game.

Zimbra

Specific to Zimbra, we’re spending some time to make sure we make it as scalable and reliable as we need it to be. VMware High Availability and NetApp storage gets us freedom from most kinds of hardware failures, and Zimbra internally supports sharding to multiple servers for scale. We know Zimbra can do the job (it’s used by organizations much, much bigger than Mozilla), and this architecture will get us there.

Zeus Load Balancers

We’re continuing to replace the 1GbE Zeus cluster (HP BL460c) with 10GbE servers (HP DL360 G7) and are working on sharding our traffic into separate Zeus clusters where it makes sense to do so. We are also looking into separating database traffic away from Zeus altogether and onto a protocol-aware load balancer that can cache results.

On a higher level, we’re pushing out services like Cedexis to augment our geo- and performance-based global load balancing. This replaces another external service (3crowd), the discontinued “Zeus GLB” app, and the newer “Zeus Multi-Site Manager” app. This gives us a unified, convenient, and high-performing way to “front” Zeus and distribute traffic efficiently between multiple Zeus clusters.

It wasn’t all bad

Of course a number of good things happened in December as well:

• BrowserID went to production
• Ramped up Tegra capacity for Native UI & Android UI testing for Firefox Mobile
• Our Inventory system got a nice overhaul and a migration to a new cluster
• Lots of CDN work, including SSL CDN trials (Akamai, Highwinds) and Cedexis experimentation / implementation
• Turn-up of a new 9-cabinet module in our PHX1 datacenter (fortunate, since it helped significantly with the load balancer issues)
• A new ESX cluster in PHX1 (apart from the Hyper-Critical one above, which happened later)
• Dozens of web content pushes – these are so reliable now we don’t even announce them anymore
• BrowserID & LDAP integration work for the Mozilla Community Directory

It can be easy to forget about the wins, because in general you’re winning whenever something isn’t broken!

We’re hoping to get back on track with more frequent updates… look for more recent info very soon!

Jake

The following repos have been added and are processed daily:

• comm-aurora
• comm-beta
• comm-release
• mozilla-release
• l10n-mozilla-release

Additionally the comm-2.0 repo was added as a one-time processing job, so it is indexed and searchable now as well.

The following repos have been moved from daily processing to weekly processing. These seem to be receiving very infrequent updates (less than monthly), so I believe it will not significantly impact anyone.

• bugzilla2.20
• bugzilla3.0
• bugzilla3.2
• firefox2
• fuel
• incubator-central
• mozilla1.8
• mozilla1.8.0
• mozilla1.9.1
• mozilla2.0
• l10n-mozilla1.8
• l10n-mozilla1.8.0
• l10n-mozilla1.9.1
• l10n-mozilla2.0
• tamarin-central
• webtools
• webtools-central

If you feel that any of these need more frequent MXR processing, please let me know about it by filing a bug with the IT Request form. You can open a bug in the main Webtools::MXR component as well, but since IT is curently managing the MXR repos it will take longer to make its way to us if you go that route.

Jake

Background

MXR is the Mozilla Cross-Reference system. It’s basically a convenient way to search a whole lot of Mozilla code for certain things. I’m sure a proper Mozilla developer could tell you all about how awesome and terrible it is, and what it can and cannot do for you. As a sysadmin, I can tell you it’s a rather intensive, complicated set of scripts that have to handle a lot of different code. It pulls code written in many different languages, from multiple different revision control systems, and indexes them. Some code trees are processed more than once a day, the rest are done daily.

As you can imagine, this results in a lot of special cases. MXR breaks it down into basically 3 steps (conveniently split into 3 scripts) – update the tree, generate the cross-reference identifier database, and generate the searchable index.

1. The first step (update-src.pl) is relatively straightforward. It consists basically of updating the code for whichever tree you pass it as an argument. It’s a lot of special cases, since most trees are handled slightly differently from one another, but all in all it’s about what you’d expect. It’s the equivalent of “hg pull” or “svn update”. The only trouble is some trees are not nearly that simple, and involve quite a bit more work to update properly. In fact some trees are actually nested source repos, which must be updated independently.
2. The second step (update-xref.pl) is where most of the time is spent. The ‘genxref’ script is a giant mess of Perl regex’s to detect identifiers (function names, etc) in various programming language. You give it a tree, and for every file in that tree it determines the file type and scans it appropriately.
3. The third step (update-search.pl) is once again relatively simple. Searching is done via “glimpse”, so this is basically running “glimpseindex” on whichever tree you call it with.

The three steps are tied together with an overall calling shell script (update-full-onetree.sh). This is responsible for feeding each of these three scripts the appropriate arguments (the name of the tree), and reporting the overall output.

This calling script is in turn called by another calling script, which calls it for every tree. This is the final link in the chain, and this script goes in cron. There are actually 2 of these- one that cron calls every 4 hours, one that cron calls daily. They’re basically identical except for the list of trees they process.

Now that you know how it works, I can tell you about how it’s better now than it was a month ago.

Improvements

First things first: the daily script was no longer reporting its output. It was working, but not telling us about it. This was due to a newly-introduced “rsync” job, which was running with the verbose flag. This caused the output to be far larger than it should be, which broke the reporting. I removed the verbose flag, and all is back to normal.

With that out of the way, I could see that the “4-hour” job was taking approximately 2 hours to run, and the daily job as much as 30 hours in some cases. That’ll typically make any sysadmin a bit squeamish, and indeed it is non-ideal. However, these jobs have no built-in parallelism, and this server has multiple CPU cores available… meaning, the overlapping wasn’t normally a big problem.

At this point I wanted to improve the running time directly. Step 2 above is by far the longest-running (and hardest on CPU time), so I started there with the very nice NYTProf Perl profiler. This made it clear that most of the time was being spent in a few of the more complicated regex’s used in this script. I was able to make some very small improvements, but ultimately nothing appeared to be severely wrong- they weren’t doing anything terrible, and in fact appeared to have already been tuned by someone who is frankly better at it than I am. I did get some great suggestions on ways to improve the whole process, however, which may yet be implemented some day.

Having quickly scanned over this issue and found nothing of significance, I moved on to “update-search.pl”… the 3rd and final step. I skipped NYTProf here, because the usage is quite obviously tied up in “glimpseindex”. I was able to make a few small tweaks here, which I believe may ultimately result in faster searching. Specifically, I upgraded our indexes from the default “tiny” size to the mid-level “small” size. This roughly doubled our disk space used for indexes, but should have provided a nice boost to search performance. This also makes the indexing take longer, but it still ends up being much faster than the “step 2″ above, so I consider it a good trade. Unfortunately as I mentioned, IT doesn’t have a lot of face-time with this app, so it’s hard to judge for myself just how much faster searching really is.

Finally, the biggest change- parallelization of MXR jobs. After some time looking around, I couldn’t see any reason why we could not process multiple trees simultaneously. So, I set out to do just that.

I wanted to start simple, just to get something in place to prove that it would work. Thus, I re-worked the 4-hour cron job into a simple loop over the list of 4-hour trees, and had the loop execute 4 tasks at once and background them, then “wait” for completion before continuing. This is extremely simple using nothing more than standard shell semantics:

PARALLEL_JOBS=4
count=0
for i in $TREES; do
    echo -n "Starting $i at "; date
    nice -n 19 ./update-full-onetree.sh -cron $i &
    let count+=1
    [[ $((count%PARALLEL_JOBS)) -eq 0 ]] && wait
done
wait

This works beautifully, for what it is. The problem is that it doesn’t always keep 4 jobs running. It starts 4 jobs, waits for all 4 to complete, then starts 4 more. This results in large gaps where we would ideally like to be running another tree, but instead just sit and wait. If the 4 jobs started at the same time all take about the same amount of time to run, it’s pretty good. But if one job takes an hour and the other 3 take only 5 minutes, you’ve got a lot of idle time. Effectively, in some cases it’s not much better than serial execution.

Still, this was enough to bring the running time on the 4-hour job down from 2 hours to about 45 minutes. A very nice win. I knew I would have to come back to this later.

You may notice this doesn’t actually change the amount of work being done, it just crams it all into a smaller amount of time. While that’s absolutely correct, I believe it still results in a performance win for searching- these update jobs are rather disk-intensive, and tend to obliterate the cache. By grouping them up, we effectively obliterate the cache for a shorter amount of time, meaning the *rest* of the time should benefit from somewhat less turbulent disk caching. Basically, longer periods of “good” and shorter periods of “bad”.

What I really wanted to do though was to actually do less work during an update cycle. One simple way to get this is to detect whether anything has changed since the last execution, and if nothing has, to skip as much as possible.

Clearly, we still need to do step 1, or we won’t actually know if anything has changed. Once that’s out of the way we can make a determination. Theoretically it should be possible to do this very efficiently… *if* the tree’s revision control system will tell you that. Unfortunately, this is a brick wall for a generic solution. I could write up specific checks for each individual tree, but I really wanted to have something more easily maintainable- I already knew it was a pain to do this, simply by working on the script for step 1. In the end, I settled for a simple “find” command, to look for any files modified more recently than the last time the tree was processed. It’s a lot more overhead, but “works every time”. The overhead is actually negligible compared to the runtime of step 2 and 3 anyway, so it actually ends up not mattering too much.

This results in a significant reduction in IOPS and CPU cycles consumed. It should also cause far less cache destruction, as most of the files are never actually read. Of course there is still some caching problems caused by this “find” command, but all in all it should be a vast reduction from actually reading each and every file.

With this in place, I was really starting to feel the inefficiency from the simple shell-based parallelization scheme. When a tree doesn’t get changed, it’s runtime gets very short- but this is a worst-case for that algorithm, so it devolves almost back into serial execution! Of course the overall runtime is still pretty good (30-45min, depending on which trees have or haven’t changed), but the efficiency is getting worse- there’s more dead time in between jobs. So I took the next step.

GNU Parallel to the Rescue

If you haven’t heard of GNU Parallel, I highly recommend it. There is an exceptionally good tutorial video on using parallel, here. Suffice it to say, I rewrote the above loop to look like this:

echo "$TREES" | parallel -j+0 'echo -n "{}: Starting " && date && nice -n 19 ./update-full-onetree.sh -cron {}; echo -n "{}: Ending " && date'


I’ll admit it’s not nearly as pretty to look at, but it has one very nice improvement- it will make sure that there is always the right number of jobs executing. If a job finishes, it will start the next one right away.

With this in place, the 4-hour job is now reduced to a maximum of 30 minutes… and a minimum of eight. Compare that to the original time of about 2 hours, every time, and you can easily see this is a massive improvement.

I’m making the same set of changes to the ‘daily’ job, and although I expect some great things, it unfortunately has one significant weakness: one of the trees alone takes about 17 hours to process, and gets pretty constant usage. So this will largely eliminate the situation where this job can ‘overrun’ itself, and will consolidate the vast majority of the trees to be completed very quickly, the overall job will still be bounded by this one tree.

Future Improvements

There are quite a few places where MXR could be improved further, but unfortunately much of it may be beyond my capabilities and I’ll need to seek some outside assistance.

• Update step 2 to be multi-threaded internally. This will speed up jobs like the 17-hour monster.
• Update step 2 to intelligently skip files within a tree if they haven’t changed since last processed. This would cut down on the per-job runtime significantly. As you can imagine, most files don’t change every day- changes are concentrated into only a very small percentage of files. If we can quickly skip the others, we can save a whole lot of wasted effort. The trick will be to do this without relying on a revision-control system, since there’s so little uniformity in that area.
• Update step 3 to be more intelligent about generating indexes. This will take some playing with “glimpseindex”, but the short version is I believe we are discarding the previous index every time and starting from scratch. Obviously this is non-ideal.
• Move MXR to a better home. It should be quite possible for the web app to live on a separate machine from the backend processing, which should help with caching. For that matter the web frontend could likely be a normal load-balanced cluster, and gain improved query time through horizontal scaling. Even the processing could be split up across multiple machines (and GNU parallel in fact makes this incredibly straightforward).

New Trees Soon!

Lastly, there are at least 2 trees I’m looking very closely at adding in the very near future. I don’t want to spoil the surprise, but both have been requested more than once, and have very good reasons for being added. Having dug into MXR quite a bit recently, I’m in a much better position now to do this than in the past. Look for an update on this soon.

if only it were this easy...

• New mailbox store – Our last Zimbra hardware upgrade was back in April. We are at the point now where we need to shed some mailboxes off to a second server to help with load.  This is not so much due to the growth in the number of mailboxes we host, but because most of our users have multiple clients hitting Zimbra all of the time. Desktop, laptop, cellphone, tablet..  multiply that by the number of mailboxes and we have to start spreading the love to an additional server before it becomes a problem.
• New VMWare servers – More VM space for QA, Labs, and more!
• New load balance test box – We have run into issues with our blade-hosted load balancer boxes.  We will swap a couple out in one of our data centers for some beefier boxes and 10gig ethernet connections.  Depending on our results here we may be reworking our existing load balancing clusters to fit this new model.  In addition, this will provide guidance for our new data center in the spring.
• New gear for addons.mozilla.org – a whole rack of blades, in fact!  We are also awaiting the arrival of some Fusion IO flash accelerator cards for the AMO database nodes.  This will greatly increase the speed of I/O for those database nodes, ending up in faster results and more capacity.  We did the same for the bugzilla databases recently with great success.
• Elastic Search servers for Socorro – 8 1U servers (aka “pizza boxes”) are on their way for Socorro.
• New servers and infrastructure for China – We are prepping infrastructure for a new data center in China that will provide plenty of room and power where our current data center is tight.  This allows us to source some sites locally there as well as cache others there for improved performance throughout the region.

On a logistical note, we are getting a new block of space in our Phoenix data center that all of this gear will go in once it is ready (except for the China servers).  We are close – just a few more pieces to that puzzle and we will start racking this new gear.

That’s all for now, it is time to get back to obsessing over FedEx trackers and logistical details.  Next week, we invent teleportation.

Here’s a breakdown for of the traffic for getpersonas.com, by region:

The CDN traffic for http://mozilla.org/firefox is similar… although Asia ranks a bit higher (22%) and North America lower (30%).

This should tell you right away that if we just focus on North America and Europe (as many CDN’s do), we’re going to cut out almost 1/3 of our visitors. Because of this, we’re spending a good bit of time researching CDN performance, trying to find platforms that will improve the user experience for our users in “the other 30%”.

Let’s drill down into South America a bit. This is a region that doesn’t get a lot of love, at least from a CDN perspective. Quite often the nearest CDN node ends up being in Miami or southern California, and is at least 250ms away… multiply that over many page elements, and it’s quite easy to end up with a 10 second page load time. Other times there might be a nearby node, but because of poor ISP peering it’s not actually reachable from very many networks. There’s also a vibrant Mozilla community here, and we’d like to be able to do a better job for them.

This lines up fairly well with the overall World Internet Stats: http://www.internetworldstats.com/stats10.htm.

So if you’re in one of these regions, fear not! We do care about you, and things will get better soon.

Prior to now, getpersonas-cdn.mozilla.net pointed to a CDN aggregator service at 3crowd. We use this for several things. For example, aus2.mozilla.org (part of the Firefox Automatic Updater Service) goes through 3crowd. It works very well for this type of thing, where what we’re really after is distributing the load across our own infrastructure, in order to gain redundancy.

One of its limitations though is that it is inherently disconnected from the actual CDNs being used. That is, it has no way to know if a particular CDN is really good in one country, but really bad in another. It can’t make that type of determination on-the-fly. By default it just assigns a weight to each CDN, and doles out the traffic in whatever ratios we specify.

For a normal website like getpersonas.com, with broad international appeal, this isn’t quite ideal. For folks in North America or Europe, it works just fine… generally CDNs have pretty good coverage there, and for the most part it won’t make a big difference which one you end up using. Outside of those regions though, CDN coverage is much more hit-and-miss. If you’re in Australia or Argentina, which CDN you get routed to might make a very big difference on your page load times.

For example, here’s a “before” graph of getpersonas.com load time from our Gomez test node in Sydney, Australia (on Telestra, to be precise):

As you can see, it’s quite spikey. If you squint just right you can identify 3 tiers of performance. Not surprisingly, they line up with the 3 different backend services configured in 3crowd.The worst tier is from a local node we control ourselves in San Jose, CA. Clearly, that’s a pretty terrible choice from Australia… and yet it’s getting 1/3 of the overall traffic, simply because 3crowd doesn’t know any better.

This morning I changed this record away from 3crowd towards a new service we’re demoing, Cedexis. This is a similar service, but they maintain a database of CDN provider response times around the world… all the way down to the individual ISP level. Using this database, they can intelligently choose which CDN to use for every single request, theoretically using the fastest one from the user’s location.

Unfortunately it’s too soon to have a useful “after” graph to share. Instead, I’d like to share some information as to just how effective the new service might end up being.

“Global Village” is (as far as our traffic mix is concerned) the 4th biggest ISP in Brazil. They account for approximately 8% of Brazil’s traffic to getpersonas.com. Brazil as a whole is 45% of our South American traffic, which in turn is around 9% of our worldwide traffic. That’s the level at which we can drill down to. Here’s a graph of the “decisions” made by Cedexis for this one ISP in Brazil, over the last 24 hours. Remember, before today it would have been 50/50.

This is 94% vs. 6%. This should give you some idea of just how important choosing the proper CDN could be… far more often than not, we were making the wrong decision for these users. Situations like this generally occur when an ISP has limited peering with upstream providers, and thus does not have a good route to some places. This particular ISP appears to have a good connection to CDNetworks, but a poor one to Edgecast. Obviously this type of thing is more prevalent in smaller ISPs, where they may not be able to get (or afford) more complete peering agreements. You generally won’t see this type of thing with very big ISPs.

Now aggregate this around the world, and you can quickly see that even if the worldwide traffic mix still comes out to approximately 50/50 (and it does… within a few percentage points), making better decisions locally can result in a much better experience for a large group of users.

Over the next few days we should get a good idea of how much this is actually helping… I expect small gains worldwide, and possibly large gains in certain regions. Especially outside of North America, I expect getpersonas.com will be substantially improved, with Asia-Pacific and South America seeing the biggest gains. When we have some data, I’ll post a graph showing the worldwide before-and-after.

The why:

At Mozilla we depend heavily on our OpenLDAP-based[1] authentication system. As we’ve grown quite a bit over the past year or so, it has become apparent that our LDAP ecosystem wasn’t scaling accordingly. Until now, we’ve relied mostly on a single master server with a few slaves all behind an aging load balancer that has given us trouble in the past. Most of the slaves were no longer in the pool as their configurations have drifted over the years and a lack of documentation and consistency made it hard to add capacity, make changes or even ensure high availability in the event of a hardware failure. All the LDAP servers were actually servers primarily dedicated to other services, so extra load on those services would have the side effect of making the LDAP service unreliable. As we’ve added a few satellite offices and an extra datacenter that all relied on having some sort of authentication system in sync with our primary LDAP server in our San Jose data center, it was decided that we need to redo this setup with something a little more scalable and with better configuration management.

The what:

Over the past 6 months or so, I’ve done quite a bit of research learning how OpenLDAP works, how it behaves, how it scales and most of all, how it is set up and being used at Mozilla. Understanding the current setup was key to designing a better architecture. The first thing to do was to gather all the information, the configurations of all the existing LDAP servers and merge those into a centralized configuration management tool. As we’ve been using puppet[2],  I wrote a module to manage our OpenLDAP infrastructure. This turned out to be a little more difficult task than anticipated at first, as it had to support managing SSL certificates, a master server, slave servers, intermediary servers (servers that act as a master to other slaves, but are slaves themselves, replicating from the main master), all of which have slightly different configuration directives but overall need a similar configuration that stays in sync. Furthermore, we have some applications that act as frontend management tools for LDAP. Some are used by our team to provision new user accounts, maintain access control groups, etc. but also our phonebook directory app, which allows users to edit their own LDAP entries through a web interface. All these things needed be taken into account when re-architecting the infrastructure. All of the apps were hosted on a single server, which was also the master LDAP server among other things. If that single box failed, we’d be in a bad situation.

The how:

I designed a new infrastructure that splits the various components out a little. The phonebook app would live on a cluster of 6 machines (shared with other, similar apps – think: intranet). Our LDAP master would move to a dedicated server, not used as an authentication backend for other services, but rather have dedicated authentication slaves in each datacenter and office. As we have more capacity in our Phoenix data center, the master would live there along with the webservers serving the phonebook app. Starting out, there are two dedicated slaves replicating from the master in Phoenix, load balanced behind our Zeus load balancer[3] cluster to be used as an authentication backend for any services in our Phoenix datacenter. This setup makes it easy to add capacity as needed, and provides high availability in the event of a failure. There are also two dedicated slaves in our San Jose data center, but rather than have them both replicate from the master in Phoenix, it was decided to add an intermediary server in San Jose. The intermediary server would replicate from the master in Phoenix, and the San Jose slaves would replicate from it. Aside from reducing the cross-datacenter traffic, this provides better data consistency within a datacenter. A similar intermediary server was set up in our Mountain View office to provide a single point for all the other office LDAP servers to replicate from. Each individual Mozilla office has two LDAP slaves and instead of a load balancer, those use a virtual floating IP address that can move from one server to the other using keepalived. We balance our other services such as DNS in this fashion and it reduces the need for extra load balancing equipment in the lower traffic offices, where LDAP is primarily used for wireless connectivity. All servers would be setup completely using puppet, so that in the event of a hardware failure, or the need to add more slaves, adding a few lines to our puppet configurations can make this happen in a matter of minutes without much thought or effort involved. The other piece to the puzzle is our internal addressbook. We use OpenLDAP to provide addressbook lookups for mail clients. In the past, this was done using a single machine that was an LDAP slave. As we scale, we needed better redundancy there too. The addressbook would now live on two machines, also behind a load balancer. As an extra security measure, since this service is directly on the internet, it was decided to change the configuration to make the addressbook “slaves” be simple proxies that only allow the lookup of a few select attributes. A compromised addressbook server would not result in a compromised LDAP database. Win! Speaking of security, the entire LDAP infrastructure was moved to a more secure VLAN, completely inaccessible to or from the internet, with the addressbook being the only thing with any exposure to the internet. Also all ACLS were audited and updated to provide the minimal access necessary, while at the same time using standardized templates with puppet making it easy to add and remove ACLs as needed with proper version control and auditing in place.

The move:

First, I set up all the new hardware and set up the full configuration using puppet. After drawing diagrams and identifying all network flows needed for replication and authentication to work, I worked closely with our network operations team to make sure everything would work as expected. Then I set up our San Jose intermediary server to replicate from the old master in San Jose to ensure that the overall flow of replication would work as expected and began testing various LDAP queries. Meanwhile, I set up the phonebook application in Phoenix on a new cluster of seamicro servers and began testing it against the new master. All of this was unknown, as we were moving from using a local LDAP server to using a remote one, going from RHEL5 to RHEL6, moving from single-hosted to multi-hosted. It was a whole new environment. I worked with the webdev team and our own tools developer to update our LDAP apps to work in the new environment. As we were already rolling out our new offices in San Francisco and Toronto and Paris, I set those up to replicate from the new intermediary server in Mountain View, so that half of the infrastructure has already been in production for a while, and was actually crucial to the testing phase. Once I was satisfied that everything would work properly, I identified what was left to do to get completely off the old infrastructure and move to the new. At this point everything was set up, and it was essentially just a matter of moving the master database to the new master server in Phoenix, changing some DNS names to point at new IPs and making sure that all the clients still worked as expected. All of this needed to happen with minimal downtime as we rely on LDAP being available 24/7 for so many things, including mail, wi-fi, svn, mercurial, our intranet wikis, shell servers, etc. I decided to tackle the project on a Saturday when the least number of people would be affected. I was pretty confident that the move could be done in under two hours, since I spent months preparing for it and ironing out the details. This was mostly the case, and there were only a few brief downtimes during the two hour window where various services failed to authenticate. This was mostly when I had to re-sync the slaves to have them catch up to their new masters. I did run into a few problems though, and here is the postmortem on that:

The Postmortem:

Everything went as planned. I shutdown the old master, copied the full database to the new server, set the intermediary slave in San Jose to replicate from the new master in Phoenix and reset synchronization on the slaves in Phoenix. At the same time I changed the DNS to point at the new load balanced IPs. I started the process around 7am on Saturday of ensuring that the shell servers could talk to the new LDAP servers, fixing some clients that had hardcoded the old master as their LDAP server and added nagios checks to all the new slaves and the new master. At 10am, the maintenance window started, so I shutdown the master and did the move and DNS changes then. By 10:30 San Jose was completely using only the new servers. Then I changed the DNS in Phoenix to point at the new servers and made sure the replication was working properly, both locally and remotely all the way to the remote offices through two intermediary servers. Everything went pretty well. By the end of the maintenance window at noon, I was just double and triple checking that replication was working. Around 1pm, I remembered that although I had made sure that our password reset app was working on our new cluster in Phoenix, I hadn’t ever tested changing my password and ensuring that the change would replicate properly. I tested it and found that it didn’t work. I worked with Rob Tucker, who happened to be online at the time to try to troubleshoot why it wasn’t working. It turned out that there was one piece missing from our new master. A password check module that we’ve had in use for a few years, but was completely undocumented. Rob helped me get the 64-bit version of the module compiled and installed in the new master and we finally got a password change to go through… but still not with the webapp that is user-facing for this purpose. We discovered that the app had one portion where it had hardcoded “localhost” as its LDAP server, rather than honoring the configuration directive. After patching the app, we finally had a successful password change.  At approximately 4pm, I was wrapping up and ready to leave for the evening, when a user e-mailed me to inform me that the phonebook app wouldn’t allow him to change anything in his profile. I couldn’t reproduce the issue, but realized this was because as an LDAP administrator, I have full admin rights to the LDAP database, whereas a regular user has a different set of ACLs that apply. This is when I realized that in all my testing, I neglected to test a normal user account. I hacked on the permissions a bit more on Saturday night to get that working and had the ACLs fixed by midnight (I took a break from 5pm – 11pm). On Sunday morning, I went through and verified that the little surprises I discovered the day before were documented and added to the puppet manifest. I checked in the temprorary patch to the password reset app into svn and tested it again to make sure there were no more glitches. I also fixed the nagios alerts for the Mountain View slaves, which were misconfigured and wouldn’t have alerted to a problem, if there had been one. I’m glad I came back to double check that.

What I learned:

Details are important. Now that we have the new infrastructure, my next priority is setting up a stage infrastructure that can be used for the phonebook app, the password reset app and other tools and in general as a place to test and stage changes to our ldap infrastructure.

Testing is important. I should have tested the password reset app. I should have tested the phonebook with a normal user account.

People can be single points of failure. Before I started working on this, there was only one person with the full set of knowledge of our LDAP infrastructure. He left earlier this year to pursue other things and although he had documented most common issues and troubleshooting steps for our infrastructure, there was a huge amount of information that we didn’t have (the password check module for instance), and I had to learn a lot from scratch. I feel like I’ve learned a tremendous amount about how OpenLDAP works in the past 6 months and I enjoy working on it, but it is now extremely important that I don’t become a single point of functionality for our LDAP environment. I’ve worked on documenting all the bits and pieces I have about our infrastructure, gave a tech talk to our team about it and will continue involving other members of my team and documenting all the changes. I hope also that this blog post provides some insight into how it is all set up, to give an idea of what is involved with the setup, explain why when you change your password, it takes ten minutes before the wireless controller in San Francisco notices the change, etc.

The next steps:

Overall, I think our LDAP infrastructure is now infinitely better than it was. It is set up to scale now and it is easy to make documented version controlled changes. I’ve put in a lot of time over the past 6 months planning this out, learning LDAP, and it seems that I’ve been eating, sleeping and breathing OpenLDAP for a while now. However, I still don’t consider myself an expert on the subject and am continually learning. I think there are still a lot of improvements that can be made to the infrastructure and what we have now is pretty much a better scaled version of what we had. The basic configuration directives are the same though. There is likely some more tuning that can be done for better performance, more reliable replication and stability. These are all things I want to pursue, but with production services that are so integral to our infrastructure, it is crucial to take things one step at a time.

Special thanks to Rob Tucker, Fred Wenzel, Corey Shields, Phong Tran, Dumitru Gherman, Pete Fritchman, Michael Coates, Guillaume Destuynder, Adam Newman and the rest of the IT team for helping make this happen.

– Jabba

[1] http://www.openldap.org/

[2] http://puppetlabs.com/

[3] http://www.zeus.com/

https://etherpad.mozilla.org/

Here’s a short list of the cool features we’re getting with this upgrade:

• More ports work. You can still connect to the old http://etherpad.mozilla.org:9000/ link, but the :9000 isn’t necessary anymore. The standard port 80 works just as well.
• SSL Support! However you access the site, you’ll get redirected to https://etherpad.mozilla.org/. This is very cool, especially for the “SSL Everywhere” folks.
• “Team Site” functionality. This is huge, and easily the biggest new feature… people have been asking for something like this for quite a while. Now Mozilla is a pretty open organization, but the reality is there are still some things that can’t be publicly discussed right away. A good example is domain name registrations… people have a habit of swiping them out from under us if we discuss a new domain name before it’s registered.
• Team Site Pads can be public or private, and can even have their own password, just for that one pad. Let’s say you’ve got a team pad, and you need to let someone not on your team access it… but only that one pad, not any others. Simply make it a public pad, but set a password on it. Your team members can still access it, and now anyone you give the shared password to can also!
• Team Site Pads can be deleted! This is a common request due to accidental information leaks (passwords, etc). Sadly this doesn’t extend to purely public sites, but it’s still a nice step forward.

Within a couple hours of migrating to this (and on a Friday at 5pm), and despite a bug on the confirmation email preventing it from “just working”, we had 8 different team sites created for various groups… from apps to UX, jetpack to infra. I suspect we’ll see some cross-functional and community team sites eventually as well.

Sadly, there are some bugs still to be worked out, especially in the area of SSL certificates. I’ve created a wiki page, mostly dealing with features and bugs associated with the upgrade: https://wiki.mozilla.org/Etherpad. Feel free to add to it!

On a side note: there’s been talk recently about Etherpad Lite, and it’s definitely something we’re considering. We didn’t go with it this time because 1) most of this work was already done by the time we knew about that (this has been in the works a long time), and 2) Etherpad Lite lacks some of the functionality we’re getting here… specifically the Team Sites. It’s in their TODO list though, so I wouldn’t be surprised if we’re on Lite in the future.

Let us know how the new system works for you! We’d love to get some feedback on it.

- Jake

So, let’s start by talking about bugs.

Whether or not bugzilla is the right tool to track and manage IT projects and requests is up for debate. The benefit to using bugzilla is that it integrates with the rest of the project, since it is used for everything else at Mozilla.  That said, let’s talk about how IT works with bugs and how you can help us when you file bugs:

Please do not assume tribal knowledge in bugs.

In the past 30 days, the IT Systems and Ops teams have grown by 5 new sysadmins.  This is great, and they are all ramping up quickly. While we are throwing out numbers, we have seen 119 new bugs added to the Server Operations component in the past 7 days.  We want our new guys to help out in these bugs as quickly as they can.  When submitting a bug, please assume as little tribal knowledge as possible on the other side.  For instance, asking for a setting change in a production site without telling us which site you work on delays the bug while someone either asks for clarification or has to ask the team what you mean.  These are minor delays of course, but when this happens multiple times a day this becomes very inefficient.  If you have a doc to link to giving background on the request you are making, please do it.  If you know the system you are asking for a change on, please make note of it.

Where does my bug go?

The IT team is growing quickly, as is the need to sort our bugs into components lest we spin our wheels all working from one component.  Here is the layout of our components for bugs coming from you as it stands today (note the change in Web Operations):

• Server Operations: Web Operations – this is where all web related bugs should go.  This is new, and is modified from the old “web content push” component to encompass web server problems, new web projects, and any general request regarding the serving of our websites.
• Server Operations: Desktop Issues – this is where the desktop team currently works. Laptop issues, software license requests, and help with the office environment should all go here.
• Server Operations: RelEng – Any issues regarding the release engineering build systems (aka “the build network”) should go here.
• Server Operations: Netops – Network requests and issues should be filed here
• Server Operations: Labs – Mozilla Labs IT requests go in here
• Server Operations: ACL Request – Firewall requests for Netops
• Server Operations – Everything else that did not fall into one of the above.

Priority and escalation

The default priority for our bugs is “normal”.  We will get to these as soon as we can, and by nature of your request we assume that you want them done as soon as possible.  If this is a request that does not fall under that assumption and you want it to fall under the “nice to have someday” category, mark it as an enhancement. Anything higher than normal demands attention soon.  Our SLA for addressing bugs higher than normal is such:

• Major – 24 hours
• Critical – 8 hours
• Blocker – immediately

These timers work around the clock, and if a bug sits unaddressed beyond those times, our oncall is paged. Blocker IT bugs will page oncall immediately.  We can not guarantee that the request will be resolved within this time (ie: if you file a critical bug for a new cluster of servers, it will take us time to procure them first), but we will have admins aware of it and start working on it.  In addition, we have our own internal prioritization of issues that come in.  If a critical bug in a dev site comes in, that may have to wait for work that we are doing on a production site.

That was a lot to read..

And if you are still with me, thanks for taking the time to understand how we work in bugzilla. By getting bugs filed more efficiently we can spend less of our time refining the bugs and more time fixing them.