http://blog.mozilla.com/justin Mozilla IT/Operations...in brief Thu, 20 Mar 2008 23:39:56 +0000 http://backend.userland.com/rss092 en We had a network outage at our San Jose datacenter tonight from 8:01 pm PDT until 9:25 pm PDT on March 18. From initial investigation, it appears that one of the switches in a blade server chassis had a software issue, causing a network-wide broadcast storm. Overall effect ... http://blog.mozilla.com/justin/2008/03/20/network-outage-report-31808-801pm-pdt-925-pm-pdt/ One Mozilla's biggest assets is our mirror network. It allows us to update over 100 million users in under 48 hours with security updates, host and push extensions, and much more - all with donated server space and bandwidth, giving us the ability to focus our efforts on supporting the ... http://blog.mozilla.com/justin/2008/02/19/call-out-for-mirrors/ As you may know from my last post on Bugzilla, a lot of improvements/fixes are in the works. Wanted to give everyone an update on what has gone live on bugzilla.mozilla.org so far: - Send mail in the background after confirming to the user instead of waiting for the ... http://blog.mozilla.com/justin/2008/02/12/bugzilla-improvment-project-update/ Bugzilla basically runs Mozilla - it's core to almost everything we do from tracking core Firefox bugs to tracking Marketing events to operating as our IT ticket queue. Quite simply, we wouldn't be who we are today without it. With all its greatness, there are quite a few ... http://blog.mozilla.com/justin/2007/12/05/bugzilla-improvments/ I was reminded of the power of open source software yet again this weekend. A little background: We here at Mozilla are big fans of OpenVPN. When we rebuilt our datacenter, we did a large search for the right VPN solution. Mozilla's requirements were somewhat specific: * Had ... http://blog.mozilla.com/justin/2007/10/13/open-source-for-the-openvpn-win/ We recently ran an experiment, code named funnelcake (see polvi's blog post for more details) - this was an interesting project from IT's perspective for a few reasons. First a little background - for one 24 hour period, we would need to serve *all* en-US and de downloads which ... http://blog.mozilla.com/justin/2007/10/04/go-go-gadget-funnelcake/ I'm about mid way through my first trip to China (in Beijing) - first time to the far east for that matter, and I have to say it's a pretty interesting place. I've been all over europe and north america, but what has really struck me is how Beijing ... http://blog.mozilla.com/justin/2007/08/28/china-similar-to-you-and-me/ As many of you may know, we released a major update to offer 1.5.0.12 -> 2.0.0.4. This is significant to the infrastructure for a few reasons. First off, all of these updates will be *full* updates, i.e. full browser downloads - no 300k mar file. That puts ... http://blog.mozilla.com/justin/2007/06/29/yes-sir-may-i-have-another-update/ Recently there have been a lot of cube moves at MoCo headquarters - IT included. We are back upstairs in building K, but the sun was creating an issue for a few of my guys. Enter Cosco to the rescue: Kinda like their own IT oasis. Umbrellas make ... http://blog.mozilla.com/justin/2007/05/15/umbrellas-make-people-happy/ Not sure if people noticed given how well the team has been doing, but I've been a bit hard to reach for the past few weeks. The reason is there is a new addition to the Fitzhugh Family - Ryan Mason Fitzhugh born on 4/22/07 at 9:15am. The ... http://blog.mozilla.com/justin/2007/05/06/new-co-pilot/