noise from a mozilla IT/Operations wrangler
Three years ago we were a team of 7 if I remember right.
Today it’s a team of 38 spanning 6 different groups & 6 different time zones. They are doing things I wasn’t even imagining we’d be doing three years ago. They are, in many respects, the smartest group of people I have ever worked with.
This is Mozilla’s Infrastructure & Operations team, an extremely talented group of sysadmins, network engineers & desktop support folk –
Two weeks ago my entire team was onsite. It’s perhaps the first real IT/Operations onsite we’ve had. The group is finally large enough and geographically spread around where having focused onsites makes sense.
Most of the week was spent with PuppetLabs and for various reasons, I missed the beginning of the week but I had purposely decided to let the teams run the week.
I wanted to wrap up the week with some sort of team event and in reflecting on my teams and what it all means to me, it made sense that we should host a party and invite Engagement and Webdev, two of the groups we interface with the most. As I said in my email to them, they “complete us”.
I prepared a speech too but as things turned out, the venue didn’t really lend itself towards a speech and as the night went on, the vibe didn’t feel like it needed to be interrupted with one.
I spent a lot of time on this and still think it’s worthwhile sharing.
I wrote this in the context of notes to myself, less of something I’d read straight through.
Hard to imagine with a group this large we’d all get along all the time. We’re like three parts of a whole. We don’t function without each other.
Let me give you a couple examples that drove that home for me. Back in March, just before the Firefox 4 launch I started working more with Engagement & Webdev and learned:
- Unflinching Mayumi who wouldn’t take any shit from me pushing back on websites
- Laura’s constant demos.mozilla.org pushes
- I had guys on my team who don’t touch websites doing websites.
- I argued with Jeff over his glow.mozilla.org site (and I let Jeff convince me he was right)
- Yet everyone was aligned
- And then the day we did 10+ pushes, no downtime, no announcements
At the start of Q2 I told my teams, I’m lazy, no more downtime announcements. And you know what? The last post I made for any website push was March 9. I’m not sure how we did it. A lot of if I’m sure was webdev’s work with AMO and dark launches. That I don’t actually know how the teams did it just means they’re awesome.
Let me share just a little bit about what my teams do and what IT does.
IT suits me. I don’t think any of you know this but in high school I was a theatre guy, a thespian. My first show was “Anything Goes” and I ran sound (my last was “West Side Story“). I like to work where I’m noticed the least and were I help others look their best.
That sort of fits IT. You only really know we’re here when things don’t work. But these are the guys and gals who are invisible so we ALL look good.
Of course the downside is that not everyone knows that my entire group was 10 last year and more than 30 today.
I bet no one here even realized we stopped announcing downtimes.
Going back to the “we don’t function without each other” -
I’ve seen my share of disasters here – but we rally. Because we really are just one team.
YearUp's Core Values Award
I met Jay and Sahaar at YearUp and immediately saw possibilities for Mozilla’s participation and by January 2010 we welcomed our first, albeit entirely overdressed, YearUp intern, Justin, and followed that with our second intern, Henry, in August.
Going into 2011, Mozilla increased it’s participation with YearUp and took on three interns.
Three semesters (and we’re doing a fourth), five different students. Yet what stands out is their utter determination and strength to rise above whatever set of cards life handed out, hardship and opportunity divide be damned; their persistent drive to excel and their eagerness to soak up knowledge and wisdom from not just me but from their peers at Mozilla.
No doubt a lot of credit goes to YearUp. No doubt a lot more credit goes to the individual student. Very little goes to me – they’ve already plotted their own course – I’m only here, if anything, as a part-time teacher.
I’ve seen Justin and Henry step into roles I never thought they’d be in when they started. Both have become leaders in their own rights. Both continue to be advocates for YearUp within Mozilla, mentoring where I don’t have the faculties to do so.
And part of that defines what Mozilla is to me. It was Mozilla, after all, that rose against seemingly insurmountable odds to become a dominant web browser and an agent of change.
I recently learned that I had been recognized with YearUp’s Core Values Award, an award that recognizes those that embody YearUp’s principles; principles that are at the foundation of everything they do.
This isn’t an award I can accept on my own though.
It’s only an award I’m able to achieve because I have the privilege and luxury to stand on shoulders of giants who have helped me, who have mentored me, who have unknowingly shaped my own life (and whose numbers exceed my ability to adequately thank).
It is because of the way Mozilla has shaped and molded me over the past five years that I’m able to be thought of as “living YearUp’s Core Values”, the ability to think less of the individual importance of self but the overall success of the whole, of the Project and of the Community (and ironically success of the latter tends to bring the former).
I’m humbled to even be considered for such an award let alone be privileged to work with YearUp. That I’ve had any part in shaping another human’s trajectory in life is mind numbingly humbling (and not something I ever specifically set out to do).
I’ll leave with a few words from an email I sent after attending my first YearUp graduation where Mozilla was presented with YearUp’s Urban Empowerment Award:
It is with a sense of awe and a feeling of pride to be involved with YearUp.
I’m posting this in the hopes others who find themselves in this predicament will find this online and find it useful. I certainly found comfort in this guy’s tale.
I took a side trip to Hungary this past weekend and inadvertently lost my passport.
This all happened on a Friday night. Thank God for a smart phone. And data roaming be damned. I quickly googled for the embassy in Budapest and called the emergency after hours number. After declaring myself an American citizen, my call was escalated to the oncall duty manager.
Here’s where definitions of emergency differ. I has a flight out of Budapest Sunday morning. This was a “blocker” for me. The US Embassy in Budapest is closed over the weekends (even for emergencies). Not an emergency to them.
The best the duty officer could offer me was to arrive Monday morning, identify myself as an American and I’d be escorted in to get a temporary passport.
I tweet’d looking for help. I crowd sourced getting help. You have no idea how helpful that alone was. (thanks everyone!) I had people sending me DMs and text messages and replies to my tweet. It felt good to know I had this network of people willing to help me.
Saturday I retraced my steps (no luck) and moved my return flight to the last possible one on Monday. It was mentally really hard to have any fun the rest of Saturday.
Sunday. Budapest is hot, hot like Yucatán hot. Budapest is also a very walkable city. However, because of the heat I spent most of of my time going from one free wifi coffee shop to another. I also scouted out my embassy.
Monday morning. Embassy opens at 9am. I’m up at 7a, dressed and fed and out by 7:45a. Way ahead of schedule. I got to the embassy at around 8:15a and told the guard I had lost my passport and showed him my California drivers license. He disappeared for a bit and then opened the gate for me. He told me to leave my bag with him, it’ll be easier to get in (made sense to me – I wanted as little drama as possible).
Currently, my most valuable possession.
I had to empty my pockets and literally turn off my phone before going through the metal detector. The guard there put all my belongings except for my ID and money into a box.
I grabbed a ticket and waited a few minutes until hey called my number. I had to fill out a passport application & lost passport form, get passport photos ($5.41), pay $135 for a new passport and wait 20 minutes for them to print out my passport.
At least I had alternate ID. The gentleman behind me had nothing but a copy of his passport which didn’t seem to be of any use. I talked to him a bit. Same thing, lost his passport and everything else he had in his “pouch”. At least my stuff is all separate.
Total time in embassy, 1:19. Didn’t have the patience for the metro and took a taxi to the airport.
So basically a huge inconvenience. Meant having to change a number of flights around (not free). Meant staying longer than I had packed for or planned to in Budapest.
Also, screwed my work schedule. Also, expensive mistake.
Souvenier from Budapest.
If you’ll indulge me,
Single points of failure suck. I can’t help thinking that this whole passport concept is a single point of failure. I lost it and was screwed. Never mind that I had a couple credit cards and a California drivers license with me -and- a color copy of my passport.
I also have a this biometric data that’s physically attached to my body and REALLY hard to lose. I felt like I had all these tools to conclusively prove who I am and some computer could verify I was okay to fly.
I learned long ago that home is wherever my stuff is. My stuff – laptop, luggage – was in Paris. I’m back in Paris, still far from my home but I can’t tell you how much this feels like home!
This came up the other day and it occurred to me that not everyone knows why I call myself what I call myself.
Shortly after moving out to Mountain View in 1996, I worked at 3Com. I worked in the engineering division that, during the two years I worked there, went by names such as NSD and ESD. This was the division that made “brouters” and if memory serves, largely came from Bridge Communications. I started there as a Solaris syadmin and left as a network engineer (and didn’t really look back).
Anyways, username convention was your first, middle and last initial. mrz stuck. Also, it’s half as long as my first initial and last name.
(Bonus points if anyone knows my middle name without using Google.)
When I started, my name tag read:
It was at this point I decided I would only go by matthew – I am neither a doormat nor a welcome mat.
Nowadays, I answer to both, but often correct to the preferred. Which you use tends to indicate how well you know me.
matthew and not with a capital M?matthew@interaccess.com), which was in all lower case.
That stuck. So did the fixed-width font. It’s weird, I know.
My first name comes from my great grandfather’s middle name and this first century Galilean. My middle name comes from my grandfather’s first.
Musica cerca de Av. Corrientes y Av. 9 de Julio.
Last week, for eight days, I was able to step outside my normal role managing Operations and wear an entirely different hat.
I had an amazing opportunity to interact with the vibrant Mozilla and Open Source community; I got to interact with those who I help from the shadows every day.
I’ve had a couple days to let my thoughts soak in.
Something about Argentina and Buenos Aires resonated with me in a way that’s hard to describe. It is, perhaps, the first time in my life I’ve had a sense of reverse home sickness. From San Telmo to La Boca Caminito to Palermo to Recoleta Cemetary and Calle Florida, Buenos Aires oozed of culture. From pizza to empanadas to more gelato than I can remember, it’s unlike anywhere else I’ve ever traveled.
Persicco, best gelato around.
The bustling energy of San Telmo’s Feria de Antigüedades was matched only by the energy of the open source community I met.
San Telmo’s Feria de Antigüedades
On Wednesday night I had the pleasure of attending the Firefox 4 Party. This was amazing. The night before I met with several of the event organizers, including Guillermo Movia. They were expecting 50 or so and instead had 150 at the party. I don’t know how to really describe what it was like, walking around and mingling with everyone (in my broken Spanish no less), hearing everyone talk about Firefox and Mozilla.
Thursday night we attended the first Hacks/Hackers MeetUp in Buenos Aires at AreaTres. The discussion was all in Spanish but I mostly kept up. I was amazed at the turn out. Was a far larger group than I would have imagined and made me realize how large the open source community in Buenos Aires is.
Hack/Hackers MeetUp
Friday wrapped up with a Design MeetUp at Urban Station that Tara led. This turned out to be one of the surprise highlights mostly because of the discussion afterwards. I don’t often get to interact with the community in such an intimate venue and speak Mozilla.
You can take me out of networking but you can’t take networking out of me. At each place we went to I’d always check to see who I had upstream connectivity from and what my path to Phoenix or San Jose looked like. Urban Station had the quickest Internet I had experienced while in Buenos Aires.
La Boca Caminito, gracias a Benito Quinquela Martín.
On a personal side, since all of these events were after 6pm local time and I was shifted 4 hours off California, I found a lot of time to explore and soak in Buenos Aires. I walked more than I can remember, slowed down more than usual to look and listen. Ate. Indulged. Walked. Explored. Saw a ballet show at Teatro Colón. Went to a Tango show. Walked to Carlos Gardel’s house (Casa Museo Carlos Gardel) in Abasto. Inadvertently walked to Palermo and had mint iced tea. Bought a crappy umbrella and walked in the rain.
No doubt I was lucky to have a fantastic travel companion (I’ve thanked you, haven’t I Tara?).
Lastly, I want to share this:
I went with very little expectations and maybe a little nervous anticipation. I came back with a profound sense of Mozilla, of the community that supports Mozilla and a feeling of renewed purpose for why I work at Mozilla. I came back with more friends than I left with, with a twitter feed half in Spanish.
John Lilly used to talk about about great companies vs good companies. How great companies last; they may change but their mission remains. Mozilla, he argued, was on a path to be a great company. Today, the vehicle for Mozilla’s mission is Firefox. Tomorrow it could be something else. But the mission will remain.
This is the sense of Mozilla I was left with when I landed in San Francisco. The emotional connection people make with Mozilla, and more precisely, its Mission, is what will make Mozilla one of the great companies.
I’ll leave you with a couple pictures I took. Tara did a better job taking photos than I did – you should check out her Flickr gallery.
Community Events
Mozilla has a long history of being a technology leader on the web and open standards. A short, and certainly not exhaustive, list includes:
IT/Operations is often the unglamorous side of Mozilla. We work in the shadows and you generally don’t see us unless something’s broken. However, we want to push technology forward too. We like to be thought leaders. We like to solve the “chicken-and-egg” problem. For example, over the summer of 2010, we laid the infrastructure ground work for DNSSEC and published a signed mozilla.org zone.
This year we’re ready for another challenge and are excited to join others in support of the World IPv6 Day! June 8, 2011 will be the first global-scale “test flight” of IPv6 and we’re going to be a part of it. This will also be our chance to see which parts of the network already support IPv6 and which parts require software (or hardware) upgrades.
As part of World IPv6 Day, we’ll offer a couple marquee Mozilla web properties over IPv6. We’ll be able to measure how many users are already dual-stack (both IPv4 & IPv6) vs. those who don’t yet have IPv6 capabilities. In short, we’ll have a view on the state of the Internet’s IPv6 end-user deployment.
What is IPv6?
IPv6 is the next generation protocol for the Internet. The current protocol, IPv4, is based on 32-bit addressing and has the familiar syntax of63.245.209.10. Unfortunately, after more than 30 years, the Internet is running out of available IPv4 address space.IPv6 (and it’s 128-bit addressing) provides a vast increase in available address space – 4 billion times the number that are available under IPv4. Addresses take on an entirely new syntax –
2620:101:8003:200:217:f2ff:fe09:d8ea, for example.
When I started at Mozilla in 2006, I continued the IPv6 work I was doing at my last job and for some time now Mozilla’s had a production v6 network announced out of our San Jose data center. I even experimented with a couple websites back in 2007. Unfortunately, there’s been a lack of user demand and, as such, the “chicken-and-egg” problem.
We recently traded in our original /48 allocation in exchange for a /44 and are slowly renumbering and bringing up native IPv6 peering with Mozilla’s transit providers. My Network Engineers even pushed out native IPv6 connectivity within Mozilla’s Mountain View office.
Those interested in tracking our progress can follow along in bug 630581.
I hope you’ll follow along with us in the coming months and join us for World IPv6 Day on June 8, 2011!
June 8, 2010.
We’ve been tracking this date since Thunderbird 3.0 was released (December 8, 2009).
We stop supporting Talkback (the crash reporting tool that was used in Firefox 2.0.0.x, Thunderbird 2.0.0.x, and Camino 1.5.x) today!
Talkback’s long since been replaced by Breakpad.
We will officially turn things off June 15 during our normal Tuesday night maintenance window. This should give us enough time to mourn communicate this to the broader community.
(Of course, if there’s any fathomable reason we shouldn’t decommission Talkback, by all means leave me comments.)
Mark your calendar. Saturday, April 17 @ 2pm.
That’s the date we’ve scheduled to do some significant NetApp upgrades in San Jose (bug 502151).
Unlike most NetApp upgrades, these will require real downtime. The nature of the upgrade involves updating the firmware on several disk array shelves and requires a power cycle of those shelves.
This will affect a large swath of the infrastructure, including:
bugzilla.mozilla.orgaddons.mozilla.org (and related AMO sites)aus2.mozilla.orgdeveloper.mozilla.orgwiki.mozilla.orgBasically a lot.
Why now? Honestly, we held off as long as we could.
About a year ago the filers started reporting a fault on all the attached disk shelves:
Jan 1 23:00:00 netapp-a [meer-netapp-a: monitor.shelf.fault:CRITICAL]: Fault reported on disk storage shelf attached to channel 0a. Please check fans, power, and temperature.
For those who like to play along, NetApp bug #258741.
“A bug in the ESH2 I/O modules firmware can lead to a I2C buffer loss which can cause the two modules in a shelf to stop communicating with each other.”
NetApp goes on to say,
“This bug does not cause any issues with data availability and the system will continue to function in the presence of these errors. This bug will cause the shelf to incorrectly report status, statistics, and environmental information.The error condition caused by this bug can be cleared by power-cycling the shelf. This should not be attempted while the system is online.”
This was basically a cosmetic error and not a real fault. Since it wasn’t affecting service we put it in the back of our mind and figured we’d address it after the Firefox 3.5 release in June.
June came by and other things were more important and before you knew it, it’s autumn and we decided to push it off until the holidays when the demand on the infrastructure is a lot less. In fact, we had originally planned to do this the same time we did an OS upgrade on another pair of NetApp filers.
I decided to delay it again until we had the Phoenix data center online and able to take production traffic.
And that’s where we are now. April 17 @ 2pm we’ll take a 3-hour window to do the OS & firmware upgrades.
During the week of April 12 we’ll manually fail over a number of production sites that service in-product features (support.mozilla.com, Personas) and support product downloads (aus2.mozilla.org, download.mozilla.org, addons.mozilla.org) to Phoenix as their primary data center.
I’m posting this well in advance of April 17 because of how much of the infrastructure this will impact. The date’s somewhat immovable and I realize this will impact many different parts of Mozilla. Feel free to comment here or in the bug with any concerns.
[root@ip-ns01 ~]# mtr www.mozilla.com --report ip-ns01.phx.mozilla.org Snt: 10 Loss% Last Avg Best Wrst StDev 10.8.75.1 0.0% 0.4 0.4 0.4 0.4 0.0 v500.core1.phx.mozilla.net 0.0% 1.1 1.3 1.0 3.1 0.6 xe-1-1-0.border1.phx.mozilla.net 0.0% 0.7 0.7 0.7 0.8 0.0 64.124.201.177 0.0% 1.1 1.1 1.1 1.1 0.0 ge-0-3-0.mpr3.lax9.us.above.net 0.0% 9.4 13.0 9.4 44.3 11.0 xe-0-1-0.er1.lax9.us.above.net 0.0% 9.5 13.7 9.4 51.4 13.3 xe-0-1-0.mpr1.lax12.us.above.net 0.0% 94.3 21.3 9.3 94.3 27.8 xe2-3.cr01.lax01.mzima.net 0.0% 10.0 14.0 10.0 23.0 4.6 xe1-0.cr01.lax02.mzima.net 0.0% 16.9 15.7 10.2 22.8 4.8 te1-3.cr02.sjc02.us.mzima.net 0.0% 18.1 22.5 18.1 30.0 4.9 ge1-mozilla.cust.sjc02.mzima.net 0.0% 18.4 18.6 18.4 19.1 0.2 v8.core2.sj.mozilla.com 0.0% 18.4 19.7 18.2 30.8 3.9 mozcom.acelb.sj.mozilla.com 0.0% 18.5 18.6 18.4 19.5 0.3
