Mozilla Labs Test Pilot is a project to experiment and analyze data from real world Firefox users to discover quantifiable ways to improve our user experience.  I was very interested and excited about the project because of the care they take to protect the user’s privacy.  They have a very user focused privacy policy that is easy to read, which always makes me happy.  Every step of the way they make sure the user is aware and comfortable with the data they are sending by making it easy for the user to see their data before they submit it and providing the user the choice to submit it or not when the data is ready. The data is always very general in nature, not containing any sensitive information like URLs and it is not associated with any personally identifying information at any time.

In the pre 1.0 releases of Test Pilot, the data that is submitted from the add-on is received by a simple script transforms the data into a flat file that is stored on an NFS server.

We are planning on making a huge drive to ramp up the volume of users and the number of experiments, and that means that this simple storage mechanism will not survive.  Here are some of the most important requirements we’ve hashed out in our planning:

• Expected minimum users: 1 million.  Design to accommodate 10 million by the end of the year and have a plan for scaling out to tens of millions. (This is the 1x 10x 100x rule of estimation of which I am a fan)
• Expected amount of data stored per experiment: 1.2 TB
• Expected peak traffic: approximately 75 GB per hour for two 8 hour periods following the conclusion of an experiment window.  This two day period will result in collection of approximately 90% of the total data.
• Remain highly available under load
• Provide necessary validation and security constraints to prevent bad data from polluting the experiment or damaging the application
• Provide a flexible and easy-to-use way for data analysts to explore the data.  While all of these guys are great with statistics and thinking about data, not all of them have a programming background, so higher-level APIs are a plus.
• Do it fast.

I am a technology nut.  I love to research technologies to keep abreast of the state-of-the-art and also potential tools.  While I’ve always been a SQL aficionado, I am also a big fan of the “NoSQL” technologies because I feel there is a great role that they serve.

When I looked at the characteristics of this project, I felt that a key-value or column-store solution was the best fit, so I started digging through my research bookmarks and doing some technology cost/benefit analysis.

Eventually, our team came down to three primary contenders:

• HBase
• Cassandra
• Riak

We recently had a meeting wherein we hashed out a lot of the pros and cons of each of these solutions.  I wanted to share that discussion with everyone, not because I was looking forward to being set-upon by the two contenders that I didn’t feel were the best fit, but rather for two reasons:

1. Crowd-sourcing — I believe that laying out the thoughts and assumptions in the open is the best way to ensure that we receive the broadest set of feedback from the experts in each of the varying technologies.  I further believe that it is better to be aware of the over-looked features and warnings raised by these experts and consider what can be done to mitigate them rather than hiding from them.
2. Sharing of knowledge — Even if it turns out that we didn’t get all the answers right or that we didn’t come up with the ideal solution, I believe that we asked a lot of good questions here and I believe that listing these questions might help some other team who has to make a similar decision.

So let’s get down to the discussion points:

• Scalability — Deliver a solution that can handle the expected starting load and that can easily scale out as that load goes up.
• Elasticity — Because the peak traffic periods are relatively short and the non-peak hours are almost idle, it is important to consider ways to ensure the allocated hardware is not sitting idle, and that you aren’t starved for resources during the peak traffic periods.
• Reliability — Stability and high availability is important.  It isn’t as critical as it might be in certain other projects, but if we were down for several hours during the peak traffic period, the client layer needs to be able to retain the data and resubmit at a later date.
• Storage — Need enough room to store active experiments and also recent experiments that are being analyzed.  It is expected that data will become stale over time and can be archived off of the active cluster.
• Analysis — What do we have to put together to provide a friendly system to the analysts?
• Cost — Actual cost of the additional hardware needed to deploy the initial solution and to scale through at least the end of the year.
• Manpower — How much time and effort will it take us to deliver the first critical stage of the project and the subsequent stages?  Also consider ongoing maintenance and ownership of the code.
• Security — Because we will be accepting data from an outside, untrusted source, we need to consider what steps are necessary to ensure the health of the system and the privacy of users.
• Extensibility — delivering a platform that can readily evolve to meet the future needs of the project and hopefully other projects as well.
• Disaster Recovery / Migration — If the original system fails to meet the requirements after going live, what options do we have to recover from that situation?  If we decide to switch to another technology, how do we move the data?

Now we iterate those points again, but this time we have the points made by the team regarding each of the three solutions being considered:

• Elasticity– Machines can be added as load increases.  Machines can be turned off and reconfigured to remove them. There is the ever-present the risk of a bug resulting in a lack of replication or corruption causing data loss.  In all three solutions, re-balancing the existing data incurs an additional load penalty as data is shifted around the cluster.  We need to consider how much time and manual administration is required, how much can be automated, how risky rebalancing is, and how long until we begin to see the benefit of the additional nodes.
  • HBase
    In HBase, the data is split into “regions”.  The backing data files for regions are stored in HDFS and hence replicated out to multiple nodes in the cluster.  Every RegionServer owns a set of regions.  Normally, the RegionServer will own regions that exist on the local HDFS DataNode.
    If you add a new node, HDFS will begin considering that node for the purposes of replication. When a region file is split, HBase will determine which machines should be the owners of the newly split files.  Eventually, the new node will store a reasonable portion of the new and newly split data.
    Re-balancing the data involves both re-balancing HDFS and then ensuring that HBase reasses the ownership of regions.
  • Cassandra
    In Cassandra, nodes claim ranges of data.  By default, when a new machine is added, it will receive half of the largest range of data.  There are configuration options during node start-up to change that behavior.  There are certain configuration requirements to ensure safe and easy balancing, and there is a rebalance command that can perform the work throughout all the data ranges.  There is also a monitoring tool that allows you to track the progress of the re-balancing.
  • Riak
    In Riak, the data is divided into partitions that are distributed among the nodes.  When a node is added, the distribution of partition ownership is changed and both old and new data will immediately begin migrating over to the new data.
• Cost — Regardless of solution, we should be able to use commodity server hardware with Linux OS.
  • HBase — Because of the heavy peak traffic periods, it is very likely that we would need a dedicated cluster. Otherwise, other projects such as Socorro might be negatively impacted.  Also, a scheduled maintenance window would affect both projects instead of just one.
    HBase is memory-hungry.  Our current nodes are dual quad core hyper-threaded boxes with 4TB of disk and 24 GB of memory.  It is unlikely that we would want to go less than that.  We would need at least two highly available master nodes, and by the end of the year we’d likely need 12 machines for a single cluster solution.
  • Cassandra — Much lighter on the memory requirements, especially if you don’t need to keep a lot of data in cache.  We would likely want to double the amount of CPU on the four nodes currently allocated to the Test Pilot project. We’d also want to order 8 more machines.  To perform analysis with Cassandra, we’ll have to leverage our Hadoop cluster.
  • Riak — Also much lighter on memory requirements.  The existing four nodes (quad core 8 GB) allocated for the project should be enough to kick it off, and we’d expect to add at least two more equivalent machines to that cluster.  We’d also set up a second cluster of 6 to 8 less powerful machines for the analysis cluster.  Because of the elasticity of Riak, we could temporarily re-purpose N-3 of those machines to the write cluster to accommodate expected peak traffic windows.
• Manpower
  • HBase — Need a front-end layer to accept experiment submissions from the client.  The fewer changes required for the client, the better.  Thrift or a roll-our-own Java are the two most likely options.  The application needs to be heavily tested for capacity and stability.  Likely two weeks for development and two weeks for testing.  Estimate is dependent on the amount of security code, sanity checks, and cluster communication fail-over that has to be implemented.  Additional maintenance burden of supporting a separate service.
    Schema design needs to be reflected in the front-end code to allow data to be parsed out and stored in the proper column families.
  • Cassandra — Mostly the same as HBase. Thrift or Java application hand developed and tested.  Schema design to accommodate storage by the front-end.
  • Riak — Built in REST server.  Already heavily tested and production ready.  Minimal schema design and no specific hooking in of the schema to the REST server should be needed.
• Security — We can’t expect to hide any sort of handshake protocol or authentication token.  If we wanted to require an authentication token, extensive changes would have to be made to the client add-on which would delay the project.  SSL doesn’t seem to gain us much because we aren’t transmitting potentially sensitive data, and it has overhead penalties.  Our firewall and proxy/load-balancer layer is our most important line of defense.  It should reject URL hacks, unusual payload sizes, and potentially be able to blacklist repeated submissions from the same IP.  Ideally, if the payload inspection could communicate IP addresses or payload signatures to blacklist, we’d be pretty well equipped to prevent degradation of the cluster health.
  • HBase/Cassandra — We would need the custom built front-end layer to be responsible for inspecting the payload to look for invalid/incomplete data and reject it.  This adds to the requirements and implementation time of the custom front-end layer.
  • Riak — We can use Webmachine pre-commit hooks to allow inclusion of business logic to perform payload inspection.
• Extensibility — When changes are made to the data stored, all three solutions will potentially require modification of the payload inspection routines and potentially the analysis entry-point to reflect the schema changes.
  • HBase — Schema changes involving adding or altering column families require disabling the table. This means a maintenance window.  Creation of new tables can be performed on the fly.
  • Cassandra — Schema changes require a rolling restart of the nodes.
  • Riak — New buckets and schema changes are completely dynamic.
• Data Migration — All three solutions make it pretty easy to replicate, export, or MapReduce data out of the system.
• Disaster Recovery  — In all three solutions, it would be best for the client add-on to have enough intelligence to be able to back-off if the cluster load is too high, and to retry submission later if it fails.
  • HBase — Custom front-end could incorporate fail-over code to locally spool submissions until cluster is back online.  A second cluster would be the most viable DR option.
  • Cassandra — Same as HBase
  • Riak — Could temporarily reassign the entire reporting cluster to handle incoming submissions. Because there is no custom front-end, if we were unable to make the Riak cluster available for client connections, we would have no buffer in place on the server side to spool submissions.
• Reliability — Small periods of downtime should not be a major issue, especially if the client add-on has retry capability and/or if the front-end layer can spool.
  • HBase — Until subsequent versions provide better High Availability options, the Hadoop NameNode and HBase Master are still a single point of failure.  Certain types of administration and upgrades require restart of the entire cluster with a maintenance window required to modify the NameNode or HBase Master.  Rolling restarts are an option for many types of maintenance, but some HBase experts discourage them.
  • Cassandra — No single point of failure. Most configuration changes can be handled via rolling restarts.
  • Riak — Same as Cassandra.
• Analysis
  • HBase — Can provide a HIVE based interface (possibly with JDBC connectivity).  Can provide a simplified MapReduce framework to allow analysts to submit certain types of common, simple jobs.
  • Cassandra — Uses Hadoop, answer same as HBase.
  • Riak — Map Reduce jobs can be written in JavaScript and submitted through the REST API.  A light-weight web interface can be created to allow submission of those jobs.

Based on the evaluation of these discussion points, and also on the availability of some Basho experts to deliver a nearly turn-key solution, we have decided to go with Riak for the implementation of the Test Pilot back-end.  While it feels a little odd to be using a technology that is similar in many ways to HBase which we are investing heavily in, I think it is the best choice for us and I actually see several areas that we could potentially consider using Riak for other projects.

If you have any questions, concerns, or clarifications, please feel free to submit them as comments and I will respond or update the post where applicable.

I was presented with the challenge of answering the question – how many Firefox users have one add-on or more installed on their Firefox. Currently, addons.mozilla.org (AMO) has statistics on the download counts of add-ons but the actual usage of add-ons has been unanswered.

The Add-ons manager inside of Firefox will check each add-on for update at AMO. This happens once every 24 hour period when Firefox is ran by the user. Updates are handled over HTTP at either addons.mozilla.org for Firefox1.0/1.5/2.0 and versioncheck.addons.mozilla.org for Firefox3.0/3.5. The add-ons manager will ping the servers with information about each add-on and if an update exists the server will respond with one. Since the update ping is handled in HTTP the update ping is recorded in a log file. If you have never seen a web server’s HTTP log file they simply are flat text files where each line contains information about the requests made to the server. Below is example line of the AMO log file and an explanation of the fields.

IP                  HOSTNAME                             TIMESTAMP          REQUEST
 255.255.255.255 versioncheck.addons.mozilla.org - [22/Jun/2009:02:00:00 -0700] "GET
/update/VersionCheck.php?reqVersion=1&id={B13721C7-F507-4982-B2E5-502A71474FED}&
version=2.2.0.102&maxAppVersion=3.*&status=userEnabled&
appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=3.0.11&appOS=WINNT&
appABI=x86-msvc&locale=en-US HTTP/1.1" 200 520 "-"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11(.NET CLR 3.5.30729)"

We choose the log files on 2009/06/22 because Firefox will ping AMO multiple times after a Firefox version update and this date was 11 days after the Firefox 3.0.11 release and 3 days after Firefox 3.5 RC2 release so most users should have already of been up-to-date by this time. The whole day’s worth of log data for both hostnames total out to be around 28GB compressed. The log files were large because they also contained requests for AMO’s website.

There is no unique identifier to determine which update pings came from which user and we had to rely on identifying pings from a single user by the IP address, and Timestamp in the update ping. The IP address is the single most unique identifier due to the nature of IP addresses but because of routers (NAT) and proxies many computers can sit behind one IP address. To add another degree of separation we decided to group the update pings by the timestamp of the ping. Update pings will happen within a few seconds from each other so pings in a certain time window would be considered as one user, and other pings from an IP address out side of this time window would be considered as a different user. For example, say that there are two Firefox users behind a router. User 1 might open up his browser in the morning at 10AM and ping AMO and User 2 might open up his browser in the afternoon and ping AMO at 2PM. Even though the pings are from the same IP address the pings at 10AM would all be grouped together and counted separately from the pings that happened at 2PM which would also be grouped together.

Setup/Config

Upon hearing about the description of the problem I thought that this would be an ideal candidate for a MapReduce job. I would be able to Map the IP address as the Key and all the other data in the log file entry as a HashMap for the value. I talked with my manager and found out that this was a technology that the metrics team was interested in exploring and was given 4 mac minis to test my implementation out on. I quickly began setting up my Hadoop Cluster running on ubuntu 9.04 desktop which soon became ubuntu 9.04 server for memory conservation and unnecessary gui (Ubuntu 9.04 Desktop takes up about 256MB of RAM on a clean install while Server takes up about 90MB of RAM on a clean install). I have been a user of hadoop in the past but I had never setup my own hadoop cluster before so I turned to the hadoop website and this blog post to help me out. These tutorials at the time did not exactly cover the hadoop version I was using, 0.20. (I believe they are up-to-date now). But was able to accommodate for this fact. The main two things different in hadoop version 0.19 than in version 0.20 is the configuration file and parts of the hadoop java API. In version 0.20, /conf/hadoop-site.xml was split in to three parts in /conf/core-site.xml, /conf/hdfs-site.xml, and /conf/mapred-site.xml and the API was re-factored slightly and some classes were depreciated.

My Mac minis were given hostnames of hadoop-node1-4. Hadoop-node1 was my master node that ran the NameNode and the JobTracker while hadoop-node2-4 were my slaves that ran the TaskTracker and the DataNode.

 # /conf/masters
 hadoop-node1.mv.mozilla.com

 # /conf/slaves
 hadoop-node2.mv.mozilla.com
 hadoop-node3.mv.mozilla.com
 hadoop-node4.mv.mozilla.com

 # The java processes
 hadoop@hadoop-node1:/usr/local/hadoop/conf$ jps
 15778 Jps
 30059 NameNode
 30187 SecondaryNameNode
 30291 JobTracker

 hadoop@hadoop-node2:/usr/local/hadoop$ jps
 16950 TaskTracker
 16838 DataNode
 20186 Jps

After getting the cluster setup I tested it out with the hadoop wordcount example and validated the results.

MapReduce

I then began writing a MapReduce Job with the Hadoop Java API. My first thought was to write my own RecordReader which is responsible for reading from an input split that would give key, value pairs to the mapper but decided to go with the default LineRecordReader which puts the file offset as the key and the line as the value because it seemed easier and more natural to have the log file line dissected inside of the Mapper’s map function.

Map

In the map function each line went through a regexp that broke each piece out of the log file line. If the line contained Firefox’s appid, and VersionCheck.php I would map the IP as the Key and construct an AddonsWritable (which is a child of MapWritable with an overridden toString() for output purposes) that contained the epoch time (converted from the date timestamp because it would be much easier to compare with), a MapWritable of add-on guids, and a count of the number of add-ons.

public static class IPAddressMapper extends Mapper{
     /* member vars for mapper which include vars for regexp and storing data */
     private AddonsWritable logInfo = new AddonsWritable();

     public void map(LongWritable key, Text logLine, Context context) throws IOException, InterruptedException {
         if(logLine.matchesRegexp() && isFirefox() && hasVersionCheckphp()) {
             logInfo.put(EPOCH, epoch); // store the epoch_ts
             logInfo.put(GUID, guid); // store the guid
             logInfo.put(TOTAL, ONE); // store the count
             context.write(ipAddress, logInfo); // map out the ipAddress as Key and logInfo as value
         }
     }
 }

Reduce

After the map the Hadoop Framework hashes keys and gives them to the reduce function. Inside the Reducer’s reduce function you are given the Key which is the IP address and an Iterable of the AddonsWritables that were from the same key/IP. I needed to group the values with update pings in a certain time window together and unfortunately the Iterable does not guarantee order. So I put the MapWritables in a PriorityQueue with a custom comparator that ordered values by the timestamp field in the AddonsWritable. Some IP’s had thousands of pings, so if I counted an IP with more than 2,000 pings I threw it out. Once all the values were placed in the PriorityQueue, I iterated over the priorityQueue popping off each value and comparing to the previous seen timestamp. If the abs(current timestamp – prev timestamp) <= 10 secs I considered them to be from the same user, and added them to a MapWritable of guids inside of the MapWritable that contained all the other information. Once I saw a current timestamp where the abs(current timestamp – prev timestamp) > 10secs I wrote/collected the previous values and started a new MapWritable for the next new window. Until there were no more values in the priorityQueue and I would write/collect the final current values.

public static class IPAddressEpochTimeReducer extends Reducer {
     private PriorityQueue pq = new PriorityQueue();
     public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {
         for(AddonsWritable val: values) {
             pq.add(new AddonsWritable(val));
             if(pq.size() > 2000) {
                 pq.clear();
                 return ;
             }
         }

         while(!pq.isEmpty()) {
             AddonsWritable val = pq.remove();
             if(lastEpoch != -1 && Math.abs(lastEpoch - currentEpoch) > SIXTY_SECONDS) {
                 writeOut();  // Write out all the information for the current collection of versioncheck pings
                 resetVars();  // Reset all the currently used vars for the next collection of versioncheck pings
             }
             addGuid(output, val.get(GUID));
             sum += val.get(TOTAL);
             lastEpoch = currentEpoch;
         }
         /* There is one more remaining.  Write it out */
         writeOut();
     }
 }

Runtime Stats

Hadoop provides a web interface that will output runtime statistics for a job, below are the stats for the job described above.

• Submitted At: 17-Jul-2009 09:55:12
• Launched At: 17-Jul-2009 09:55:12 (0sec)
• Finished At: 17-Jul-2009 14:05:52 (4hrs, 10mins, 39sec)
• Average time taken by Map tasks: 2mins, 45sec
• Average time taken by Shuffle: 2hrs, 33mins, 9sec
• Average time taken by Reduce tasks: 1hrs, 4mins, 10sec

Output/Results

The outputted files ended up having lines looking like the one below.

 IP              EPOCH_TS   ADDON_COUNT                           LIST_OF_GUIDS
 255.255.255.255 1245665519000	2 {CAFEEFAC-0016-0000-0013-ABCDEFFEDCBA} {CAFEEFAC-0016-0000-0000-ABCDEFFEDCBA}

I created a python script to gather statistics out of the output. With the 10 second window I ended up finding out that there was a total of

• 244,727,644 add-on update pings
• 117,557,228 users
• average of 2.14 add-ons per user
• variance of 5.68

Since our Active Daily User (ADU) count for that day was 98,000,000 users this data didn’t make much sense. And we decided to repeat the processes with a 60 second window instead of the previous 10 second window.

With the 60 second window I ended up finding that there was

• 94,656,833 users
• average of 2.63 add-ons per user
• variance of 12.34

These numbers still seemed fairly large so I decided to reduce on IP address which would allow us a base where there is at least 1 user behind an ip. To reduce by IP I changed the reducer to not account for the timestamp and simply reduce all values sharing the same IP.

The output of this was

• 32,848,771 IP/USERS
• average of 5.04 add-ons per user
• variance of 779.91

To compare this data with Firefox’s ADU I ran a similar mapreduce job on our Firefox ADU data that counted

• 61,460,501 IPs
• average of 1.60 Users per IP
• variance of 86.57