Few reasons why log collection is usually at tail of priorities:

1. Nagios alerts usually do a good job of monitoring for critical situations. The scripts make sure the app’s always online by grep’ing for “ERROR, WARN” and other magic terms in logs, but what about errors that occur often but don’t bring down the app completely?
2. Web-analytics give us all information we need. -Yes on a macroscopic view, but it’s really hard for an analytical software to provide fine granularity, such as how many hits did we receive pertaining to a given country for a given page for a given time-period?
3. Ganglia graphs help us find out what machine/s are under heavy load – Absolutely, but trying to figure what triggered the load in first place is not always easy.

Chukwa, Scribe and Flume are headed in the right direction, but the final piece of puzzle of analyzing the data still remained unsolved, until few weeks back as we, at Mozilla, started integrating Flume with Hive.

Merge everything - Image courtesy Wikipedia.org

Flume is an open-source distributed log collection software that can be installed on multiple machines for monitoring log files with data slurped to a single HDFS location. The out of box solution only solved part of our problem of collecting data, but we needed a way to query it and thereby make intelligent decisions based on the results.

The teams first foray was to add a gzip patch that compressed the log data before transferring the files to HDFS. Once the data was transferred, we needed a way to query it. Our current production Hadoop cluster consists of modest 20 machines, has excellent monitoring in terms of nagios and ganglia, but the question of what might we be missing always lingered on our heads. A list of basic things needed to be taken care of while integrating Flume with Hive was created:

1. How do we handle fail-overs when the HIVE metastore service, a possible single point of failure for HIVE goes down?
2. How to query data by the day and hour.
3. Can separate tables be used for different log locations?
4. Can we split a single log line in its respective columns?

1. Handling fail-overs: We are currently running HIVE metastore in remote mode using MySQL. More information on metastore setup can be found at http://wiki.apache.org/hadoop/Hive/AdminManual/MetastoreAdmin. Flume node-agents reliable handle fail-overs by maintaining checksums at regular intervals and making sure data isn’t inserted twice. The same principle was extended by adding marker points. i.e. a file containing HQL query and the location of data will be written to HDFS after every successful FLUME roll-over. Flume agents would look at a common location for pending HIVE writes before writing any log data to HDFS, attempt to move the data inside HIVE and only delete the marker file if successful. In situations where two or more flume agents attempt to move files to HIVE partition, one of them will encounter an innocuous HDFS file not found error and proceed as usual.

2. Appending to sub-partitions: Flume supports rollover where data is written to disk every ‘x’ millis. This is particularly useful as data is available inside HDFS at regular intervals and can be queried by the hour or minute granularity. While whole data can be written to a single partition, partitioning data inside HIVE is a huge performance benefit as it only siphons through a specific range rather than whole data set. This was achieved by having two partitions for a table – by date and hour. An equivalent HIVE query looks something like:

LOAD DATA INPATH ‘” + dstPath + “‘ INTO TABLE ” + hiveTableName + ” PARTITION (ds=’” + dateFormatDay.format(cal.getTime()) +
“‘, ts=’” + dateFormatHourMinute.format(cal.getTime()));

3. Using separate tables for different log locations: We wanted to use separate tables for Hadoop and HBase log locations. Our initial approach was to add a config setting in flume-site.xml, but half way down that road we realized that config is wrong place, as it needs to exist on each node-agent and mapping different folders to tables will be a logistical nightmare.

A new sink named hiveCollectorSink(hdfs_path, prefix, table_name) was added to the existing family (http://archive.cloudera.com/cdh/3/flume/UserGuide.html#_output_bucketing). This allowed us to add hive tables on the fly for each log folder location, thereby giving a separate placeholder for Hadoop and Hbase logs.

4. Splitting a single log line in respective columns (a.k.a. regex): Log4J is a standard log file convention used by quite a few applications including Hadoop and HBase. A sample line looks something like this:

2010-08-15 12:36:59,850 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_-1857716372571578738_336272

Given the above structure, we decided to split the line in 5 columns:
date, time, message_type, class_name and message; the table definition given below -

CREATE TABLE cluster_logs (
line_date STRING,
line_time STRING,
message_type STRING,
classname STRING,
message STRING
)
PARTITIONED BY (ds STRING, ts STRING, hn STRING)

ROW FORMAT SERDE ‘org.apache.hadoop.hive.contrib.serde2.RegexSerDe’
WITH SERDEPROPERTIES (
“input.regex” =
“^(?>(\\d{4}(?>-\\d{2}){2})\\s((?>\\d{2}[:,]){3}\\d{3})\\s([A-Z]+)\\s([^:]+):\\s)?(.*)”
)
STORED AS TEXTFILE;

NOTE: The “hn” (hostname) partition was added so we could query the data based on individual hostnames, enabling us to know what hostname has biggest chunk of ERROR, WARN messages.

The above framework has allowed us to reliably collect logs from our entire cluster to a single location and then query the data from a SQLish interface.

Future Steps:

• Flume + Hive patch is still a work in progress and will be committed to the trunk in a couple of weeks.
• Socorro (Mozilla’s crash reporting system) 1.9 and above will be using processors in a distributed mode and we plan to insert the processor’s log data inside HIVE thereby helping us better understand throughput, avg. time to process each crash-data and other metrics. Watch this space for more related posts.

The developers of Flume + Hive usually hang on IRC (irc.freenode.net) in the following channels: #flume, #hive, #hbase
Feel free to ask questions/thoughts/suggestions and I will reply to them below.

-Anurag Phadke (email: first letter of my firstname followed by last name at – the – rate  mozilla dot com)

I was asked this evening if the nightly report was correct in showing that we had a 128% increase in Firefox downloads today.  The answer is a resounding yes that figure is correct, but I figured it wouldn’t hurt to put a bit more detail on it.

Histograms demonstrating the sharp flood of upgrade patch downloads on release day

Wednesday, February 17th, 2010 was a release day for both Firefox 3.0.18 and 3.5.8.  Starting around noon Pacific time yesterday, hundreds of millions of Firefox users would eventually see a prompt notifying them that there was an upgrade available to install.

If the user was running the latest security release for their version of Firefox, the upgrade would consist of a small patch file that would quickly bring them up to the new security release.

If the user was not on the latest security release, they would instead download a special file that was the full size of the Firefox application.  The upgrade process would then automatically install the upgrade.

The question at hand is, “why do we see an increase in people visiting the Mozilla website and downloading a new installer?”  In the past, we’ve had concerns about whether people were having trouble with the automatic update system and were being forced to download the application and install it manually.  From the data I reviewed today, I think it is safe to say that is not a common scenario.  Instead, I’m happy to report that what I see is a lot of people who get a reminder that they should upgrade Firefox and they decide that it is about time for them to go ahead and download the latest and greatest Firefox 3.6 instead.

The chart above (generated courtesy of Tableau Software) shows that manual downloads of the 3.0 and 3.5 versions remained relatively flat, but manual downloads of Firefox 3.6 climbed by almost 3x over the previous day’s peak traffic time.
I would also like to point out that an infrastructure that can handle in increase of over 5 million requests per hour in a three hour window isn’t too shabby.

While my Pentaho Data Integration ETL processes do slow down considerably when processing this huge influx of data, they keep up, managing to process these 10 or so million requests per hour in 30 to 40 minutes.  It could actually be much quicker if I moved this processing to a separate server, but the primary reason it is slow is because once it falls outside its normal 5 minute run time, it has to compete with other ETL processes that are scheduled to run later in the hour.

I was presented with the challenge of answering the question – how many Firefox users have one add-on or more installed on their Firefox. Currently, addons.mozilla.org (AMO) has statistics on the download counts of add-ons but the actual usage of add-ons has been unanswered.

The Add-ons manager inside of Firefox will check each add-on for update at AMO. This happens once every 24 hour period when Firefox is ran by the user. Updates are handled over HTTP at either addons.mozilla.org for Firefox1.0/1.5/2.0 and versioncheck.addons.mozilla.org for Firefox3.0/3.5. The add-ons manager will ping the servers with information about each add-on and if an update exists the server will respond with one. Since the update ping is handled in HTTP the update ping is recorded in a log file. If you have never seen a web server’s HTTP log file they simply are flat text files where each line contains information about the requests made to the server. Below is example line of the AMO log file and an explanation of the fields.

IP                  HOSTNAME                             TIMESTAMP          REQUEST
 255.255.255.255 versioncheck.addons.mozilla.org - [22/Jun/2009:02:00:00 -0700] "GET
/update/VersionCheck.php?reqVersion=1&id={B13721C7-F507-4982-B2E5-502A71474FED}&
version=2.2.0.102&maxAppVersion=3.*&status=userEnabled&
appID={ec8030f7-c20a-464f-9b0e-13a3a9e97384}&appVersion=3.0.11&appOS=WINNT&
appABI=x86-msvc&locale=en-US HTTP/1.1" 200 520 "-"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.11)
Gecko/2009060215 Firefox/3.0.11(.NET CLR 3.5.30729)"

We choose the log files on 2009/06/22 because Firefox will ping AMO multiple times after a Firefox version update and this date was 11 days after the Firefox 3.0.11 release and 3 days after Firefox 3.5 RC2 release so most users should have already of been up-to-date by this time. The whole day’s worth of log data for both hostnames total out to be around 28GB compressed. The log files were large because they also contained requests for AMO’s website.

There is no unique identifier to determine which update pings came from which user and we had to rely on identifying pings from a single user by the IP address, and Timestamp in the update ping. The IP address is the single most unique identifier due to the nature of IP addresses but because of routers (NAT) and proxies many computers can sit behind one IP address. To add another degree of separation we decided to group the update pings by the timestamp of the ping. Update pings will happen within a few seconds from each other so pings in a certain time window would be considered as one user, and other pings from an IP address out side of this time window would be considered as a different user. For example, say that there are two Firefox users behind a router. User 1 might open up his browser in the morning at 10AM and ping AMO and User 2 might open up his browser in the afternoon and ping AMO at 2PM. Even though the pings are from the same IP address the pings at 10AM would all be grouped together and counted separately from the pings that happened at 2PM which would also be grouped together.

Setup/Config

Upon hearing about the description of the problem I thought that this would be an ideal candidate for a MapReduce job. I would be able to Map the IP address as the Key and all the other data in the log file entry as a HashMap for the value. I talked with my manager and found out that this was a technology that the metrics team was interested in exploring and was given 4 mac minis to test my implementation out on. I quickly began setting up my Hadoop Cluster running on ubuntu 9.04 desktop which soon became ubuntu 9.04 server for memory conservation and unnecessary gui (Ubuntu 9.04 Desktop takes up about 256MB of RAM on a clean install while Server takes up about 90MB of RAM on a clean install). I have been a user of hadoop in the past but I had never setup my own hadoop cluster before so I turned to the hadoop website and this blog post to help me out. These tutorials at the time did not exactly cover the hadoop version I was using, 0.20. (I believe they are up-to-date now). But was able to accommodate for this fact. The main two things different in hadoop version 0.19 than in version 0.20 is the configuration file and parts of the hadoop java API. In version 0.20, /conf/hadoop-site.xml was split in to three parts in /conf/core-site.xml, /conf/hdfs-site.xml, and /conf/mapred-site.xml and the API was re-factored slightly and some classes were depreciated.

My Mac minis were given hostnames of hadoop-node1-4. Hadoop-node1 was my master node that ran the NameNode and the JobTracker while hadoop-node2-4 were my slaves that ran the TaskTracker and the DataNode.

 # /conf/masters
 hadoop-node1.mv.mozilla.com

 # /conf/slaves
 hadoop-node2.mv.mozilla.com
 hadoop-node3.mv.mozilla.com
 hadoop-node4.mv.mozilla.com

 # The java processes
 hadoop@hadoop-node1:/usr/local/hadoop/conf$ jps
 15778 Jps
 30059 NameNode
 30187 SecondaryNameNode
 30291 JobTracker

 hadoop@hadoop-node2:/usr/local/hadoop$ jps
 16950 TaskTracker
 16838 DataNode
 20186 Jps

After getting the cluster setup I tested it out with the hadoop wordcount example and validated the results.

MapReduce

I then began writing a MapReduce Job with the Hadoop Java API. My first thought was to write my own RecordReader which is responsible for reading from an input split that would give key, value pairs to the mapper but decided to go with the default LineRecordReader which puts the file offset as the key and the line as the value because it seemed easier and more natural to have the log file line dissected inside of the Mapper’s map function.

Map

In the map function each line went through a regexp that broke each piece out of the log file line. If the line contained Firefox’s appid, and VersionCheck.php I would map the IP as the Key and construct an AddonsWritable (which is a child of MapWritable with an overridden toString() for output purposes) that contained the epoch time (converted from the date timestamp because it would be much easier to compare with), a MapWritable of add-on guids, and a count of the number of add-ons.

public static class IPAddressMapper extends Mapper{
     /* member vars for mapper which include vars for regexp and storing data */
     private AddonsWritable logInfo = new AddonsWritable();

     public void map(LongWritable key, Text logLine, Context context) throws IOException, InterruptedException {
         if(logLine.matchesRegexp() && isFirefox() && hasVersionCheckphp()) {
             logInfo.put(EPOCH, epoch); // store the epoch_ts
             logInfo.put(GUID, guid); // store the guid
             logInfo.put(TOTAL, ONE); // store the count
             context.write(ipAddress, logInfo); // map out the ipAddress as Key and logInfo as value
         }
     }
 }

Reduce

After the map the Hadoop Framework hashes keys and gives them to the reduce function. Inside the Reducer’s reduce function you are given the Key which is the IP address and an Iterable of the AddonsWritables that were from the same key/IP. I needed to group the values with update pings in a certain time window together and unfortunately the Iterable does not guarantee order. So I put the MapWritables in a PriorityQueue with a custom comparator that ordered values by the timestamp field in the AddonsWritable. Some IP’s had thousands of pings, so if I counted an IP with more than 2,000 pings I threw it out. Once all the values were placed in the PriorityQueue, I iterated over the priorityQueue popping off each value and comparing to the previous seen timestamp. If the abs(current timestamp – prev timestamp) <= 10 secs I considered them to be from the same user, and added them to a MapWritable of guids inside of the MapWritable that contained all the other information. Once I saw a current timestamp where the abs(current timestamp – prev timestamp) > 10secs I wrote/collected the previous values and started a new MapWritable for the next new window. Until there were no more values in the priorityQueue and I would write/collect the final current values.

public static class IPAddressEpochTimeReducer extends Reducer {
     private PriorityQueue pq = new PriorityQueue();
     public void reduce(Text key, Iterable values, Context context) throws IOException, InterruptedException {
         for(AddonsWritable val: values) {
             pq.add(new AddonsWritable(val));
             if(pq.size() > 2000) {
                 pq.clear();
                 return ;
             }
         }

         while(!pq.isEmpty()) {
             AddonsWritable val = pq.remove();
             if(lastEpoch != -1 && Math.abs(lastEpoch - currentEpoch) > SIXTY_SECONDS) {
                 writeOut();  // Write out all the information for the current collection of versioncheck pings
                 resetVars();  // Reset all the currently used vars for the next collection of versioncheck pings
             }
             addGuid(output, val.get(GUID));
             sum += val.get(TOTAL);
             lastEpoch = currentEpoch;
         }
         /* There is one more remaining.  Write it out */
         writeOut();
     }
 }

Runtime Stats

Hadoop provides a web interface that will output runtime statistics for a job, below are the stats for the job described above.

• Submitted At: 17-Jul-2009 09:55:12
• Launched At: 17-Jul-2009 09:55:12 (0sec)
• Finished At: 17-Jul-2009 14:05:52 (4hrs, 10mins, 39sec)
• Average time taken by Map tasks: 2mins, 45sec
• Average time taken by Shuffle: 2hrs, 33mins, 9sec
• Average time taken by Reduce tasks: 1hrs, 4mins, 10sec

Output/Results

The outputted files ended up having lines looking like the one below.

 IP              EPOCH_TS   ADDON_COUNT                           LIST_OF_GUIDS
 255.255.255.255 1245665519000	2 {CAFEEFAC-0016-0000-0013-ABCDEFFEDCBA} {CAFEEFAC-0016-0000-0000-ABCDEFFEDCBA}

I created a python script to gather statistics out of the output. With the 10 second window I ended up finding out that there was a total of

• 244,727,644 add-on update pings
• 117,557,228 users
• average of 2.14 add-ons per user
• variance of 5.68

Since our Active Daily User (ADU) count for that day was 98,000,000 users this data didn’t make much sense. And we decided to repeat the processes with a 60 second window instead of the previous 10 second window.

With the 60 second window I ended up finding that there was

• 94,656,833 users
• average of 2.63 add-ons per user
• variance of 12.34

These numbers still seemed fairly large so I decided to reduce on IP address which would allow us a base where there is at least 1 user behind an ip. To reduce by IP I changed the reducer to not account for the timestamp and simply reduce all values sharing the same IP.

The output of this was

• 32,848,771 IP/USERS
• average of 5.04 add-ons per user
• variance of 779.91

To compare this data with Firefox’s ADU I ran a similar mapreduce job on our Firefox ADU data that counted

• 61,460,501 IPs
• average of 1.60 Users per IP
• variance of 86.57