They posted a nice five minute screencast on their Hadoop landing page demonstrating a couple of pieces of Hive integration.  In it, they retrieve data using Hive, and they also use a Hive user defined function that is implemented as an embedded Kettle transformation.

I’m very excited to see this announcement.  Besides the significant work we’ve been doing on the Metrics team to integrate HBase into the Socorro project, we also have major plans for our Hadoop clusters for general data storage and processing.

Right now, we have Kettle jobs and transformations that manipulate gigabytes of data per hour, loading it into our data warehouse.  One of the things I love about Kettle is the ability to quickly and easily define, review, and extend complex jobs such as our end-of-day data aggregation:

In the future, as we have more data stored in Hadoop, I want to be able to run transformations on that data.  Sometimes, if the transformations involve lots of RDBMS work, I’ll want to be streaming the data out of HDFS.  For other types of transformations that involve mostly business logic and text transformations, being able to run that code directly in a Hadoop Map Reduce job will be a fantastic feature.

My personal feeling is that people in the Hadoop community really need something visual and flexible like the Kettle interface for defining and manipulating this type of business logic.  Great strides have been made with projects such as Cascading, but it is still raw code, and I feel that excludes a lot of people who could be getting work done faster and better if they had a good tool to help them adapt to the world of Map Reduce.

Currently, someone can start up Kettle’s GUI and start constructing jobs and transformations simply by piecing together steps of work such as reading a set of text files, performing a regex on them, doing some value lookups, then aggregating the data.  If they could then save that transformation and execute it as a Hadoop Map Reduce job, I think it will be revolutionary for both worlds of ETL and Hadoop.

When Mozilla Metrics starts tackling some of the Hadoop data processing jobs that we have scheduled, we’ll be making significant open source contributions to both communities to realize this vision, and I really hope that it will help widen the accessibility of Hadoop to a new group of potential users.

I added a bug severity dimension, added a component level onto the product dimension, added a team level onto the person dimension, and a days dimension to track bugs over a distribution.

I have made some mock up dashboards here and posted bellow are a few snap shots of the charts in them. Tell me what you think and what you would find useful!

For anyone else, follow along..
While my standard processing libraries parse out a lot of information from our daily traffic logs, the user agent string is not something that I parse out of the main log data because the current processing engine I have for user agent strings is a little slow and it takes a lot of CPU.

Sometimes, especially when dealing with log files as your source data, it just takes too much time to build up a big system to do one-off processing.  Way back when, Unix was founded on the idea of being able to chain together small specific tools to be able to do larger jobs efficiently.  That is something that I find very relevant in my day to day work.

So, in order to provide the numbers requested above, I turned to my tool chain of command line utilities.  One of my favorite tools in that tool chain is AWK.  It may not be quite as powerful as Perl or Python, but it is quick and simple and it is quite good at getting the job done.

Now, our automatic update web log traffic is a medium sized source of data I have to churn through.  Here are a few data points:

• We generate about four to six gigabytes of log data per day for download.mozilla.org.  It spikes to as high as thirty gigabytes on release days and on release weeks, the median is much higher, somewhere between ten to fifteen gigabytes.
• We generate about thirty gigabytes of log data per day for aus.mozilla.org during the week.  It dips as low as twenty-two gigabytes on Saturdays.
• We generate more than two hundred and sixty gigabytes of log data per day for addons.mozilla.org and versioncheck.addons.mozilla.org during the week.  It dips to just over two hundred on the weekends.

Update:  I don’t know if anyone noticed, but I had a typo in the file glob for the versioncheck histogram. That is why there was a spike in the middle instead of the larger volume of traffic for the whole time period.

How did I get those numbers you ask? Why with a shell script:

[me@etl01 download.mozilla.org]$ gzip -l access_2009-07*.gz | awk -f ~/bin/histogram_sum.awk scale=500000000 column=8 width=10
2009-07-08 -   6.44GB: **************
2009-07-09 -   6.19GB: **************
2009-07-10 -   5.68GB: *************
2009-07-11 -   4.59GB: **********
2009-07-12 -   4.90GB: ***********
2009-07-13 -   6.02GB: *************
2009-07-14 -   5.85GB: *************
2009-07-15 -   5.81GB: *************
2009-07-16 -   6.13GB: **************
2009-07-17 -   9.37GB: *********************
2009-07-18 -   6.45GB: **************
2009-07-19 -   6.00GB: *************
2009-07-20 -   7.12GB: ****************
2009-07-21 -  12.72GB: ****************************
2009-07-22 -  33.78GB: *************************************************************************
2009-07-23 -  23.09GB: **************************************************
2009-07-24 -  15.41GB: **********************************
2009-07-25 -  10.95GB: ************************
2009-07-26 -  10.38GB: ***********************
2009-07-27 -  11.75GB: **************************
2009-07-28 -  10.39GB: ***********************
2009-07-29 -   7.32GB: ****************
Total counts = 216.93GB

[me@etl01 aus.mozilla.org]$ gzip -l access_2009-{06-30,07}*.gz | awk -f ~/bin/histogram_sum.awk scale=500000000 column=8 width=10
2009-06-30 -  30.55GB: ******************************************************************
2009-07-01 -  29.88GB: *****************************************************************
2009-07-02 -  29.21GB: ***************************************************************
2009-07-03 -  26.12GB: *********************************************************
2009-07-04 -  21.97GB: ************************************************
2009-07-05 -  25.63GB: ********************************************************
2009-07-06 -  30.20GB: *****************************************************************
2009-07-07 -  29.88GB: *****************************************************************
2009-07-08 -  29.78GB: ****************************************************************
2009-07-09 -  29.22GB: ***************************************************************
2009-07-10 -  27.06GB: ***********************************************************
2009-07-11 -  22.74GB: *************************************************
2009-07-12 -  25.69GB: ********************************************************
2009-07-13 -  29.81GB: *****************************************************************
2009-07-14 -  29.49GB: ****************************************************************
2009-07-15 -  29.34GB: ****************************************************************
2009-07-16 -  28.79GB: **************************************************************
2009-07-17 -  26.98GB: **********************************************************
2009-07-18 -  22.44GB: *************************************************
2009-07-19 -  25.11GB: ******************************************************
2009-07-20 -  29.50GB: ****************************************************************
2009-07-21 -  29.32GB: ***************************************************************
2009-07-22 -  28.15GB: *************************************************************
2009-07-23 -  25.53GB: *******************************************************
2009-07-24 -  24.34GB: *****************************************************
2009-07-25 -  20.89GB: *********************************************
2009-07-26 -  23.84GB: ****************************************************
2009-07-27 -  28.23GB: *************************************************************
2009-07-28 -  28.34GB: *************************************************************
2009-07-29 -  22.12GB: ************************************************
Total counts = 810.15GB

[me@etl01 logs]$ cat
 <(cd addons.mozilla.org; gzip -l access_2009-{06-30,07}*.gz | tail -n +2)
 <(cd versioncheck.addons.mozilla.org; gzip -l access_2009-{06-30,07}*.gz | tail -n +2)
 | sort -k 4 | awk -f ~/bin/histogram_sum.awk scale=5000000000 column=8 width=10
2009-06-30 - 284.10GB: **************************************************************
2009-07-01 - 285.41GB: **************************************************************
2009-07-02 - 259.99GB: ********************************************************
2009-07-03 - 227.03GB: *************************************************
2009-07-04 - 192.92GB: ******************************************
2009-07-05 - 216.33GB: ***********************************************
2009-07-06 - 247.53GB: ******************************************************
2009-07-07 - 244.36GB: *****************************************************
2009-07-08 - 243.49GB: *****************************************************
2009-07-09 - 235.46GB: ***************************************************
2009-07-10 - 218.33GB: ***********************************************
2009-07-11 - 190.16GB: *****************************************
2009-07-12 - 207.19GB: *********************************************
2009-07-13 - 231.92GB: **************************************************
2009-07-14 - 232.91GB: ***************************************************
2009-07-15 - 230.37GB: **************************************************
2009-07-16 - 228.75GB: **************************************************
2009-07-17 - 236.00GB: ***************************************************
2009-07-18 - 203.75GB: ********************************************
2009-07-19 - 214.33GB: ***********************************************
2009-07-20 - 238.31GB: ****************************************************
2009-07-21 - 242.12GB: ****************************************************
2009-07-22 - 277.91GB: ************************************************************
2009-07-23 - 272.68GB: ***********************************************************
2009-07-24 - 235.63GB: ***************************************************
2009-07-25 - 202.74GB: ********************************************
2009-07-26 - 216.85GB: ***********************************************
2009-07-27 - 241.81GB: ****************************************************
2009-07-28 - 238.79GB: ****************************************************
2009-07-29 - 232.97GB: ***************************************************
Total counts = 7099.05GB

The command I ran to generate the stats driving the Many Eyes site above was this:

pv access_2009-{06-30,07}*.gz |
 gzip -cd | grep -F "Macintosh" |
 sed -rne '/"GET \/update\/[1-3]\/Firefox\/.*?Macintosh/s/^.*?\[([0-9]{2}\/[a-zA-Z]{3}\/[0-9]{4}).*?(Intel|PPC) Mac OS X.*/\1\t\2/p' |
 awk -f ~/bin/tabulate_fields.awk kf=1 cf=2

The interesting parts of this command are:

pv access_2009-{06-30,07}*.gz — Pipe Viewer a utility for monitoring the progress of data through a pipeline.  When the argument to pv is a file, it stats file file to determine how large it is so that it can calculate an ETA to finish processing it.  This is why I do the otherwise foolish trick of cat’ing to gzip -cd instead of passing the file list direclty to gzip or zgrep.  Here, I’m passing in a fileglob of all the download access logs from June 30 to July 28th (the day I ran it).

grep -F “Macintosh” — I go ahead and pay the penalty for an extra process here because grep -F is much more efficient at throwing away the 98% of the log data that I’m not interested in than the much more complicated sed regex that follows.  The best part of pv is that it lets you actually test optimizations like this.  When I ran without the grep, I was processing about 1.5 MB/s.  When I put in the simple grep filter, that number jumped up to 5.5 MB/s.

sed -rne … — I won’t go into the gritty details about the regex for now.  Suffice to say that this regex looks for Firefox update checks that have the string Macintosh later in the request (hopefully in the user agent data).  For those matches, it then pulls out the request date and a single word, “Intel” or “PPC” if it is immediately followed by “Mac OS X”.

awk -f ~/bin/tabulate_fields.awk kf=1 cf=2 — This is my pièce de résistance.  Given many different types of tabular data, it can count occurrences of values for a given key value, and it can also sum a numeric column for the counted occurrences. This is a fairly simple use case.  It just keys off of the date in the first column of the output of sed and counts the occurrences of “Intel” or “PPC” for each day.

Hope you enjoyed this geek session.  If you would like to take the time to show me how I can do this things much more elegantly or quickly using Perl or Python, I’d actually love to hear it!  I really should learn to stop avoiding these wonderful languages just because I’m more familiar with an older tool. ;)

-Daniel

I ran an analysis on a year and a half of FTP log files, filtering for some specific requests, and filtering out but summarizing uninteresting traffic from our heartbeat monitors and such.

I put together this simple Kettle transformation, and ran it with a cluster consisting of 32 fairly low powered slaves. The results were pleasing, especially considering I didn’t even go through a tuning process to determine the optimal number of step copies or row set sizes.

8782 files containing 432 million records. The processing was completed in 47 minutes giving a throughput of about 156 thousand rows per second.

There are a couple of screenshots after the cut.

Transformation flow

Transformation log

• verions
• products
• resolution
• bug_status
• priority
• bugs
• bugs_activity
• profiles

These tables below could just be transferred in whole because of there small size and the information contained in them was just meta/attribute data of the bugs and besides that they didn’t have any related information to the bugs.bug_id:

• verions
• products
• resolution
• bug_status
• priority

For the bugs I had to construct a statistical sampling and based on the bugs statistical sampling gather corresponding entries in the bugs_activity and profiles tables. I selected data from the bugs table and passed it through a reservoir sampling step that would pull a random amount of rows from the returned result set. I decided to pull a random sample of 1,000 bugs. After that I had a new job that matches the bug_id’s pulled from the reservoir sample with the bugs_activity table, and I then had to match the assigned_to and reporter fields from the random sample with the userid in the profiles table.

The creating of the Bugzilla Sample Database inside of spoon consists of 3 main jobs:

1. Resetting the Sample Bugzilla Database
2. Transferring the Non-bugs tables
3. Transferring the Bugs tables

The first step will reset the sample Bugzilla database where we reset or clear any information stored in the database. The step follows the pseudo code below.

For Table t in the set of Tables T {
   If t exists {
      Drop t
   }
   Create t
}


The second step will transfer the trivial tables mentioned above that do not involve the bugs table.

The third step will collect a sampling from the bugs table, output the entries into the new bugs table, select the bugs_activity entries based on bug_id, and profiles entries based on assigned_to and reporters, and output these into the new bugs_activity and profiles entries.

To begin you will want to have is access to a MySQL database where you can create, drop, and insert. The name of the database doesn’t matter but mine was called bugs_sample. After you have access to a MySQL database open up spoon, create a job, and setup the database connections as described below.

1. Create a new job by going to File > New > Job .
2. Right click in the Staging area and select Job settings from the menu that appears. This will open up the Job properties dialog window.
3. Change the Job name: field to something like “Create Bugzilla Sample Database”.
4. In the top left of spoon click on the Magnifying Glass with the label “View” under it.
5. Right-Click on the label Database connections. From the drop down menu select New. This will open up the Database connection dialog box.
6. The first database connection to create will be for the real Bugzilla database. Fill in the credentials to log into the Bugzilla database. An example is below.
   
7. Click Test and you should get a message like the one below but with your credentials.
   
   If you didn’t receive this message check your MySQL connections and credentials.
8. Repeat previous two steps above to create a database connection for the new Bugzilla Sample database.
9. After the two database connections are set up it is now time to create a new job for the first step in the process. This will be done by creating a new job, and inside the job checking to see if each table exists. If it does drop it. Create the table. As described earlier. Follow this video to create the first part of the job.
   The schema for each table can be found here.
10. Repeat the last part of the previous step for each table mentioned above. The end result should look something like the image below.
    
11. Now we will create a new job for the second part of the process where we transfer all tables that don’t directly relate to the bugs table over to the new Sample Database. Create a new job entitled Transfer Non-bugs. This job will be composed of multiple transformations that will transfer tables from the Bugzilla database to our new Bugzilla Sample database. To construct the transformation to transfer a table from the Bugzilla database to the new Bugzilla Sample database follow this video. Create a new transformation for each non related to bugs table that are listed above.
12. Once a transformation for each non related bug table has been completed and saved. Create a job to connect them together like the image below.
    
13. Now it is time to create the third and last part of the transformation where we collect a sampling from the bugs table and based of this sampling select corresponding bugs_activity, and profiles entries. This is composed of two parts. The first part we will gather bugs and then do a reservoir sampling on this gathering. Then we will store the data to the new table and finally we will save some values to variables for use in a later query. This video describes how to construct the first part.

    NOTE: It was necessary to assign these values into a variable. I tried feeding the output fields from the group by CSV directly into a table input step query but the JDBC driver will only accept the first entry in the CSV when using the a WHERE IN (?) statement. By using the variable in the WHERE IN (${VARIABLE}) query eliminated this problem.

14. The second part is two take the variables that were just assigned and used them in a WHERE of a query to select data that we need from the Bugzilla database and then dump the data into our new Bugzilla Sample database. This video describes this second part of the transformation.
15. Finally we want to construct jobs that tie everything all together. This video shows the final job with everything connected.

If you would like to change your sample size change it in the reservoir sampling step. I would even recommend having a variable in this location to easily change the sample size when running the job. Also change the seed value inside of the reservoir sampling step each time to guarantee a new sample.

I have attached and included the source files in a compressed file for anyone interested. Read the README after unzipping.

A large part of this project involves doing an ETL(Extract Transform Load) on the bugzilla database into a star schema in side of a data warehouse. To design and run the ETL process I am using a program from the open source community project Pentaho BI (Business Intelligence) Suite called Pentaho Data Integration (PDI, and formerly known as kettle) and spoon — the graphical tool which is used to design and test every PDI process. Once the data is loaded in to the star schema, I will then use the Pentaho BI server to create graphs and charts to visualize and and drill down the data.

I will be reporting my work through out this blog and I hope that it will allow any one interested to participate, learn, or contribute. To get started right now you can read up about Pentaho, Pentaho Data Integration (formerly known as kettle), and Nick Goodman’s Software Quality Reports can be found on sourceforge.