They posted a nice five minute screencast on their Hadoop landing page demonstrating a couple of pieces of Hive integration.  In it, they retrieve data using Hive, and they also use a Hive user defined function that is implemented as an embedded Kettle transformation.

I’m very excited to see this announcement.  Besides the significant work we’ve been doing on the Metrics team to integrate HBase into the Socorro project, we also have major plans for our Hadoop clusters for general data storage and processing.

Right now, we have Kettle jobs and transformations that manipulate gigabytes of data per hour, loading it into our data warehouse.  One of the things I love about Kettle is the ability to quickly and easily define, review, and extend complex jobs such as our end-of-day data aggregation:

In the future, as we have more data stored in Hadoop, I want to be able to run transformations on that data.  Sometimes, if the transformations involve lots of RDBMS work, I’ll want to be streaming the data out of HDFS.  For other types of transformations that involve mostly business logic and text transformations, being able to run that code directly in a Hadoop Map Reduce job will be a fantastic feature.

My personal feeling is that people in the Hadoop community really need something visual and flexible like the Kettle interface for defining and manipulating this type of business logic.  Great strides have been made with projects such as Cascading, but it is still raw code, and I feel that excludes a lot of people who could be getting work done faster and better if they had a good tool to help them adapt to the world of Map Reduce.

Currently, someone can start up Kettle’s GUI and start constructing jobs and transformations simply by piecing together steps of work such as reading a set of text files, performing a regex on them, doing some value lookups, then aggregating the data.  If they could then save that transformation and execute it as a Hadoop Map Reduce job, I think it will be revolutionary for both worlds of ETL and Hadoop.

When Mozilla Metrics starts tackling some of the Hadoop data processing jobs that we have scheduled, we’ll be making significant open source contributions to both communities to realize this vision, and I really hope that it will help widen the accessibility of Hadoop to a new group of potential users.

I added a bug severity dimension, added a component level onto the product dimension, added a team level onto the person dimension, and a days dimension to track bugs over a distribution.

I have made some mock up dashboards here and posted bellow are a few snap shots of the charts in them. Tell me what you think and what you would find useful!

I ran an analysis on a year and a half of FTP log files, filtering for some specific requests, and filtering out but summarizing uninteresting traffic from our heartbeat monitors and such.

I put together this simple Kettle transformation, and ran it with a cluster consisting of 32 fairly low powered slaves. The results were pleasing, especially considering I didn’t even go through a tuning process to determine the optimal number of step copies or row set sizes.

8782 files containing 432 million records. The processing was completed in 47 minutes giving a throughput of about 156 thousand rows per second.

There are a couple of screenshots after the cut.

Transformation flow

Transformation log

• verions
• products
• resolution
• bug_status
• priority
• bugs
• bugs_activity
• profiles

These tables below could just be transferred in whole because of there small size and the information contained in them was just meta/attribute data of the bugs and besides that they didn’t have any related information to the bugs.bug_id:

• verions
• products
• resolution
• bug_status
• priority

For the bugs I had to construct a statistical sampling and based on the bugs statistical sampling gather corresponding entries in the bugs_activity and profiles tables. I selected data from the bugs table and passed it through a reservoir sampling step that would pull a random amount of rows from the returned result set. I decided to pull a random sample of 1,000 bugs. After that I had a new job that matches the bug_id’s pulled from the reservoir sample with the bugs_activity table, and I then had to match the assigned_to and reporter fields from the random sample with the userid in the profiles table.

The creating of the Bugzilla Sample Database inside of spoon consists of 3 main jobs:

1. Resetting the Sample Bugzilla Database
2. Transferring the Non-bugs tables
3. Transferring the Bugs tables

The first step will reset the sample Bugzilla database where we reset or clear any information stored in the database. The step follows the pseudo code below.

For Table t in the set of Tables T {
   If t exists {
      Drop t
   }
   Create t
}


The second step will transfer the trivial tables mentioned above that do not involve the bugs table.

The third step will collect a sampling from the bugs table, output the entries into the new bugs table, select the bugs_activity entries based on bug_id, and profiles entries based on assigned_to and reporters, and output these into the new bugs_activity and profiles entries.

To begin you will want to have is access to a MySQL database where you can create, drop, and insert. The name of the database doesn’t matter but mine was called bugs_sample. After you have access to a MySQL database open up spoon, create a job, and setup the database connections as described below.

1. Create a new job by going to File > New > Job .
2. Right click in the Staging area and select Job settings from the menu that appears. This will open up the Job properties dialog window.
3. Change the Job name: field to something like “Create Bugzilla Sample Database”.
4. In the top left of spoon click on the Magnifying Glass with the label “View” under it.
5. Right-Click on the label Database connections. From the drop down menu select New. This will open up the Database connection dialog box.
6. The first database connection to create will be for the real Bugzilla database. Fill in the credentials to log into the Bugzilla database. An example is below.
   
7. Click Test and you should get a message like the one below but with your credentials.
   
   If you didn’t receive this message check your MySQL connections and credentials.
8. Repeat previous two steps above to create a database connection for the new Bugzilla Sample database.
9. After the two database connections are set up it is now time to create a new job for the first step in the process. This will be done by creating a new job, and inside the job checking to see if each table exists. If it does drop it. Create the table. As described earlier. Follow this video to create the first part of the job.
   The schema for each table can be found here.
10. Repeat the last part of the previous step for each table mentioned above. The end result should look something like the image below.
    
11. Now we will create a new job for the second part of the process where we transfer all tables that don’t directly relate to the bugs table over to the new Sample Database. Create a new job entitled Transfer Non-bugs. This job will be composed of multiple transformations that will transfer tables from the Bugzilla database to our new Bugzilla Sample database. To construct the transformation to transfer a table from the Bugzilla database to the new Bugzilla Sample database follow this video. Create a new transformation for each non related to bugs table that are listed above.
12. Once a transformation for each non related bug table has been completed and saved. Create a job to connect them together like the image below.
    
13. Now it is time to create the third and last part of the transformation where we collect a sampling from the bugs table and based of this sampling select corresponding bugs_activity, and profiles entries. This is composed of two parts. The first part we will gather bugs and then do a reservoir sampling on this gathering. Then we will store the data to the new table and finally we will save some values to variables for use in a later query. This video describes how to construct the first part.

    NOTE: It was necessary to assign these values into a variable. I tried feeding the output fields from the group by CSV directly into a table input step query but the JDBC driver will only accept the first entry in the CSV when using the a WHERE IN (?) statement. By using the variable in the WHERE IN (${VARIABLE}) query eliminated this problem.

14. The second part is two take the variables that were just assigned and used them in a WHERE of a query to select data that we need from the Bugzilla database and then dump the data into our new Bugzilla Sample database. This video describes this second part of the transformation.
15. Finally we want to construct jobs that tie everything all together. This video shows the final job with everything connected.

If you would like to change your sample size change it in the reservoir sampling step. I would even recommend having a variable in this location to easily change the sample size when running the job. Also change the seed value inside of the reservoir sampling step each time to guarantee a new sample.

I have attached and included the source files in a compressed file for anyone interested. Read the README after unzipping.

A large part of this project involves doing an ETL(Extract Transform Load) on the bugzilla database into a star schema in side of a data warehouse. To design and run the ETL process I am using a program from the open source community project Pentaho BI (Business Intelligence) Suite called Pentaho Data Integration (PDI, and formerly known as kettle) and spoon — the graphical tool which is used to design and test every PDI process. Once the data is loaded in to the star schema, I will then use the Pentaho BI server to create graphs and charts to visualize and and drill down the data.

I will be reporting my work through out this blog and I hope that it will allow any one interested to participate, learn, or contribute. To get started right now you can read up about Pentaho, Pentaho Data Integration (formerly known as kettle), and Nick Goodman’s Software Quality Reports can be found on sourceforge.