Pig UDF Library Collection

I will collect the set of Pig UDF where ever i find online at this page

1) http://sna-projects.com/datafu/

UDF for Dates , Pagerank , Bags , Geo , Hash , Numbers , Sessions , Stats



2) Amazon Pig UDF Library

http://aws.amazon.com/code/2730




 







If you find something interesting and would like to add anything to this page please comment below , or write to me at jagatsingh [at] gmail dot com

Thanks

 

Sample data for practice with Hadoop

Go to http://www.infochimps.com/datasets

Filter by Free data sets available

Filter by Downloadable data only

Choose the data type which is of interest to you.

Happy Hadooping :)

 

Pig Editor for eclipse

I downloaded PigEditor from http://romainr.github.com/PigEditor/ and installed it in Eclipse

I was not able to configure PigPen which is being told as better feature rich editor then this.

Would be trying to get hold of PigPen also.

Incase you want to use PigEditor here is what you have to do

In Eclipse Update site use the URL

http://romainr.github.com/PigEditor/updates/

Let it install

Restart Eclipse

Create a New General project

It would ask you to allow XUnit use with project , say yes and you are done.

 

 

 

Pig Return codes

Pig Return codes

Value        Meaning            Comment
0        Success
1        Failure         Would retry again
2        Failure   
3        Partial Failure        Used with multiquery
4        Illegal argument        
5        IOException throws    UDF raised exception
6        PigException        Python UDF raised exception   
7        ParseException         Can happen after variable parsing if variable substitution is being done
8        Throwable        Unexcepted exception

Apache Pig Introduction Tutorial

Apache Pig is a platform to analyze large data sets.

In simple terms you have lots and lots of data on which you need to do some processing or analysis , one way is to write Map Reduce code and then run that processing on data.

Other way is to write Pig scripts which would inturn be converted to Map Reduce code and would process your data.

Pig consists of two parts

• Pig latin language
• Pig engine

Pig latin is a scripting language which allows you to describe how data flow from one or more inputs should be read , how it should be processed and then where it should be stored.

The flows can be simple or complex where some processing is applied in between. Data can be picked from multiple inputs.
We can say Pig Latin describes a directed acyclic graphs where edges are data flows and the nodes are operators that process the data

The job of engine is to exectute the data flow written in Pig latin in parallel on hadoop infrastructure.

Why Pig is required when we can code all in MR

Pig provides all standard data processing operations like sort , group , join , filter , order by , union right inside pig latin
In MR we have to lots of manual coding.

Pig does optimization of Pig latin scripts while creating them into MR jobs.
It creates optimized version of Map reduce to run on hadoop

It takes very less time to write Pig latin script then to write corresponding MR code

Where Pig is useful

Transactional ETL Data pipelines ( Mostly used)
Research on raw data
Iterative processing

You can read next about how to install Pig



 

Installing Pig ( Apache Hadoop Pig)

Apache Pig can be downloaded from http://pig.apache.org

Download the latest release from its website

Unzip the downloaded tar file

Set the environment variables in your system as

export PIG_HOME="/home/hadoop/software/pig-0.9.2"
export PATH=$PATH:$PIG_HOME/bin

Set the place where you downloaded pig and also set its path

If you plan to run Pig on hadoop cluster then one additional variable needs to be set

export PIG_CLASSPATH="/home/hadoop/software/hadoop-1.0.1/conf"

It tells about the place where to look for hdfs-site.xml and other configuration files for hadoop

Restart your computer

Thats it , now lets test the installation

On the command prompt

type

# pig -h

it shoud show the help related to Pig , and its various commands.

Done :) ,

Next you can read about

How to run your first Pig script in local mode

Or about various Pig running modes

Hadoop Pig Local mode Tutorial

The below example is explaining how to start programming in Pig.

I followed the book , Programming Pig.

This post assumes that you have already installed PIG in your computer. If you need help you can read the turorial to install pig.

So lets get start to write out first Pig program , using the same code example given in book chapter 2

Download the code examples from github website ( link below)

https://github.com/alanfgates/programmingpig

Pig can run in local mode and mapreduce mode.

When we say local mode it means that source data would be picked from the directory which is local in your computer. So to run some program you would go to the directory where data is and then run pig script to analyze the data.

I downloaded the code examples from above link

Now i go to the data directory where all data is present.

# cd /home/hadoop/Downloads/PigBook/data

Change the path depending upon where you copied the code in your computer

Now lets start pig in local mode

# pig -x local

-x local tells that Dear Pig , lets start working locally in this computer.

The output is similar to below

2012-03-11 11:44:13,346 [main] INFO  org.apache.pig.Main - Logging error messages to: /home/hadoop/Downloads/PigBook/data/pig_1331446453340.log
2012-03-11 11:44:13,720 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop filesystem at: file:///

It would enter grunt> shell , grunt is the shell to write pig scripts.

Lets try to see all the files which are present in data directory

grunt> ls

Output is shown below

file:/home/hadoop/Downloads/PigBook/data/webcrawl<r 1>    255068
file:/home/hadoop/Downloads/PigBook/data/baseball<r 1>    233509
file:/home/hadoop/Downloads/PigBook/data/NYSE_dividends<r 1>    17027
file:/home/hadoop/Downloads/PigBook/data/NYSE_daily<r 1>    3194099
file:/home/hadoop/Downloads/PigBook/data/README<r 1>    980
file:/home/hadoop/Downloads/PigBook/data/pig_1331445409976.log<r 1>    823

It is showing the list of files which are present in that folder (data)

Lets run on program , In chapter 2 there is one pig script.

Go to PigBook/examples/chap2 folder and there is one script named average_dividend.pig

The code of script is as follows

dividends = load 'NYSE_dividends' as (exchange, symbol, date, dividend);
grouped   = group dividends by symbol;
avg       = foreach grouped generate group, AVG(dividends.dividend);
store avg into 'average_dividend';

In plain english the above code is saying following

Load the NYSE_dividends file in contains fields as exchange, symbol, date, dividend
Group the records in that file by symbol

calculate average for dividend and

store the average results in average_divident folder

Result

After lots of processing the output would look like

 

Input(s):
Successfully read records from: "file:///home/hadoop/Downloads/PigBook/data/NYSE_dividends"

Output(s):
Successfully stored records in: "file:///home/hadoop/Downloads/PigBook/data/average_dividend"

Job DAG:
job_local_0001


2012-03-11 11:47:10,994 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!

 

To check the output go to average_dividend directory which is created within data directory ( remember we started pig in this directory)

There is one MR part file part-r-00000 that has the final results

Thats it , PIG latin has done all the magic behind the scene.

Coming next running Pig latin in mapreduce mode

 

 

Integrating Pig and Accumulo

Accumulo

Accumulo is a distributed key/value store that provides expressive, cell-level access labels.

Accumulo is a sorted, distributed key/value store based on Google's BigTable design. It is built on top of Apache Hadoop, Zookeeper, and Thrift.

http://www.covert.io/post/18605091231/accumulo-and-pig

The above post explains use of Pig and Accumulo together.