Jugnu Life :-)

HDFS Federation in Cluster

HDFS Federation was one of the new featruees introduced in Hadoop recently.

If you have read the instructions present on Apache Website

http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html

It guides us to configure the Hadoop Federation.

Some of the additional things which are required are telling clients

In core-site.xml

fs.defaultFs
viewFs://ClusterID

Above means that fs.defaultFs should have value of your cluster ID

Additionally we have to tell Clients where to go for which path

fs.viewds.mounttable.MyNewClusterID.link./PATH
hdfs://NN-host1:port/PATH

The above means that for this path on cluster id go to this namenode and path

e.g

fs.viewds.mounttable.MyNewClusterID.link./fruits
hdfs://10.10.20.10:8020/fruits

Rest you follow the instructions given on apache website

Besides this there are other few good tutorials about Federation

http://blog.cloudera.com/resource/hadoop-world-2011-presentation-video-hdfs-federation/
http://www.slideshare.net/huguk/hdfs-federation-hadoop-summit2011

Leadership lessons

This posts is all what i learned over the years on leadership. I will update it as and when i learn something and i want to be part of my life so that i don't forget that lesson.

Be ready to listen and learn from others , especially juniors

Beautiful data Gapminder.com

Automating tasks using crontab

Crontab provides an easy way to automate the repeated execution of tasks.

Following directories and files are of importance to know

/etc/cron.allow

/etc/cron.deny

The above two files tells which all users are allowed to run and not run cron jobs

/etc/crontab

This file tells what to run and when

The simplest entry by default is

$ sudo cat /etc/crontab

SHELL=/bin/bash

PATH=/sbin:/bin:/usr/sbin:/usr/bin

MAILTO=root

HOME=/

# run-parts

01 * * * * root run-parts /etc/cron.hourly

02 4 * * * root run-parts /etc/cron.daily

22 4 * * 0 root run-parts /etc/cron.weekly

42 4 1 * * root run-parts /etc/cron.monthly

The above is telling that you run-parts will run all the scripts present in cron.hourly directory every hour

cron.daily every day

cron.weekely every week and so on

To simply schedule your job drop your shell script in any of the 4 directories of schedule appropriate for you

The cron will run your job at required time

Last one thing , if you want more granular control of jobs then you can also create own crontab file and run at specific time

Min Hr Day/Month Month Day/Week

30 0 * * * /home/myhome/runThis.sh

The above entry will run runThis.sh every day at 00:30 hours

More more details about crontab you can also read man page

$ crontab man

http://www.vbseo.com/f34/automated-mysql-datestamp-backup-using-cron-via-shell-13426/

Happy Scheduling

Automatically Backup MySql

#! /bin/bash

# Automated database MySQL backup

# The below command will backup one database in SQL which has been gzip.

mysqldump -h IPofMySQLMachine -u username -pPassword databaseName | gzip > /pathToBackupHome/database-`date --iso-8601`.sql.gz

# The below command will backup all databases in SQL which has been gzip.

mysqldump -h IPofMySQLMachine -u username -pPassword --all-databases| gzip > /pathToBackupHome/database-`date --iso-8601`.sql.gz

I have given examples for gzip format. You can skip gzip and pipe if you want plain sql. The first example would change to

# The below command will backup one database in SQL format.

mysqldump -h IPofMySQLMachine -u username -pPassword databaseName > /pathToBackupHome/database-`date --iso-8601`.sql

To Automate this backup process you can execute this shell script using crontab

See mysqldump manual for more details

http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html

Machine learning with Hama

Machine learning with Apache Hama from Tommaso Teofili

Java Tutorials in Kindle , eReader , iPad

Do you know Oracle has started providing all the Java Tutorials which are present on its website in ePub and mobi format?

This has been pretty good surpsise for me.

You can download all of them at

http://www.oracle.com/technetwork/java/javase/downloads/java-se-7-tutorial-2012-02-28-1536013.html

Check for the latest version of above page.

The best way is to go to Tutorials first

http://docs.oracle.com/javase/tutorial/index.html

and then on right side

There is section of eBook Download

Click there

Happy Learning :)