HDFS Federation was one of the new featruees introduced in Hadoop recently.
If you have read the instructions present on Apache Website
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html
It guides us to configure the Hadoop Federation.
Some of the additional things which are required are telling clients
In core-site.xml
fs.defaultFs
viewFs://ClusterID
Above means that fs.defaultFs should have value of your cluster ID
Additionally we have to tell Clients where to go for which path
fs.viewds.mounttable.MyNewClusterID.link./PATH
hdfs://NN-host1:port/PATH
The above means that for this path on cluster id go to this namenode and path
e.g
fs.viewds.mounttable.MyNewClusterID.link./fruits
hdfs://10.10.20.10:8020/fruits
Rest you follow the instructions given on apache website
Besides this there are other few good tutorials about Federation
http://blog.cloudera.com/resource/hadoop-world-2011-presentation-video-hdfs-federation/
http://www.slideshare.net/huguk/hdfs-federation-hadoop-summit2011
If you have read the instructions present on Apache Website
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html
It guides us to configure the Hadoop Federation.
Some of the additional things which are required are telling clients
In core-site.xml
fs.defaultFs
viewFs://ClusterID
Above means that fs.defaultFs should have value of your cluster ID
Additionally we have to tell Clients where to go for which path
fs.viewds.mounttable.MyNewClusterID.link./PATH
hdfs://NN-host1:port/PATH
The above means that for this path on cluster id go to this namenode and path
e.g
fs.viewds.mounttable.MyNewClusterID.link./fruits
hdfs://10.10.20.10:8020/fruits
Rest you follow the instructions given on apache website
Besides this there are other few good tutorials about Federation
http://blog.cloudera.com/resource/hadoop-world-2011-presentation-video-hdfs-federation/
http://www.slideshare.net/huguk/hdfs-federation-hadoop-summit2011
I downloaded hadoop-0.23.5 and i am not clear on how to start dfs and yarn daemons. The link that you have mentioned above is not very clear. I tried to follow http://hadoop.apache.org/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/SingleCluster.html as well, but i guess also expect the dfs to be running. Any idea, how to get the dfs running on hadoop-0.23.5.
ReplyDeleteFrom the release notes, i also understand that hadoop-0.23.5 does not contain HDFS federation.
So my questions are
1. Which download contains HDFS federation?
2. Any good documentation to get daemons running on HDFS federation?
3. I guess, hadoop-0.23.5 contains only yarn not HDFS federation. Is it right?
4. Is Yarn compatible with older HDFS?
5. Any download which contains both yarn and HDFS federation? I think its in alpha state
1) 2.X.X - current alpha version contains HA
ReplyDelete2) May be i can Jot down instructions , have you checked Apache website?
3) Yes
4) New API started coming with 1.x , how much old you are saying
5) Yes version 2.x
4. Hadoop-1.1.1
DeleteMay be if you blog on how to integrate YARN with HDFS federation or older HDFS (may be 1.1.1), then that will help.