Build HTTPFs package
Clone the Hadoop code
git clone https://github.com/apache/hadoop.git
cd hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/
mvn package -Pdist -DskipTests
cd target/
tar -cf hadoop-hdfs-httpfs-2.6.0.tar.gz hadoop-hdfs-httpfs-2.6.0
Go to server where you want to setup Https server
Extract the tar file
tar -xvf hadoop-hdfs-httpfs-2.6.0.tar.gz
cd hadoop-hdfs-httpfs-2.6.0
In your cluster manager ( e.g Ambari / Cloudera manager etc)
Make change to core-site.xml so that , httpfs user can do proxy
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>
<value>httpfs-host.foo.com</value>
</property>
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>
<value>*</value>
</property>
Change #HTTPFSUSER# to extact user who will be starting httpfs service.
Restart cluster services
Copy core-site.xml and hdfs-site.xml from your cluster to /etc/hadoop directory
Start Httpfs
./sbin/httpfs.sh start
Check
curl -i "http://<HTTPFSHOSTNAME>:14000?user.name=YourHadoopusername&op=homedir"
Output should be
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"homeDir":"http:\/\/<HTTPFS_HOST>:14000\/user\/YourHadoopusername"}
Clone the Hadoop code
git clone https://github.com/apache/hadoop.git
cd hadoop/hadoop-hdfs-project/hadoop-hdfs-httpfs/
mvn package -Pdist -DskipTests
cd target/
tar -cf hadoop-hdfs-httpfs-2.6.0.tar.gz hadoop-hdfs-httpfs-2.6.0
Go to server where you want to setup Https server
Extract the tar file
tar -xvf hadoop-hdfs-httpfs-2.6.0.tar.gz
cd hadoop-hdfs-httpfs-2.6.0
In your cluster manager ( e.g Ambari / Cloudera manager etc)
Make change to core-site.xml so that , httpfs user can do proxy
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.hosts</name>
<value>httpfs-host.foo.com</value>
</property>
<property>
<name>hadoop.proxyuser.#HTTPFSUSER#.groups</name>
<value>*</value>
</property>
Change #HTTPFSUSER# to extact user who will be starting httpfs service.
Restart cluster services
Copy core-site.xml and hdfs-site.xml from your cluster to /etc/hadoop directory
Start Httpfs
./sbin/httpfs.sh start
Check
curl -i "http://<HTTPFSHOSTNAME>:14000?user.name=YourHadoopusername&op=homedir"
Output should be
HTTP/1.1 200 OK
Content-Type: application/json
Transfer-Encoding: chunked
{"homeDir":"http:\/\/<HTTPFS_HOST>:14000\/user\/YourHadoopusername"}
No comments:
Post a Comment
Please share your views and comments below.
Thank You.