Git clone via ssh tunnel

Follow the steps mentioned in below post to configure ssh tunnel
http://jugnu-life.blogspot.com/2015/04/ssh-tunnell.html
Then in git

git config --global http.proxy 'socks5://127.0.0.1:9999'

git config --global https.proxy 'socks5://127.0.0.1:9999'

Now you can do the git clone

SSH Tunnell

I spent lot of time trouble shooting SSH tunnel problem.
I thought to jot down the notes for my future reference
The dynamic ssh tunnel can be set by simple command
ssh -vvvvv -D port username@remotehost
Example
sss -vvvvv -D 9999 jagatsingh@10.20.30.40
Now i firefox use the
Socks 5 proxy setting as
localhost 9999
Keep in mind of unselect all other proxy , e.g HTTP , HTTPS etc
I spent lot of time wasting on this

Errors
channel 2: open failed: administratively prohibited: open failed
debug2: channel 2: zombie
Resolution steps
Check on
/etc/ssh/sshd_config
On the remote host that you have enabled
TCP forwarding.
AllowTCPForwarding yes


The webpage below is good reference
http://www.slashroot.in/ssh-port-forwarding-linux-configuration-and-examples

Parallel download for file wget alternative

http://hortonassets.s3.amazonaws.com/2.2/Sandbox_HDP_2.2_VirtualBox.ova

On mac we can use

brew install aria2

aria2c -x 16 -s 16 http://hortonassets.s3.amazonaws.com/2.2/Sandbox_HDP_2.2_VirtualBox.ova

This will spawn 16 parallel connections

Source

http://stackoverflow.com/questions/3430810/wget-download-with-multiple-simultaneous-connections

Hadoop conf files in Pivotal Hadoop

Pivotal stores files in different location then the default

Actual binaries can be found under path

/usr/lib/gphd/

Example

/usr/lib/gphd/sqoop/

Conf files can be found under path

/etc/gphd

Example

/etc/gphd/sqoop/conf

Open source job dependency tools

I was looking for open source Alternatives for job dependency management.

Few things i found

Taskforest is a simple but expressive open-source job scheduler that allows you to chain jobs/tasks and create time dependencies. It uses text config files to specify task dependencies.

http://www.taskforest.com/


schedulix is the Open Source Enterprise Job Scheduling System, which meets the complex requirements of modern IT process automation.

http://www.schedulix.org/

Some other tools popular in Hadoop world

https://github.com/mesos/chronos

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows.
https://azkaban.github.io/


Luigi

Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
https://github.com/spotify/luigi

Luigi allows you to run batch types of jobs with complex scheduling via Python code.
By default it supports running Hadoop , MySQL , Scalding , Spark etc jobs.
You can see the list of available configurations here
https://luigi.readthedocs.org/en/latest/configuration.html

Change jenkins default port Redhat

Open file

vi /etc/sysconfig/Jenkins

Edit the port

# Port Jenkins is listening on. from 8080

Running Docker behind http proxy

Error message

dial tcp: lookup index.docker.io: no such host

HTTP Proxy

This example overrides the default docker.service file.

If you are behind a HTTP proxy server, for example in corporate settings, you will need to add this configuration in the Docker systemd service file.

First, create a systemd drop-in directory for the docker service:

mkdir /etc/systemd/system/docker.service.d

Now create a file called /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:

[Service]

Environment="HTTP_PROXY=http://proxy.example.com:80/"

If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable:

Environment="HTTP_PROXY=http://proxy.example.com:80/" "NO_PROXY=localhost,127.0.0.0/8,docker-registry.somecorporation.com"

Flush changes:

$ sudo systemctl daemon-reload

Restart Docker:

$ sudo systemctl restart docker

From https://docs.docker.com/articles/systemd/