Change jenkins default port Redhat

Open file

vi /etc/sysconfig/Jenkins

Edit the port

# Port Jenkins is listening on. from 8080

Running Docker behind http proxy

Error message

dial tcp: lookup index.docker.io: no such host

HTTP Proxy
This example overrides the default docker.service file.

If you are behind a HTTP proxy server, for example in corporate settings, you will need to add this configuration in the Docker systemd service file.

First, create a systemd drop-in directory for the docker service:

mkdir /etc/systemd/system/docker.service.d
Now create a file called /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:
[Service]
Environment="HTTP_PROXY=http://proxy.example.com:80/"

If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable:

Environment="HTTP_PROXY=http://proxy.example.com:80/" "NO_PROXY=localhost,127.0.0.0/8,docker-registry.somecorporation.com"

Flush changes:
$ sudo systemctl daemon-reload
Restart Docker:
$ sudo systemctl restart docker




How to create CRAN mirror

We wanted to setup local copy of cran to speed up installs of packages behind the corporate proxy.

Following instructions given on website.

Create simple script

Install Apache http server

sync.sh has contents

rsync -rvCtL --delete --include="*.tar.gz" --include="PACKAGES*" --exclude="*/*" cran.r-project.org::CRAN/src/contrib /var/www/cran

Above script just deletes all the old packges , copies new ones into folder /var/www/cran

You can choose any other location

Rsync used to stop many times due to connection or other issues so i just added it in loop , if exist is not clean run again

call.sh has contents

./sync.sh
while [ $? -ne 0 ]; do
    ./sync.sh
done


Now just schedule the script call.sh via cron to run at regular intervals , the official website say to run every 2 days.

http://cran.r-project.org/mirror-howto.html

You can now use the above location in contrib url for R packages






Around the world in Bigdata space 4 March 2015

Few links for what’s happening across the world in Bigdata space.

Vendors

Cloudera , Hortonworks , Pivotal war of word


Tools

Spark Streaming to process 480K records per second. (Similar requirements what we need for being proactive)


How others are solving PCAP and network security problem space ( Being proactive)


Architecture

Towards future enterprise for unified data flow with Kafta



How to do rsync via http proxy


How to do rsync via proxy

Assuming we do HTTP_PROXY something like below

export HTTP_PROXY=http://user:password@proxyhost:port

Then for Rsync do the following

export RSYNC_PROXY=user:password@proxyhost:port

then start your rync command

Error ODBC headers sql.h and sqlext.h not found

Error while installing RODBC package in R

checking for sqlext.h... no
configure: error: "ODBC headers sql.h and sqlext.h not found"
ERROR: configuration failed for package ‘RODBC’
* removing ‘/usr/lib64/R/library/RODBC’


Solution

yum install unixODBC-devel



How to configure HAWQ to talk to RStudio and R

How to configure HAWQ to talk to RStudio and R

HAWQ is installed in server which is used by analyst to run all queries.

RStudio has plugin named RPostgreSQL we can use that to talk to HAWQ

Here is command run sheet to do the same

Load the Plugin
>library(RPostgreSQL)

Declare the driver
>drv <- dbDriver("PostgreSQL")

Create the connection
con <- dbConnect(drv,host="10.1.1.1",port="5432",user="username",password="mypasswrd",dbname="mydatabase")

Run the query
rs <- dbSendQuery(con,"select count(*) from mytable")

Fetch the results
> fetch(rs,n=-1)
    count
1 3713399
> 

RS-PostgreSQL.h:23:26: error: libpq-fe.h: No such file or directory

Error in installing RPostgreSQL

m=ssp-buffer-size=4 -m64 -mtune=generic  -c RS-PQescape.c -o RS-PQescape.o
In file included from RS-PQescape.c:7:
RS-PostgreSQL.h:23:26: error: libpq-fe.h: No such file or directory
RS-PQescape.c: In function ‘RS_PostgreSQL_escape’:
RS-PQescape.c:21: error: ‘PGconn’ undeclared (first use in this function)
RS-PQescape.c:21: error: (Each undeclared identifier is reported only once
RS-PQescape.c:21: error: for each function it appears in.)
RS-PQescape.c:21: error: ‘my_connection’ undeclared (first use in this function)
RS-PQescape.c:28: error: expected expression before ‘)’ token
RS-PQescape.c:32: warning: implicit declaration of function ‘PQescapeStringConn’
make: *** [RS-PQescape.o] Error 1
ERROR: compilation failed for package ‘RPostgreSQL’
* removing ‘/usr/lib64/R/library/RPostgreSQL’

The downloaded source packages are in
        ‘/tmp/RtmpKut82D/downloaded_packages’


Solution

yum install postgresql-devel

Install R packages from proxy

Install R packages from proxy

In the linux shell

export http_proxy=http://username:password@proxy:port
export https_proxy=http://username:password@proxy:port
export HTTP_PROXY=http://username:password@proxy:port

Start R

sudo R

>Sys.setenv(http_proxy="http://username:password@proxy:port")

>install.packages("RODBC", dependencies=TRUE)