Jugnu Life :-): March 2015

Change jenkins default port Redhat

Open file

vi /etc/sysconfig/Jenkins

Edit the port

# Port Jenkins is listening on. from 8080

Running Docker behind http proxy

Error message

dial tcp: lookup index.docker.io: no such host

HTTP Proxy

This example overrides the default docker.service file.

If you are behind a HTTP proxy server, for example in corporate settings, you will need to add this configuration in the Docker systemd service file.

First, create a systemd drop-in directory for the docker service:

mkdir /etc/systemd/system/docker.service.d

Now create a file called /etc/systemd/system/docker.service.d/http-proxy.conf that adds the HTTP_PROXY environment variable:

[Service]

Environment="HTTP_PROXY=http://proxy.example.com:80/"

If you have internal Docker registries that you need to contact without proxying you can specify them via the NO_PROXY environment variable:

Environment="HTTP_PROXY=http://proxy.example.com:80/" "NO_PROXY=localhost,127.0.0.0/8,docker-registry.somecorporation.com"

Flush changes:

$ sudo systemctl daemon-reload

Restart Docker:

$ sudo systemctl restart docker

From https://docs.docker.com/articles/systemd/

How to create CRAN mirror

We wanted to setup local copy of cran to speed up installs of packages behind the corporate proxy.

Following instructions given on website.

Create simple script

Install Apache http server

sync.sh has contents

rsync -rvCtL --delete --include="*.tar.gz" --include="PACKAGES*" --exclude="*/*" cran.r-project.org::CRAN/src/contrib /var/www/cran

Above script just deletes all the old packges , copies new ones into folder /var/www/cran

You can choose any other location

Rsync used to stop many times due to connection or other issues so i just added it in loop , if exist is not clean run again

call.sh has contents

./sync.sh
while [ $? -ne 0 ]; do
./sync.sh
done

Now just schedule the script call.sh via cron to run at regular intervals , the official website say to run every 2 days.

http://cran.r-project.org/mirror-howto.html

You can now use the above location in contrib url for R packages

Around the world in Bigdata space 4 March 2015

Few links for what’s happening across the world in Bigdata space.

Vendors

Cloudera , Hortonworks , Pivotal war of word

https://gigaom.com/2015/03/03/cloudera-ceo-declares-victory-over-big-data-competition/

Tools

Spark Streaming to process 480K records per second. (Similar requirements what we need for being proactive)

https://www.sigmoid.com/spark-streaming/

How others are solving PCAP and network security problem space ( Being proactive)

https://github.com/OpenSOC

Architecture

Towards future enterprise for unified data flow with Kafta

http://blog.confluent.io/2015/02/25/stream-data-platform-1/

How to do rsync via http proxy

How to do rsync via proxy

Assuming we do HTTP_PROXY something like below

export HTTP_PROXY=http://user:password@proxyhost:port

Then for Rsync do the following

export RSYNC_PROXY=user:password@proxyhost:port

then start your rync command

Error ODBC headers sql.h and sqlext.h not found

Error while installing RODBC package in R

checking for sqlext.h... no

configure: error: "ODBC headers sql.h and sqlext.h not found"

ERROR: configuration failed for package ‘RODBC’

* removing ‘/usr/lib64/R/library/RODBC’

Solution

yum install unixODBC-devel

How to configure HAWQ to talk to RStudio and R

HAWQ is installed in server which is used by analyst to run all queries.

RStudio has plugin named RPostgreSQL we can use that to talk to HAWQ

Here is command run sheet to do the same

Load the Plugin

>library(RPostgreSQL)

Declare the driver

>drv <- dbDriver("PostgreSQL")

Create the connection

con <- dbConnect(drv,host="10.1.1.1",port="5432",user="username",password="mypasswrd",dbname="mydatabase")

Run the query

rs <- dbSendQuery(con,"select count(*) from mytable")

Fetch the results

> fetch(rs,n=-1)

count

1 3713399

RS-PostgreSQL.h:23:26: error: libpq-fe.h: No such file or directory

Error in installing RPostgreSQL

m=ssp-buffer-size=4 -m64 -mtune=generic -c RS-PQescape.c -o RS-PQescape.o

In file included from RS-PQescape.c:7:

RS-PostgreSQL.h:23:26: error: libpq-fe.h: No such file or directory

RS-PQescape.c: In function ‘RS_PostgreSQL_escape’:

RS-PQescape.c:21: error: ‘PGconn’ undeclared (first use in this function)

RS-PQescape.c:21: error: (Each undeclared identifier is reported only once

RS-PQescape.c:21: error: for each function it appears in.)

RS-PQescape.c:21: error: ‘my_connection’ undeclared (first use in this function)

RS-PQescape.c:28: error: expected expression before ‘)’ token

RS-PQescape.c:32: warning: implicit declaration of function ‘PQescapeStringConn’

make: *** [RS-PQescape.o] Error 1

ERROR: compilation failed for package ‘RPostgreSQL’

* removing ‘/usr/lib64/R/library/RPostgreSQL’

The downloaded source packages are in

‘/tmp/RtmpKut82D/downloaded_packages’

Solution

yum install postgresql-devel

Install R packages from proxy

Install R packages from proxy

In the linux shell

export http_proxy=http://username:password@proxy:port
export https_proxy=http://username:password@proxy:port
export HTTP_PROXY=http://username:password@proxy:port

Start R

sudo R

>Sys.setenv(http_proxy="http://username:password@proxy:port")

>install.packages("RODBC", dependencies=TRUE)