How to get internet access inside Jupyter notebook

Create a file

vi ~/.ipython/profile_default/startup/00-startup.py

Add the following inside it
(Change proxy details as per your own environment)

import sys,os,os.path
os.environ['HTTP_PROXY']="http://127.0.0.1:3128"
os.environ['http_proxy']="http://127.0.0.1:3128"
os.environ['HTTPS_PROXY']="http://127.0.0.1:3128"
os.environ['https_proxy']="http://127.0.0.1:3128"

# Test

Inside the notebook cell

import requests
requests.get("http://google.com")

Atom editor proxy

To use Atom editor behind proxy , use the below. Restart Atom if needed

apm config set strict-ssl false
apm config set https-proxy http://127.0.0.1:3128

Change proxy as per your requirements

Use Kafka command line with Kerberos

kdestroy
kinit -k -t myprincipal.keytab myprincipal/HOST.com

export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/etc/kafka/conf/kafka_client_jaas.conf"

/usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh \
--zookeeper zookeeper.com:2181 \
--topic topicname \
--from-beginning \
--security-protocol SASL_PLAINTEXT

Login to Container in Kubernetes

Step 0

Get all pods

kubectl get po --all-namespaces

Find the pod name in which your container is running

Step 1

Get the container name from running pod

kubectl describe po pod_name --namespace my_namespace

Example output

Name: pod_name
Namespace: my_namespace
Node: docker-host-03/10.1.3.115
Start Time: Wed, 23 Nov 2016 16:57:15 +1100
Labels: name=my_container_name,pod-template-hash=2802333548
Status: Running
IP: 10.20.71.4

Note the label with value name

Step 2

Login to container

Example

kubectl exec -it pod_name --namespace my_namespace -c my_container_name bash

Why does my LinkedIn inbox always show one unread message ?

Even if I have read all of my Linkedin messages , it still shows one unread message. Strange but true.
I am not sure what is the problem but i can tell how i fixed that.

1. Click on any message that you have already read.
   
2. Mark it as UNREAD
   
3. Click on another message that you have already read.
   
4. Mark it as UNREAD
   

See , if you notice any change in unread messages indicator ?

For me it changed correctly and then I read both messages one by one and it made my all messages as Read.

Have great time using Linkedin :)

Websites for learning Regular expressions

Regular expressions are like swiss knife which can do lot of work in small amount of code.

However knowing them deeply is very important to write code which is optimal.

I found below good websites for them while using and learning about them.

A very good website to have visual debugging for Regular expressions
https://www.debuggex.com/

Website to create and test regular expressions
http://regexr.com/
https://regex101.com/

A good website for Regular expressions tutorials
http://www.rexegg.com/

SBT Native Packager Building Rpm with Python files

I was working on project in which we were having Airflow python code packaged as part of RPMs. By default rpmbuild process compiles the python files which Airflow server does not likes when it seem them. Airflow likes to compile themselves. So rpm job was to ship files which raw.

Here is complete run sheet to make rpm files using Sbt native packager and disable java jar recompress and python compilation.

Edit to file project > plugins.sbt and add the plugin

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.1.1")

Disable Java jar repackaging and Python compilation

Add the following file src/main/rpm/pre

The rpm pre scriplet is executed just before rpm is installed. Read more about rpm scripts at this link.

%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-java-repack-jars[[:space:]].*$!!g' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')

The above script is removing the bra-java-repack-jars and brb-python-bytecompile process from pre step of RPM installation.

To make the pre file work add the following in your build configuration file

import RpmConstants._ 
maintainerScripts in Rpm := maintainerScriptsAppendFromFile((maintainerScripts in Rpm).value)(
  Pre -> (sourceDirectory.value / "main" / "rpm" / "pre")
)

The complete rpm.sbt is shown below

import sbt.Keys._
import NativePackagerHelper._

enablePlugins(UniversalPlugin)
enablePlugins(RpmPlugin, RpmDeployPlugin)

maintainer in Linux := "admin@mail.com"
packageSummary in Linux := "Package code"
packageDescription := s"Application ${name.value}"
rpmVendor := "My company"

version in Rpm <<= version { (v: String) =>
  v.trim.replace("-SNAPSHOT", s".${System.currentTimeMillis}")
}

rpmLicense := Option("Apache")
rpmObsoletes := Seq(s"${name.value}")
// This does not work
rpmBrpJavaRepackJars := false
defaultLinuxInstallLocation := "/var/lib/airflow"
// This allows to override the install location using RPM prefix
rpmPrefix := Some(defaultLinuxInstallLocation.value)
import RpmConstants._

// Remove all jars
mappings in Universal := (mappings in Universal).value.filterNot{
case (file, fname) => fname.endsWith(".jar")}

// Add fat jar
mappings in Universal += {
  val fatJar = (assembly in Compile).value
  fatJar -> s"${name.value}.jar"
}

// Copy contents of dags folder
mappings in Universal ++= directory("src/main/dags")

// Add version.txt file
mappings in Universal += {
  val file = target.value / "version.txt"
  IO.write(file, s"${(version in Rpm).value}")
  file -> "version.txt"
}

import RpmConstants._
maintainerScripts in Rpm := maintainerScriptsAppendFromFile(
(maintainerScripts in Rpm).value)(
  Pre -> (sourceDirectory.value / "main" / "rpm" / "pre")
)

publishTo <<= version { (v: String) =>
  val artifactory = "https://artifactory.com/"
  if (v.trim.endsWith("SNAPSHOT"))
    Some("Artifactory Realm" at artifactory + "artifactory/rpm-snapshots")
  else
    Some("Artifactory Realm" at artifactory + "artifactory/rpm-releases/")
}

Build the rpm using sbt command

rpm:packageBin

To publish the rpm use

rpm:publish

References

http://www.rpm.org/max-rpm/s1-rpm-specref-scripts.html