Why does my LinkedIn inbox always show one unread message ?

Even if I have read all of my Linkedin messages , it still shows one unread message. Strange but true.
I am not sure what is the problem but i can tell how i fixed that.

1. Click on any message that you have already read.
   
2. Mark it as UNREAD
   
3. Click on another message that you have already read.
   
4. Mark it as UNREAD
   

See , if you notice any change in unread messages indicator ?

For me it changed correctly and then I read both messages one by one and it made my all messages as Read.

Have great time using Linkedin :)

Websites for learning Regular expressions

Regular expressions are like swiss knife which can do lot of work in small amount of code.

However knowing them deeply is very important to write code which is optimal.

I found below good websites for them while using and learning about them.

A very good website to have visual debugging for Regular expressions
https://www.debuggex.com/

Website to create and test regular expressions
http://regexr.com/
https://regex101.com/

A good website for Regular expressions tutorials
http://www.rexegg.com/

SBT Native Packager Building Rpm with Python files

I was working on project in which we were having Airflow python code packaged as part of RPMs. By default rpmbuild process compiles the python files which Airflow server does not likes when it seem them. Airflow likes to compile themselves. So rpm job was to ship files which raw.

Here is complete run sheet to make rpm files using Sbt native packager and disable java jar recompress and python compilation.

Edit to file project > plugins.sbt and add the plugin

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.1.1")

Disable Java jar repackaging and Python compilation

Add the following file src/main/rpm/pre

The rpm pre scriplet is executed just before rpm is installed. Read more about rpm scripts at this link.

%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-java-repack-jars[[:space:]].*$!!g' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')

The above script is removing the bra-java-repack-jars and brb-python-bytecompile process from pre step of RPM installation.

To make the pre file work add the following in your build configuration file

import RpmConstants._ 
maintainerScripts in Rpm := maintainerScriptsAppendFromFile((maintainerScripts in Rpm).value)(
  Pre -> (sourceDirectory.value / "main" / "rpm" / "pre")
)

The complete rpm.sbt is shown below

import sbt.Keys._
import NativePackagerHelper._

enablePlugins(UniversalPlugin)
enablePlugins(RpmPlugin, RpmDeployPlugin)

maintainer in Linux := "admin@mail.com"
packageSummary in Linux := "Package code"
packageDescription := s"Application ${name.value}"
rpmVendor := "My company"

version in Rpm <<= version { (v: String) =>
  v.trim.replace("-SNAPSHOT", s".${System.currentTimeMillis}")
}

rpmLicense := Option("Apache")
rpmObsoletes := Seq(s"${name.value}")
// This does not work
rpmBrpJavaRepackJars := false
defaultLinuxInstallLocation := "/var/lib/airflow"
// This allows to override the install location using RPM prefix
rpmPrefix := Some(defaultLinuxInstallLocation.value)
import RpmConstants._

// Remove all jars
mappings in Universal := (mappings in Universal).value.filterNot{
case (file, fname) => fname.endsWith(".jar")}

// Add fat jar
mappings in Universal += {
  val fatJar = (assembly in Compile).value
  fatJar -> s"${name.value}.jar"
}

// Copy contents of dags folder
mappings in Universal ++= directory("src/main/dags")

// Add version.txt file
mappings in Universal += {
  val file = target.value / "version.txt"
  IO.write(file, s"${(version in Rpm).value}")
  file -> "version.txt"
}

import RpmConstants._
maintainerScripts in Rpm := maintainerScriptsAppendFromFile(
(maintainerScripts in Rpm).value)(
  Pre -> (sourceDirectory.value / "main" / "rpm" / "pre")
)

publishTo <<= version { (v: String) =>
  val artifactory = "https://artifactory.com/"
  if (v.trim.endsWith("SNAPSHOT"))
    Some("Artifactory Realm" at artifactory + "artifactory/rpm-snapshots")
  else
    Some("Artifactory Realm" at artifactory + "artifactory/rpm-releases/")
}

Build the rpm using sbt command

rpm:packageBin

To publish the rpm use

rpm:publish

References

http://www.rpm.org/max-rpm/s1-rpm-specref-scripts.html

How to configure SSL/TLS for Jenkins

Configuring SSL/TLS AD signed certificate for Jenkins

Enable only TLS 1.2

JENKINS_JAVA_OPTIONS="-Dhttps.protocols=TLSv1.2 -Djava.awt.headless=true"

Configure the certificate

Covert and export pkcs12 version to import into keystore


sudo openssl pkcs12 -inkey /var/lib/jenkins/ssl/myhost.key -in /var/lib/jenkins/ssl/myhost.cer -export -out /var/lib/jenkins/ssl/myhost.pkcs12
sudo keytool -importkeystore -srckeystore /var/lib/jenkins/ssl/myhost.pkcs12 -srcstoretype pkcs12 -destkeystore /var/lib/jenkins/ssl/jenkins.jks

Enter some password where ever it asks. (I have used jenkins)

Edit the /etc/sysconfig/jenkins with following

JENKINS_ARGS="--httpsKeyStore=/var/lib/jenkins/ssl/jenkins.jks --httpsKeyStorePassword=jenkins --httpsPort=8080"

Printing on mac with command line

Printing from mac command line

Get the printer name

lpstat -v

Send the print and set paper size

lp -d NAMEofPRINTER -o media=A4 ./*.doc

Using Spark with different version of Hive

Spark by default comes packaged with one version of Hive.

If we want to use something different then we need to tell about version and jars.

See example

Default version is 1.2.0
Our cluster version is 0.14.0

The example would be , see the properties in bold.

export SPARK_HOME=/opt/spark/spark-1.5.2-bin-hadoop2.6
cd /opt/spark/spark-1.5.2-bin-hadoop2.6
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HIVE_CONF_DIR=/etc/hive/conf
HIVE_LIB_DIR=/usr/phd/3.0.0.0-249/hive/lib
GUAVA_CLASSPATH=/usr/phd/3.0.0.0-249/hive/lib/guava-11.0.2.jar
hive_metastore_classpath=$HIVE_CONF_DIR:$HIVE_LIB_DIR/*:/usr/phd/3.0.0.0-249/hadoop/*:/usr/phd/3.0.0.0-249/hadoop-mapreduce/*:/usr/phd/3.0.0.0-249/hadoop-yarn/*:/usr/phd/3.0.0.0-249/hadoop-hdfs/*

SPARK_REPL_OPTS="-XX:MaxPermSize=512m" bin/spark-shell \
--master yarn-client \
--packages "com.databricks:spark-csv_2.10:1.2.0" \
--repositories "http:/… \
--files ${SPARK_HOME}/conf/hive-site.xml \
--conf spark.executor.memory=5g \
--conf spark.executor.cores=2 \
--conf spark.driver.memory=10g \
--conf spark.driver.maxResultSize=512m \
--conf spark.executor.instances=2 \
--conf "spark.driver.extraJavaOptions=-Dstack.name=phd -Dstack.version=3.0.0.0-249 -XX:+UseG1GC -Xms2g -Xmx10g -XX:InitiatingHeapOccupancyPercent=35 -XX:ParallelGCThreads=5 -XX:ConcGCThreads=3" \
--conf "spark.yarn.am.extraJavaOptions=-Dstack.name=phd -Dstack.version=3.0.0.0-249" \
--conf "spark.executor.extraJavaOptions=-Dstack.name=phd -Dstack.version=3.0.0.0-249 -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:ParallelGCThreads=5 -XX:ConcGCThreads=3" \
--conf spark.sql.hive.metastore.version=0.14.0 \
--conf spark.sql.hive.metastore.jars=$hive_metastore_classpath \
--conf spark.driver.extraClassPath=$GUAVA_CLASSPATH

Spark FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. Invalid method name: 'alter_table_with_cascade'

Exception

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. Invalid method name: 'alter_table_with_cascade'

Solution

The error indicates the version of Hive metastore is different from jar packaged with Spark
Give the correct Hive jar

Example

export SPARK_HOME=/opt/spark/spark-1.5.2-bin-hadoop2.6
cd /opt/spark/spark-1.5.2-bin-hadoop2.6
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HIVE_CONF_DIR=/etc/hive/conf
HIVE_LIB_DIR=/usr/phd/3.0.0.0-249/hive/lib
GUAVA_CLASSPATH=/usr/phd/3.0.0.0-249/hive/lib/guava-11.0.2.jar
hive_metastore_classpath=$HIVE_CONF_DIR:$HIVE_LIB_DIR/*:/usr/phd/3.0.0.0-249/hadoop/*:/usr/phd/3.0.0.0-249/hadoop-mapreduce/*:/usr/phd/3.0.0.0-249/hadoop-yarn/*:/usr/phd/3.0.0.0-249/hadoop-hdfs/*

SPARK_REPL_OPTS="-XX:MaxPermSize=512m" bin/spark-shell \
--master yarn-client \
--packages "com.databricks:spark-csv_2.10:1.2.0" \
--repositories "http...." \
--files ${SPARK_HOME}/conf/hive-site.xml \
--conf spark.executor.memory=5g \
--conf spark.sql.hive.version=0.14.0 \
--conf spark.executor.cores=2 \
--conf spark.driver.memory=10g \
--conf spark.driver.maxResultSize=512m \
--conf spark.executor.instances=2 \
--conf "spark.driver.extraJavaOptions=-Dstack.name=phd -Dstack.version=3.0.0.0-249 -XX:+UseG1GC -Xms2g -Xmx10g -XX:InitiatingHeapOccupancyPercent=35 -XX:ParallelGCThreads=5 -XX:ConcGCThreads=3" \
--conf "spark.yarn.am.extraJavaOptions=-Dstack.name=phd -Dstack.version=3.0.0.0-249" \
--conf "spark.executor.extraJavaOptions=-Dstack.name=phd -Dstack.version=3.0.0.0-249 -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35 -XX:ParallelGCThreads=5 -XX:ConcGCThreads=3" \
--conf spark.sql.hive.metastore.version=0.14.0 \
--conf spark.sql.hive.metastore.jars=$hive_metastore_classpath \
--conf spark.driver.extraClassPath=$GUAVA_CLASSPATH

References

http://ben-tech.blogspot.com.au/2016/01/using-hivecontext-to-read-hive-tables.html