Using Komodo edit from command line

Mac

alias komodo='open -a "Komodo Edit 9"'

Now at the command line you can say

komodo hello.txt

Machine learning for large organisations

While doing machine learning and analytics in large organisations often we have to cater many different objectives to fulfil business needs. This post shares my learnings of past while working with large scale organisations who have diverse teams often distributed silos of departments teams who work using similar set of technologies but with varying business goals.

The design of this organisational machine learning pipeline based on following goals:

• Have set of tools which user prefer
• Have unified way of executing the models made by them
• Have quick way of doing A/B testing for the models 
• Have ability to follow continuous integration principles

I will share both open source and the licensed tools which can be used to create such kind of environment.

From high level typical data science flow is

1. Analyse the data
2. Play with it to create machine learning model
3. Export the model in PMML format
4. Execute the PMML model to generate scores in batch or real time

You can have different terms for scores / batch / real time but I will keep above flow for simplicity for now. With both of the patterns (buy or open source) one of the goal was to reuse whatever is available open source and avoid reinventing the wheel.

There are different set of people involved in people who make PMML files and people who execute (operate) PMML files. But there is very fine overlap between them. The people who make the models and hence PMML artefacts are also responsible for running validations pipelines (more on this later)

Analysis and Modelling 

Analyst and Data scientists use various tools for example R , Spark , Scikit-learn etc . Each of them accessed via web front end or native clients provided for them. Majority of the people prefer these three tools hence first goal to support them makes almost 90% people happy. 

PMML export

Most of the tools support export to PMML. The end artefact for the analytics team is the PMML file. See the links below for various tools in references. The only tricky bit is if your model is not supported by PMML. But that is very rare. Most of times people work with regressions. See the complete list of models supported by PMML below. 

Continuous Integration and A/B Testing

Each of the PMML is committed to git repo which produces deployable artefact (rpm) with each A/B test a new release of models is done by teams. Each team has own project so they can make changes / deploy as per their own release cycles. This is also controlled as per environments for production / development etc.

Scoring engine 

Now only last task is to run the PMML file , Spark at this moment cannot run PMML file , it can only produce one.  In the open source world JPMML is the  scoring engine which supports running PMML files. It has Rest API which gives teams ability to trigger models as and when they want. But teams also want to bring back results back to there systems and this is where Spring XD comes to picture. It allows anyone to write simple few lines of statements and do all end to end work.

See sample code for running PMML in Spring XD

analytic-pmml

     --location=/models/iris-flower-naive-bayes.pmml.xml

     --inputFieldMapping=

       'sepalLength:Sepal.Length,

        sepalWidth:Sepal.Width,

        petalLength:Petal.Length,

        petalWidth:Petal.Width'

     --outputFieldMapping='Predicted_Species:predictedSpecies' | SomeSINK“

Many teams want to push results to database , many want in hdfs , others want in Rabbit MQ. 

All they change is the Sink location and Spring XD does the rest.

With REST API integration with applications of business units the modelling and data science pipeline is self service with business empowered enough to make machine learning modes , deploy with CI practices and run as and when they want. 

The time to A/ B test is very less and you can fail fast.

Validation pipelines

Validation pipelines run of same dataset but are run small sample volume and tool of the choice for people use made models. For example I will generate model using scikit-learn , I will export as PMML. I will also run my validation using scikit-learn and compare with what output the real scoring engine gives.

Thanks for reading , please do share your comments and best practices.

References and further readings

PMML

http://www.dmg.org/v4-1/GeneralStructure.html

R PMML package

http://cran.r-project.org/web/packages/pmml/pmml.pdf

Spark PMML export

https://issues.apache.org/jira/browse/SPARK-1406

JPMML

https://github.com/jpmml/jpmml-evaluator

Open scoring engine

http://openscoring.io/

Spring XD PMML execution

http://docs.spring.io/spring-xd/docs/2.0.0.BUILD-SNAPSHOT/reference/html/#analytics

Operating system level tuning for Hadoop

1)

THC should be disabled

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
Add the above to /etc/rc.local

2)

The noatime mount for disks to speed up reads

3)

The open file limit ulimit to high number 32000

4)

Turn off caching on controller

5)

net.core.somaxconn=1024
Socket listen size

6)

Disable SE Linux

SELINUX=disabled


7)

umask 022

8)

sysctl.conf

vm.swappiness=0
vm.overcommit_memory = 1
vm.overcommit_ratio = 100

9)

Disable ip6

10)

Host DNS is properly set

References

http://www.slideshare.net/vgogate/hadoop-configuration-performance-tuning

Passing and set hive conf variables in JDBC and ODBC server

If we want to pass the JDBC and ODBC server with custom hive conf which are normally set via set commands in the script.

The syntax s

jdbc:hive2://<host>:<port>/dbName;sess_var_list?hive_conf_list#hive_var_list

Example can be

jdbc:hive2://foobar:10000/database;auth=noSasl?mapreduce.map.memory.mb=8000;mapreduce.map.java.opts=-Xmx6277m;#foo=bar

Hadoop Yarn tuning calculator

Yarn tuning calculator

I just tarted one excel sheet where we can plug in the nodes numnber of cores , disks , RAM and
it will give us the values for various yarn properties.

It is based on Hortonworks suggestions.

You can see it at

https://goo.gl/gzSz27

Hortonworks also has the python srcipt.

https://github.com/hortonworks/hdp-configuration-utils

But keep in mind the fact that python script does not takes into consideration for things
which are already running in machine.

Those things we can configure manually in excel sheet

Why my Hive Sqoop job is failing

Find few basics about cluster from your Administrator about cluster configuration.

Sample talk Example can be

How many nodes cluster has , what are its configuration

Answer can be

Each node has 120GB RAM . Out of that memory which we can ask for our jobs is about 80GB
We have 14 cpu cores in each datanode , we have 5 right now the maximum we can ask for processing from each datanode is 8 cores

Leaving rest for other processes like OS / Hadoop / Monitoring services

When we run any job in Mapreduce world you will get minimum RAM of 2GB and Max any task can ask for is 80GB ( See the capacity above)

Given the fact that we are running big jobs for one off loads please tell the system to give you higher RAM for your job. ( RAM in increments of 1024 MB)

Besides RAM you can also ask how many CPU cores you want.

The max cores which given node can provide is 8 for processing

This can be controlled via following parameters

Hive jobs

mapreduce.map.memory.mb
mapreduce.reduce.memory.mb
mapreduce.map.java.opts
mapreduce.reduce.java.opts
mapreduce.map.cpu.vcores
mapreduce.reduce.cpu.vcores

Typically if your jobs fails while inserting to hive query please see check if you need to tune any memory parameter. Hive insert jobs are reduce jobs.

Since you are inserting large amount of data in one go you will face the issues of memory overrun.

Always see the logs , its always mentioned there why the job is failing.

Sqoop jobs

Sqoop jobs spawn only map jobs.

So if Sqoop job is not moving through the indicator is following

Memory issue on our side.

So just add the following parameter in the code

-Dmapreduce.map.memory.mb=5120 -Dmapreduce.map.speculative=false

Tune the above 5120 parameter to based no need.


Where to see logs and job status

You can see whats the status of your job and logs at Resource manager

http://resource:8088/cluster

You can also login to Ambari to see what value has been set as default for given property

http://ambari:8080

Ask for Username and password with readonly access from your administrator

Find out what current default values are

mapreduce.map.java.opts=-Xmx5012m
mapreduce.reduce.java.opts=-Xmx6144m
mapreduce.map.memory.mb=4096
mapreduce.reduce.memory.mb=8192

What if , my job is not even accepted by the cluster :)

You are asking for resources which cluster don't have. Means its crossing the max limit of the cluster. So check with your job what its really asking for and what the cluster can provide.

Why is my job being killed ?

If your job is crossing the resource limit which it has originally asked for from the RM the Yarn will kill your job.

You will see something like below in logs

Killing container....

Remember Google is your friend

Moment you see your job has failed see the error from the above logs and search in Google , try to find which parameters people have suggested to change in the job.

Resources

https://altiscale.zendesk.com/hc/en-us/articles/200801519-Configuring-Memory-for-Mappers-and-Reducers-in-Hadoop-2

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/maven/cli/MavenCli : Unsupported major.minor version 51.0

I got the below error on Maven

Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/maven/cli/MavenCli : Unsupported major.minor version 51.0

Solution

Set the Java home 

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home