Removing the git merge commit if no commit has been made to master

Removing the merge commit if no commit has been made to master

$ cd gitlearn/

$ ls

$ git init

Initialized empty Git repository in /Users/jaggija/dev_home/code/open/gitlearn/.git/

Create some text file in master

$ pwd

/Users/jaggija/dev_home/code/open/gitlearn

$ vi readme.txt

$ git add readme.txt 

$ git commit -m "Initial commit"

[master (root-commit) 5202907] Initial commit

 1 file changed, 1 insertion(+)

 create mode 100644 readme.txt

Create a new branch and edit the text file

$ git checkout -b "new branch"

fatal: 'new branch' is not a valid branch name.

$ git checkout -b "newbranch"

Switched to a new branch 'newbranch'

$ git checkout newbranch

Already on 'newbranch'

$ vi readme.txt 

$ git add readme.txt 

$ git commit -m "new branch"

[newbranch 0ec90b8] new branch

 1 file changed, 2 insertions(+)

See the log

$ git log

commit 0ec90b8fd3174fe0ad8d0e538f5c7a162a0b2216

Author: Jagat Singh <jaggija@cba.com.au>

Date:   Mon Jul 21 10:17:32 2014 +1000

    new branch

commit 52029078b1635437aec297321c5d23274267e7d6

Author: Jagat Singh <jaggija@cba.com.au>

Date:   Mon Jul 21 10:16:23 2014 +1000

    Initial commit

Switch to master and do the merge

$ git checkout master

Switched to branch 'master'

$ git merge --no-ff newbranch

Merge made by the 'recursive' strategy.

 readme.txt | 2 ++

 1 file changed, 2 insertions(+)

$ git log

commit 4778269d6e3a8be0a68b904653f349b0252736b0

Merge: 5202907 0ec90b8

Author: Jagat Singh <jaggija@cba.com.au>

Date:   Mon Jul 21 10:19:43 2014 +1000

    Merge branch 'newbranch'

commit 0ec90b8fd3174fe0ad8d0e538f5c7a162a0b2216

Author: Jagat Singh <jaggija@cba.com.au>

Date:   Mon Jul 21 10:17:32 2014 +1000

    new branch

commit 52029078b1635437aec297321c5d23274267e7d6

Author: Jagat Singh <jaggija@cba.com.au>

Date:   Mon Jul 21 10:16:23 2014 +1000

See the graph

$ git log --oneline --graph

*   4778269 Merge branch 'newbranch'

|\  

| * 0ec90b8 new branch

|/  

* 5202907 Initial commit

Lets fix the merge commit

$ git reset --hard 5202907

HEAD is now at 5202907 Initial commit

$ git status

On branch master

nothing to commit, working directory clean

Do fast forward merge

$ git merge --ff-only newbranch

Updating 5202907..0ec90b8

Fast-forward

 readme.txt | 2 ++

 1 file changed, 2 insertions(+)

See the log again

$ git log --oneline --graph

* 0ec90b8 new branch

* 5202907 Initial commit

$ git status

On branch master

nothing to commit, working directory clean

$

Done

Functional programming in Haskell notes

Some of the notes and links useful for learning Functional programming in Haskell

http://www.cse.chalmers.se/edu/course/TDA342_Advanced_Functional_Programming/lectures.html
http://www.cs.nott.ac.uk/~gmh/fun.html
http://www.cse.iitk.ac.in/users/ppk/teaching/cs653/notes/single-page.lhs.pdf
http://web.cecs.pdx.edu/~sheard/course/funprog/lecturenotes/
http://verify.rwth-aachen.de/fp14/
http://www.haskell.org/haskellwiki/Functional_programming
http://www.inf.ed.ac.uk/teaching/courses/inf1/fp/
OCaml
http://www.cs.princeton.edu/~dpw/courses/cos326-12/notes/basics.php

Coding standard guidelines with Bigdata projects

Coding standard guidelines with Bigdata projects

With lots of technologies and tools coming into picture it becomes
extremely important that all team members follows the coding guidelines.

I am documenting standards which should be followed.

R

Google R style guide
https://google-styleguide.googlecode.com/svn/trunk/Rguide.xml

Related : RLint: Reformatting R Code to Follow the Google Style Guide
RLint is simple utility to reformat your code to Google R Style guide
https://code.google.com/p/google-rlint/
http://research.google.com/pubs/archive/42577.pdf

Hive

We made our own internal standard to follow.

Python

http://google-styleguide.googlecode.com/svn/trunk/pyguide.html
Google Style VIM settings Python
http://google-styleguide.googlecode.com/svn/trunk/google_python_style.vim

Scala

http://docs.scala-lang.org/style/

http://www.scalastyle.org/
It has plugins for sbt , eclipse etc

Java

http://google-styleguide.googlecode.com/svn/trunk/javaguide.html

Eclipse settings
http://google-styleguide.googlecode.com/svn/trunk/eclipse-java-google-style.xml

Intellij settings Google style guide
http://google-styleguide.googlecode.com/svn/trunk/intellij-java-google-style.xml

Shell scripts

http://google-styleguide.googlecode.com/svn/trunk/shell.xml

Scala in Action Chapter 3 Classes and Objects notes

class MongoDBClinet(val host: String, val port: Int)

val means these values are immutable

Primary constructor - It is called when object is created or called from overloaded constructors

If use var then scala also creates getters and setters
If use val then scala creates only getters , remember val are not changable so no setters

When both val and var are missing the instance values are treated as private and not accessible to anyone outside the class

Section 3.1

Download mongodb from http://www.mongodb.org/downloads

Extract it to some location

Start the mongodb

cd C:\Jagat\tools\mongodb-win32-i386-2.4.9\
mkdir db_data
bin\mongod.exe --db_path=db_data

:http://127.0.0.1:28017/

This will show admin path

The client waits at port  27017

Section 3.2

Classes and Objects

Construct the class with default values for host and port

class MongoClient ( val host : String , val port : Int ) {
def this() = this("127.0.0.1",27017)
}

The first statement in overloaded constructor has to be either other overloaded consturcotr or the primaty constructor.

To do some otherration before you invoke construcotr we use companion objects

Add the mongodb jar driver to class path of REPL session


Add jar to scala REPL classpath

scala> :cp C:\Jagat\tools\scala-eclipse\workspace\scalaination\lib\mongo-java-dr
iver-2.12.0.jar
Added 'C:\Jagat\tools\scala-eclipse\workspace\scalaination\lib\mongo-java-driver
-2.12.0.jar'.  Your new classpath is:

Section 3.3

Packaging

In scala you can have nested packages

package A {

package B {


}


}

You can also use java style packaging with package declared at the top of the file

Scala packaging structure does not have to be matching with the folder structure in file system like Java. But when we compile the classes the required folder structure is auto generated as JVM needs that for working. Remember under the hood scala is running on the top of JVM

To add the jar in scala classpath via command line

scalac -classpath my.jar MyNewClass.sclaa

Section 3.4

Scala imports

You can add import statement at any point in the code.

The import will be visible lexically in the code.

* equivalent of java is _ in scala

To import all use

my.package._

If you declare classes or objects without any package , then they all belong to empty package. The cannot be imported to any other package. But members of empty package can see each other

Remap package scala class to avoid conflicts

import java.sql.{Data => SqlDate }

Hide a class , the Date class below cannot be used

import java.sql.{Data => _ }

Section 3.5

Objects and companion objects

There is no static variables in scala

Scala supports concept of having companion object and classes.

bstract class Role {

  def canAccess(page: String): Boolean
}

class Root extends Role {
  override def canAccess(page: String) = page != "Admin"

}

object Role {

  def apply(roleName: String) = roleName match {
    case "root" => new Root
    case "analyst" => new Root
  }
}

Note the object Role and class Role in above code , they are companion object and companion class

Package objects

Package objects allow you to define something at central place which can be used by all the memebers of the package

They are generally defined in file named

package.scala , in the package that corresponds to it

package object ch3 {
 
  val minAge = 18

}

This above variable can be used anywhere in this package named ch3

Scala in Action Chapter 1 Why scala notes

Mixin : Class that provides certain functionality that can be inherited by subclass. But it is not made to be instantiated self. It can be seen as interface with implemented methods

Self Type : A mixin doesnt depend on any method or fields of class that it is mixed into. Sometimes it is useful to use those. This is known as self Type

Type abstraction

• Parametrization
• Abstract members

Concepts

• Referential transparency : Value can be replaced by expression which calculated that value
• Higher order functions : Functions that take functions as input
• Lexical closures
• Pattern matching
• Single assignment (val)
• LAzy evaluation
• Type inference
• Tail call recursion
• List comprehension
• Monodaic effects
• No side effects

Types of languages

Static
Variables have type , values have type and are checked at compile time
Dynamic
Values have types , variables dont have type

Type inference : Compiler try to infer the type at compile time

Macros : Functions loaded at compile time and used by compiler. Compile time meta programming

Design and Architecture considerations for storing time series data

Design and Architecture considerations for storing time series data

Know your Access patterns in advance

• Are we going to do analysis on full day of data or data for just one hour. Having advance notes on what would be the use cases with which data will be used is highly recommended. 
• The granularity of information required by the client application helps deciding underlining data model for storing the information.
• Frequency with which data is generated
• Identify the speed with which time is being produced by the source system. Do you produce multiple data points every second.

Though we might need to persist all the time series data but more than often we don't need to store each data point as a separate record in the database.

Most of the time series problems are similar in nature and predominant issues comes when we need to Scale the system . Making  the systems evolve with changing schema is another dimension which adds to the complexity. All the problem show similar patterns with only variations in the data model.

If we can define a time window and store all the readings for that period of time as an array then
we can significantly cut the number of actual records persisted in the database, improving the performance of the system.

For example
Stocktock tick information is generated once a second for each stock i.e. 86K ticks per stock per day.
If we store that many records as separate rows then the time complexity to access this information would be huge, so we can group 5 minutes or 1 hour or one day worth of records as a single vector record.

The benefits of storing information in larger chunks is obvious as you would do way fewer lookups into the
NoSQL store to fetch the information for a specific period of time. Other point to remember is that
if your window size is very small then you will be doing a lot of read/write operation and if it is
too big then durability would be a concern as you can lose the information in the event of system failure.

So you need to balance out both the forces.

There is no single size fit each time series problem is different.Fine tune the system based on the requirements and the access patterns.

If the access pattern changes in future then you might have to re-index, re-calculate the array size to optimize your queries.

So each time-series application is very custom made, where you can apply the best practices but cannot just import the data modelling templates to a different time-series problem.

Links
http://www.infoq.com/articles/nosql-time-series-data-management
https://blog.twitter.com/ko/node/4403

Migrating Large Hadoop Cluster

“The text book launch” , any Indian would have heard this phrase The text book launch, often used by ISRO after successful launch of satellites into space.

Last weekend we did similar exercise , The textbook hadoop cluster migration. This blog posts shares thoughts and experiences around the same.

Just for story making i am masking customer specific details.

Epilogue

It will be two years this June (2014) when I landed in this totally new city for me , Sydney. I came to do implementation of Hadoop for this large financial customer. This journey has been great in-fact awesome learning experience and opportunity to work with best brains of the world.

The Weekend Task

We had to migrate to new cluster as we were almost out of space and also needed more computation power. While writing this blog post i used the term WE , it has been combined effort of awesome bunch of people.

Old Cluster configuration 11 Nodes having about 250 TB space

New Cluster 25 Nodes

The two cluster were running different versions of Hadoop , so move was from CDH 4.3 to CDH 4.6 and luckily there was binary compatibility between these two releases.

The cluster also runs Hbase in production and has replica mirror cluster ( 3rd cluster)

The Plan

This activity had in background lots of planning around

• Testing of the sample production jobs on new cluster
• Total Automation and setup of new cluster via Cloudera manager API
• Code Migration of over 300 production oozie jobs which pump data into cluster and produce scores
• Trickle feed resuming the jobs in batches
• Roaster for shifts in which people will work to complete the activity within one weekend

Good clear communication on what is happening for all the people involved was very important , Given the fact that our team was spread across Sydney and India.

Besides traditional email we also used whatsapp for communication so that all people can be aware of things and can read

messages when they are available. ( not to awake someone from sleep , right after completing roaster shift )

The downtime of the cluster was started from Friday evening , data copy via distcp was started few days back during nights so that in weekend we can use distcp in update mode to transfer the newly added data.

The Action

New Cluster setup

Given the experience with managing old cluster , it was priority of the team that any new cluster bought would be setup via 100% automation , the new cluster configuration , setup is driven via puppet and Cloudera Manager API. The non hadoop components are installed via normal Puppet packages. After the setup of the new cluster via automation the machines were ready to be loaded.

Data Migration

During the Friday night distcp was started in update mode to complete the data copy process moving all onto new cluster. Overnight stay in office was planned and distcp with World Cup soccer match was good combination :) Since HBase is also part of the production we had to move it. To move HBase we used the similar distcp copy process , we had brought down everything (except HDFS and MR) from source cluster. This approach is to be used if and only if the HBase is down on both sides.

See the steps mentioned here on Apache wiki

http://wiki.apache.org/hadoop/Hbase/MigrationToNewCluster

The distcp was complete by 7:00 AM Saturday morning and full data migration was verified by taking full folder tree dump on source and destination cluster and compared the sizes.

The job of overnight stay team was over and with the dawn of Saturday morning the job of Code migration team was about to begin.

Code Migration

The activity to sample test the existing production jobs was carried out few days in advance. This allowed us to find the issues from binary compatibilities to

network issues for Sqoop jobs which needs firewall to be opened for talking to new cluster.

Given the large number of production jobs testing and being confident that everything would work 100% in new cluster was challenging task.

One of the main focus was to capture what's running at this moment in the cluster ( Thanks to our very strict Development manager :) ) We ran extensive checks to capture the current state of code in the system and moved back to code repository.

From over 300 jobs we found 3 oozie workflows which were having

definitions out of date in our code repo from what was running in production. With large number of property files , oozie workflow testing can be difficult.

I will write a new blog post of learning about best practices in handling large number of oozie workflows , especially around regression testing and structuring the oozie code.

Oozie is awesome tool , we never had issues and many of jobs were running silently from over 12 months now.

We created new branch after dumping the current state of production cluster code and started changing the old code base to configuration specifications of new cluster.

By late Saturday afternoon the code migration was complete and ready to be run onto the new cluster. We ran our first job onto new cluster with success and shared the news with all over watsapp.

Whatsapp group messaging has worked very well I think , keeping all team members aware of current happening in base camp.

By late evening work of code migration shift was over and new team arrived to take over for resuming the production jobs.

Trickle resuming the production jobs

The cluster is under SLA for downstream consumers , we started resuming the production jobs which were having highest priority. The team managing the platform for operations had now control in hands. They started verifying the oozie jobs after starting them incrementally into new cluster. We have our own custom job tracking database , so writing one simple query into MySQL gave clear view which job is having problem and needs attention.

Hive and Metadata

We took the dump of Hive MySQL metastore from the old cluster , created corresponding database on the new cluster. Since there is mirror cluster we also configured MySQL replication for the same.

The closing thoughts

We did everything without any support call , ticket , Apache mailing list email. This shows that our team is capable and learned enough to deal with wide range of things now in the Hadoop ecosystem.

However there are few lessons which we need to learn from and make truly the textbook hadoop migration.

There is always something which is not in code base. Getting discipline in the large team that everything lands up into code base is the most difficult thing.

The last minute fixes often lead to changes which are running into Production but never find the way into source control. Hence we missed few minor things.

Although the changes did not affect our planned time lines since we knew how to fix the issues ( since we had seen time earlier too ) , we fixed them and task of adding the

fixes back to source control is the action point.

Automation , is the key to manage large number of multiple clusters. Platform team did awesome job in capturing the current state of old cluster and tuning required

properties in new cluster. All the deployment of new cluster was done via Puppet and Cloudera Manager API. So any configuration changes were also driven by the same.

The properties which code migration team found were missing during actual resumption of production jobs were passed back into the loop to be added to puppet.

Permissions , the jobs which failed running in the new cluster were due to permission issues. One of the action point we noted was to how to capture this information back into our Puppet so that all future deployments take care of folders , owners and permissions for us.

Oozie code structuring , Managing oozie code given the large amount of xmls and configurations can be difficult and this can be annoying at times. Understanding of concept of oozie globals , share lib and  bundles is very important given the fact when you are using it in production for large deployments. I will write additional followup post for oozie code.

-----

Good bye for now , soon would be back with another post for the next task for moving on to YARN by end of next quarter.