Scala in Action Chapter 1 Why scala notes

Mixin : Class that provides certain functionality that can be inherited by subclass. But it is not made to be instantiated self. It can be seen as interface with implemented methods

Self Type : A mixin doesnt depend on any method or fields of class that it is mixed into. Sometimes it is useful to use those. This is known as self Type

Type abstraction
  • Parametrization
  • Abstract members

Concepts

  • Referential transparency : Value can be replaced by expression which calculated that value
  • Higher order functions : Functions that take functions as input
  • Lexical closures
  • Pattern matching
  • Single assignment (val)
  • LAzy evaluation
  • Type inference
  • Tail call recursion
  • List comprehension
  • Monodaic effects
  • No side effects
Types of languages

Static
Variables have type , values have type and are checked at compile time
Dynamic
Values have types , variables dont have type

Type inference : Compiler try to infer the type at compile time

Macros : Functions loaded at compile time and used by compiler. Compile time meta programming

Design and Architecture considerations for storing time series data



Design and Architecture considerations for storing time series data

Know your Access patterns in advance

  • Are we going to do analysis on full day of data or data for just one hour. Having advance notes on what would be the use cases with which data will be used is highly recommended. 
  • The granularity of information required by the client application helps deciding underlining data model for storing the information.
  • Frequency with which data is generated
  • Identify the speed with which time is being produced by the source system. Do you produce multiple data points every second.

Though we might need to persist all the time series data but more than often we don't need to store each data point as a separate record in the database.

Most of the time series problems are similar in nature and predominant issues comes when we need to Scale the system . Making  the systems evolve with changing schema is another dimension which adds to the complexity. All the problem show similar patterns with only variations in the data model.

If we can define a time window and store all the readings for that period of time as an array then
we can significantly cut the number of actual records persisted in the database, improving the performance of the system.

For example
Stocktock tick information is generated once a second for each stock i.e. 86K ticks per stock per day.
If we store that many records as separate rows then the time complexity to access this information would be huge, so we can group 5 minutes or 1 hour or one day worth of records as a single vector record.

The benefits of storing information in larger chunks is obvious as you would do way fewer lookups into the
NoSQL store to fetch the information for a specific period of time. Other point to remember is that
if your window size is very small then you will be doing a lot of read/write operation and if it is
too big then durability would be a concern as you can lose the information in the event of system failure.

So you need to balance out both the forces.

There is no single size fit each time series problem is different.Fine tune the system based on the requirements and the access patterns.

If the access pattern changes in future then you might have to re-index, re-calculate the array size to optimize your queries.

So each time-series application is very custom made, where you can apply the best practices but cannot just import the data modelling templates to a different time-series problem.

Links
http://www.infoq.com/articles/nosql-time-series-data-management
https://blog.twitter.com/ko/node/4403











Migrating Large Hadoop Cluster

“The text book launch” , any Indian would have heard this phrase The text book launch, often used by ISRO after successful launch of satellites into space.

Last weekend we did similar exercise , The textbook hadoop cluster migration. This blog posts shares thoughts and experiences around the same.

Just for story making i am masking customer specific details.

Epilogue

It will be two years this June (2014) when I landed in this totally new city for me , Sydney. I came to do implementation of Hadoop for this large financial customer. This journey has been great in-fact awesome learning experience and opportunity to work with best brains of the world.

The Weekend Task

We had to migrate to new cluster as we were almost out of space and also needed more computation power. While writing this blog post i used the term WE , it has been combined effort of awesome bunch of people.

Old Cluster configuration 11 Nodes having about 250 TB space

New Cluster 25 Nodes

The two cluster were running different versions of Hadoop , so move was from CDH 4.3 to CDH 4.6 and luckily there was binary compatibility between these two releases.

The cluster also runs Hbase in production and has replica mirror cluster ( 3rd cluster)

The Plan

This activity had in background lots of planning around

  • Testing of the sample production jobs on new cluster
  • Total Automation and setup of new cluster via Cloudera manager API
  • Code Migration of over 300 production oozie jobs which pump data into cluster and produce scores
  • Trickle feed resuming the jobs in batches
  • Roaster for shifts in which people will work to complete the activity within one weekend

Good clear communication on what is happening for all the people involved was very important , Given the fact that our team was spread across Sydney and India.

Besides traditional email we also used whatsapp for communication so that all people can be aware of things and can read
messages when they are available. ( not to awake someone from sleep , right after completing roaster shift )

The downtime of the cluster was started from Friday evening , data copy via distcp was started few days back during nights so that in weekend we can use distcp in update mode to transfer the newly added data.

The Action

New Cluster setup

Given the experience with managing old cluster , it was priority of the team that any new cluster bought would be setup via 100% automation , the new cluster configuration , setup is driven via puppet and Cloudera Manager API. The non hadoop components are installed via normal Puppet packages. After the setup of the new cluster via automation the machines were ready to be loaded.

Data Migration

During the Friday night distcp was started in update mode to complete the data copy process moving all onto new cluster. Overnight stay in office was planned and distcp with World Cup soccer match was good combination :) Since HBase is also part of the production we had to move it. To move HBase we used the similar distcp copy process , we had brought down everything (except HDFS and MR) from source cluster. This approach is to be used if and only if the HBase is down on both sides.

See the steps mentioned here on Apache wiki


The distcp was complete by 7:00 AM Saturday morning and full data migration was verified by taking full folder tree dump on source and destination cluster and compared the sizes.

The job of overnight stay team was over and with the dawn of Saturday morning the job of Code migration team was about to begin.

Code Migration

The activity to sample test the existing production jobs was carried out few days in advance. This allowed us to find the issues from binary compatibilities to
network issues for Sqoop jobs which needs firewall to be opened for talking to new cluster.

Given the large number of production jobs testing and being confident that everything would work 100% in new cluster was challenging task.

One of the main focus was to capture what's running at this moment in the cluster ( Thanks to our very strict Development manager :) ) We ran extensive checks to capture the current state of code in the system and moved back to code repository.

From over 300 jobs we found 3 oozie workflows which were having
definitions out of date in our code repo from what was running in production. With large number of property files , oozie workflow testing can be difficult.

I will write a new blog post of learning about best practices in handling large number of oozie workflows , especially around regression testing and structuring the oozie code.
Oozie is awesome tool , we never had issues and many of jobs were running silently from over 12 months now.

We created new branch after dumping the current state of production cluster code and started changing the old code base to configuration specifications of new cluster.

By late Saturday afternoon the code migration was complete and ready to be run onto the new cluster. We ran our first job onto new cluster with success and shared the news with all over watsapp.

Whatsapp group messaging has worked very well I think , keeping all team members aware of current happening in base camp.

By late evening work of code migration shift was over and new team arrived to take over for resuming the production jobs.

Trickle resuming the production jobs

The cluster is under SLA for downstream consumers , we started resuming the production jobs which were having highest priority. The team managing the platform for operations had now control in hands. They started verifying the oozie jobs after starting them incrementally into new cluster. We have our own custom job tracking database , so writing one simple query into MySQL gave clear view which job is having problem and needs attention.

Hive and Metadata

We took the dump of Hive MySQL metastore from the old cluster , created corresponding database on the new cluster. Since there is mirror cluster we also configured MySQL replication for the same.

The closing thoughts

We did everything without any support call , ticket , Apache mailing list email. This shows that our team is capable and learned enough to deal with wide range of things now in the Hadoop ecosystem.

However there are few lessons which we need to learn from and make truly the textbook hadoop migration.

There is always something which is not in code base. Getting discipline in the large team that everything lands up into code base is the most difficult thing.
The last minute fixes often lead to changes which are running into Production but never find the way into source control. Hence we missed few minor things.
Although the changes did not affect our planned time lines since we knew how to fix the issues ( since we had seen time earlier too ) , we fixed them and task of adding the
fixes back to source control is the action point.

Automation , is the key to manage large number of multiple clusters. Platform team did awesome job in capturing the current state of old cluster and tuning required
properties in new cluster. All the deployment of new cluster was done via Puppet and Cloudera Manager API. So any configuration changes were also driven by the same.
The properties which code migration team found were missing during actual resumption of production jobs were passed back into the loop to be added to puppet.

Permissions , the jobs which failed running in the new cluster were due to permission issues. One of the action point we noted was to how to capture this information back into our Puppet so that all future deployments take care of folders , owners and permissions for us.

Oozie code structuring , Managing oozie code given the large amount of xmls and configurations can be difficult and this can be annoying at times. Understanding of concept of oozie globals , share lib and  bundles is very important given the fact when you are using it in production for large deployments. I will write additional followup post for oozie code.

-----

Good bye for now , soon would be back with another post for the next task for moving on to YARN by end of next quarter.

Hive Warning: Value had a \n character in it

You stare a hive query it gives following warning

Warning: Value had a \n character in it

Check that your query has proper semi colons for all statements.

Git Tutorial

git init learngit

This will create a new git repo in the folder named learngit.

We are ready now to use git within our local system.

When we say git init , it creates a directory named .git inside the folder.
This contains the stuff needed by git to work.

If you want to copy existing code base in git to your local system , you have another way.

git clone url_name

Create a new file named readme.txt

Add some content to it

git add readme.txt

This command it telling that i want this file to be considered for saving it git.
When we say this , git stores that to index.

Commit the file

git commit --message "Adding a new file"

This command saves the file which we added to index into the git.

git log

Git log shows the history about what work has been done and by whom

git diff

We can use this command to find difference the two commits

git diff master~1 master

Another similar syntax is
master^1


This will show changes to all the files onto master on current version from previous commit

git diff master~1 master --filename.txt

If you want to see diff only for specific file you can use above command.

git diff

without an argument views the differences between the current
working directory and the index staging area.

git diff master

views the differences between the current working directory and the last commit on the
default master branch


While working with word documents you might want to use the

--word-diff

This shows diff at the word level rather then lines

git diff --word-diff master~1 master --filename.txt

Another format of diff is

--stat

This command shows the summary in terms of line changes + and - , rather than details about changes.


To see what is the sha for given reference (branch or master)
use the following example

git rev-parse master

Sbt in Action Summary Notes

Notes from Book Sbt in Action

http://www.manning.com/suereth2/


Chapter 1

Maven concepts

Maven has set of lifecycles

Lifecycle
              Phases
                        Goals ( Tasks in Ant)


Maven has following lifecycles

default ( build and deployment )
clean ( used to clean )
site ( used for documentation )

Each phase can have number of goals
Phases are executed in sequence
All previous phases ( and goals ) are executed if you call some phase

You can assign goals to phases via plugins goals

e.g Maven syntax


SBT is better for

Type safety
Ease of making new tasks
Test and development features for incremental builds

SBT concepts

Sbt just has tasks
Tasks can have dependency with each other.
Tasks with no dependency are executed in parallel
Tasks output can be passed on to next output

Settings
Settings are values ( e.g ) Project name

Plugins
If something is reusable tasks which is done again and again in different projects , we can abstract it as plugin.

This plugin can be used in different projects and can be sourced by build file of projects.


Chapter 2

Download and add sbt to your path

http://www.scala-sbt.org/download.html

Every project using sbt should have two files:

• project/build.properties
• build.sbt

build.properties tell which version of sbt to use
build.sbt tells which all settings are applicable for this project , including the library dependencies

start sbt by typing sbt at command prompt


After sbt starts , you can see the help by typing

> help

This will show all of the help topics

> help

  help                                    Displays this help message or prints detailed help on requested commands (run 'help <command>').
  about                                   Displays basic information about sbt and the build.
  tasks                                   Lists the tasks defined for the current project.
  settings                                Lists the settings defined for the current project.
  reload                                  (Re)loads the project in the current directory
  projects                                Lists the names of available projects or temporarily adds/removes extra builds to the session.
  project                                 Displays the current project or changes to the provided `project`.
  set [every] <setting>                   Evaluates a Setting and applies it to the current project.
  session                                 Manipulates session settings.  For details, run 'help session'.
  inspect [uses|tree|definitions] <key>   Prints the value for 'key', the defining scope, delegates, related definitions, and dependencies.
  <log-level>                             Sets the logging level to 'log-level'.  Valid levels: debug, info, warn, error
  ; <command> (; <command>)*              Runs the provided semicolon-separated commands.
  ~ <command>                             Executes the specified command whenever source files change.
  last                                    Displays output from a previous command or the output from a specific task.
  last-grep                               Shows lines from the last output for 'key' that match 'pattern'.
  export <tasks>+                         Executes tasks and displays the equivalent command lines.
  exit                                    Terminates the build.
  --<command>                             Schedules a command to run before other commands on startup.
  show <key>                              Displays the result of evaluating the setting or task associated with 'key'.


> tasks

This will show which all tasks are defined for this project

> settings

This will show which all settings are applicable for this project

For example which is sScalaversion etc

> settings

This is a list of settings defined for the current project.
It does not list the scopes the settings are defined in; use the 'inspect' command for that.

  autoCompilerPlugins            If true, enables automatically generating -Xplugin arguments to the compiler based on the classpath for the plugin configuration.
  autoScalaLibrary               Adds a dependency on scala-library if true.
  baseDirectory                  The base directory.  Depending on the scope, this is the base directory for the build, project, configuration, or task.
  classDirectory                 Directory for compiled classes and copied resources.
  crossPaths                     If true, enables cross paths, which distinguish output directories for cross-building.
  fork                           If true, forks a new JVM when running.  If false, runs in the same JVM as the build.
  initialCommands                Initial commands to execute when starting up the Scala interpreter.
  javaHome                       Selects the Java installation used for compiling and forking.  If None, uses the Java installation running the build.
  javaSource                     Default Java source directory.
  libraryDependencies            Declares managed dependencies.
  managedResourceDirectories     List of managed resource directories.
  maxErrors                      The maximum number of errors, such as compile errors, to list.
  name                           Project name.
  offline                        Configures sbt to work without a network connection where possible.
  organization                   Organization/group ID.
  publishArtifact                Enables (true) or disables (false) publishing an artifact.
  publishTo                      The resolver to publish to.
  resourceDirectory              Default unmanaged resource directory, used for user-defined resources.
  scalaHome                      If Some, defines the local Scala installation to use for compilation, running, and testing.
  scalaSource                    Default Scala source directory.
  scalaVersion                   The version of Scala used for building.
  sourceDirectories              List of all source directories, both managed and unmanaged.
  sourceDirectory                Default directory containing sources.
  target                         Main directory for files generated by the build.
  unmanagedBase                  The default directory for manually managed libraries.
  unmanagedResourceDirectories   Unmanaged resource directories, containing resources manually created by the user.
  unmanagedSourceDirectories     Unmanaged source directories, which contain manually created sources.
  version                        The version/revision of the current module.

More settings may be viewed by increasing verbosity.  See 'help settings'.

> settings name
Project name.

More settings may be searched by increasing verbosity.  See 'help settings'.

> settings scalaVersion
The version of Scala used for building.

More settings may be searched by increasing verbosity.  See 'help settings'.

> name
[info] learn_sbt
> scalaVersion
[info] 2.10.3
>


Starting Scala console

> console

This will start scala console where you can run scala code

> scala

To exist the scala console type

:q

If you change build.sbt ( or Build.scala) you have to reload the settings using

> reload

To run tests

> test

To continuously run tasks

Prefix them by ~

e.g

> ~comple

This will keep on compiling code with each change

Auto completion works in sbt 

> test<TAB>

This will show set of possible options with tasks similar to test

To run only particular test in sbt

> testOnly <TESTNAME>


Chapter 8

When sbt reads your project description,

it reads the .sbt files in the root directory,
the .sbt files in /project and
.scala files in /project.

It compiles everything that it finds and this gives you your build definition. sbt then runs this build and creates your artifact, your jar or whatever.

Difference between dependency and plugin

A plugin is simply a jar which contains settings or tasks. The jar contains a definition
file which sbt reads to work out what settings and tasks are available.

To include a plugin in your build,  add the following lines to project/plugins.sbt:

project/plugins.sbt

addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "0.4.0")

resolvers += "sonatype-releases" at
"https://oss.sonatype.org/content/repositories/releases/"

Repositories of plugins and dependeicies are different from each other , it means  If you've a repository which contains both plugins and dependencies of your artifact, you'll need to include in both project/plugins.sbt and build.sbt.


Version

When sbt resolves the plugin, it appends both the scala version and the sbt version. But the
appended scala version number is not the scala version that you've declared in the build.sbt,
but the version of scala which is used by sbt.

Example

build.sbt

scalaVersion := "2.9.2"

and you are running sbt version 0.13, then the version appended by addSbtPlugin will be
2.10, not 2.9.2, because sbt 0.13 uses version 2.10 of scala. Similarly, if you are running
sbt version 0.12 and you have scalaVersion := "2.10", then the version appended will be
2.9.

Create your own plugin

Define the code in

project/MyPlugin.scala

In build.sbt

Declare the plugin

Call the plugin defined in project/MyPlugin.scala

Credentials

To add credentials for some repo

Use the file

$HOME/.sbt/0.13/credentials.sbt








SSH multihop from one server to another

At my work i login to server via an intermediary server.

So to go to machine i have to first 

Laptop > Inter > WorkMachine

Lets abbrevate the as

L > I > W

So to directly login via ssh from my L to W i setup multi hop ssh

First we need to generate some key on L

Laptop changes

ssh-keygen -t rsa -C "your_email@example.com"

Then add your new key to the ssh-agent:

# start the ssh-agent in the background
eval "$(ssh-agent -s)"
# Agent pid 59566
ssh-add ~/.ssh/id_rsa

Run the following code to copy the key to your clipboard.

pbcopy < ~/.ssh/id_rsa.pub
# Copies the contents of the id_rsa.pub file to your clipboard

Inter changes

Now login to machine Inter (I)

Add the details of newly created ssh key

Run the following code to copy the key to your clipboard.

vi ~/.ssh/authorized_keys
# Paste the contents of clipboard

Save it and exit

Test

Just try to ssh from Laptop to Inter

ssh inter

This should allow you to directly ssh from laptop to inter without password.

Work changes

Now login to WorkMachine and add the key to its authorized keys also following above steps

Laptop changes

In your laptop local ssh config add the following

vi ~/.ssh/config


Paste the content below

 

Host inter

HostName hostname.inter.com

Host workm

Hostname hostname.work.com

ProxyCommand ssh -q inter nc %h 22

 

Final run

To login from laptop directly to WorkMachine

ssh workm