Jugnu Life :-): HBase

Showing posts with label HBase. Show all posts

org.apache.hadoop.hbase.regionserver.LeaseException lease does not exists

It can be prevented by having

hbase.rpc.timeout > hbase.regionserver.lease.period

Error

org.apache.hadoop.hbase.regionserver.LeaseException: org.apache.hadoop.hbase.regionserver.LeaseException: lease '-8899230701586772940' does not exist

HBase transfer to another cluster using distcp

We had to take full copy of HBase from one cluster to another.

We decided to take brute force approach of copying via Distcp.

Although it’s not recommended but we took it as time was very less and we knew it works very quick.

clusterA

clusterB

Steps

This assumes that you don’t have any tables on destination side. If you have then you need to backup them first.

Stop HBase on both sides this will ensure all the data in memory will be dumped to local disks

Start the distcp to copy data

Commands executed on clusterA side

# Create directory on destination side

sudo –u hdfs hadoop fs –mkdir hdfs://clusterB/hbase_copy_20130320

# Start distcp job

sudo –u hdfs hadoop distcp –update /hbase hdfs://clusterB/hbase_copy_20130320

Commands executed on destination side

Verify that data size matches on both sides

clusterA

hadoop fs -du -h /hbase

clusterB

hadoop fs -du -h /hbase_copy_20130320

Commands executed on destination side clusterB

sudo -u hdfs hadoop fs -chown -R hbase:hbase /hbase_copy_20130320

sudo -u hdfs hadoop fs -mv /hbase /hbase_clusterB_backup

sudo -u hdfs hadoop fs -mv /hbase_clusterB_backup /hbase

Do meta repair

sudo -u hbase hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -base hdfs://clusterB_NamenodeService/hbase

This will take some time , once it’s done.

Restart HBase on destination side and let region balancing happens

you can verify data by

hbase list

Lastly , the recommended approach is snapshots and copytable. But today we did not use these. I will write another post to use snapshots and copytable

Thanks for reading

Possibly transient ZooKeeper exception

[WARN] RecoverableZooKeeper - Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid

[INFO] RetryCounter - Sleeping 4000ms before retry

Solution

Check that you have correct hbase-site.xml in /etc/hbase/conf folder

Or your hbase configuration settings

Property to look for is

<name>hbase.zookeeper.quorum</name>

<value>yournode01,yournode02 </value>

</property>

Kiji Maven Setup

To develop applications using Kiji schema and Mapreduce , add following to your maven

If you have custom changes to your maven settings.xml then add following in relevant sections. Otherwise you can download the settings.xml provided by kiji website

<profile>
     <repositories>
       <repository>
         <id>kiji-repos</id>
         <name>kiji-repos</name>
         <url>https://repo.wibidata.com/artifactory/kiji</url>
       </repository>
     </repositories>
     <pluginRepositories>
       <pluginRepository>
         <snapshots>
           <enabled>true</enabled>
         </snapshots>
         <id>kiji-plugins</id>
         <name>kiji-plugins</name>
         <url>https://repo.wibidata.com/artifactory/kiji</url>
       </pluginRepository>
     </pluginRepositories>
     <id>kiji-profile</id>
   </profile>

<activeProfiles>
<activeProfile>kiji-profile</activeProfile>
</activeProfiles>

Changes to pom.xml

Add the dependency like following

<dependency>
      <groupId>org.kiji.schema</groupId>
      <artifactId>kiji-schema</artifactId>
      <version>1.0.0</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.kiji.mapreduce</groupId>
      <artifactId>kiji-mapreduce</artifactId>
      <version>1.0.0-rc5</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.kiji.platforms</groupId>
      <artifactId>kiji-cdh4-platform</artifactId>
      <version>1.0.0</version>
      <scope>provided</scope>
    </dependency>

For org.kiji.platforms

Please read below , you need to choose right version depending upon your hadoop cluster

https://github.com/kijiproject/kiji-platforms/blob/master/README.md

Installing Kiji Schema and Shell

Make sure that your Hadoop and HBase are running up.

If you need help to install Hadoop and HBase please see following posts

http://jugnu-life.blogspot.com.au/2012/03/hadoop-installation-tutorial.html

http://jugnu-life.blogspot.com.au/2013/03/hbase-pseudo-mode-install.html

Download Kiji Schema and extract it to some location

https://github.com/kijiproject/kiji-schema

Set following variables

export KIJI_HOME="/home/jj/software/wibi/kiji/kiji-schema-1.0.0-rc5"
export PATH=$PATH:$KIJI_HOME/bin

Install Kiji system tables

$ kiji install

jj@jj-VirtualBox:~$ kiji install
Warning: $HADOOP_HOME is deprecated.

Creating kiji instance: kiji://localhost:2181/default/
Creating meta tables for kiji instance in hbase...
13/03/30 18:03:37 INFO org.kiji.schema.KijiInstaller: Installing kiji instance 'kiji://localhost:2181/default/'.
13/03/30 18:03:43 INFO org.kiji.schema.KijiInstaller: Installed kiji instance 'kiji://localhost:2181/default/'.
Successfully created kiji instance: kiji://localhost:2181/default/

Installing Kiji Schema Shell

Download from

https://github.com/kijiproject/kiji-schema-shell

export KIJI_SHELL_HOME="/home/jj/software/wibi/kiji/kiji-schema-shell-1.0.0-rc5"
export PATH=$PATH:$KIJI_SHELL_HOME/bin

Start kiji shell by

jj@jj-VirtualBox:~$ kiji-schema-shell
Warning: $HADOOP_HOME is deprecated.

Kiji schema shell v1.0.0-rc5
Enter 'help' for instructions (without quotes).
Enter 'quit' to quit.
DDL statements must be terminated with a ';'
schema>

Congrats , you have installed Kiji Schema successfully. Lets play :)

HBase pseudo mode install

If you have done already , then install Hadoop by following

http://jugnu-life.blogspot.com.au/2012/03/hadoop-installation-tutorial.html

Lets get HBase working

Download HBase and Zookeeper Tar ball from Apache website

Extract to some place and set environment variables ( in say .profile of your home )

export HBASE_HOME="/home/jj/software/hbase-0.94.5"
export PATH=$PATH:$HBASE_HOME/bin

export ZOOKEEPER_HOME="/home/jj/software/zookeeper-3.4.5"
export PATH=$PATH:$ZOOKEEPER_HOME/bin

HBase settings

Check DNS settings

jj@jj-VirtualBox:~$ cat /etc/hosts
127.0.0.1    localhost
127.0.0.1    jj-VirtualBox

Check both IP should be same , by default in Ubuntu its not.
HBase expects the loopback IP address to be 127.0.0.1. Ubuntu and some other distributions, for example, will default to 127.0.1.1 and this will cause problems for you.

In hbase-env.sh

export JAVA_HOME="/home/jj/software/java/jdk1.6.0_43"

Changes in
hbase-site.xml properties

Add the following

<property>
    <name>hbase.rootdir</name>
    <value>hdfs://localhost:8020/hbase</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>true</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>localhost</value>
</property>

In HDFS

Create hbase directory in HDFS

$ hadoop fs -mkdir /hbase

Zookeeper settings

In conf directory of Zookeeper

Rename zoo_sample.cfg
to zoo.cfg

Change the path of

dataDir=/home/jj/software/hadoopData/zookeeper

We are ready to test

Start Hadoop

$ start-all.sh

Warning: $HADOOP_HOME is deprecated.

starting namenode, logging to /home/jj/software/hadoop-1.0.4/libexec/../logs/hadoop-jj-namenode-jj-VirtualBox.out
localhost: starting datanode, logging to /home/jj/software/hadoop-1.0.4/libexec/../logs/hadoop-jj-datanode-jj-VirtualBox.out
localhost: starting secondarynamenode, logging to /home/jj/software/hadoop-1.0.4/libexec/../logs/hadoop-jj-secondarynamenode-jj-VirtualBox.out
starting jobtracker, logging to /home/jj/software/hadoop-1.0.4/libexec/../logs/hadoop-jj-jobtracker-jj-VirtualBox.out
localhost: starting tasktracker, logging to /home/jj/software/hadoop-1.0.4/libexec/../logs/hadoop-jj-tasktracker-jj-VirtualBox.out

Check Hadoop pages

localhost:50030
localhost:50070

All fine ?

Let start HBase

HBase automatically starts zookeeper also so no need to start by own

$ start-hbase.sh

localhost: starting zookeeper, logging to /home/jj/software/hbase-0.94.5/bin/../logs/hbase-jj-zookeeper-jj-VirtualBox.out
starting master, logging to /home/jj/software/hbase-0.94.5/logs/hbase-jj-master-jj-VirtualBox.out
localhost: starting regionserver, logging to /home/jj/software/hbase-0.94.5/bin/../logs/hbase-jj-regionserver-jj-VirtualBox.out

Check HBase pages

Master
localhost:60010
Region Server

localhost:60030

All done :)

Struck somewhere ?

Post below

GUI Graphical Interface for HBase region servers

Hannibal is a tool to help monitor and maintain HBase-Clusters that are configured for manual splitting.

This helps us to answer following questions

How well are regions balanced over the cluster?
How well are the regions split for each table?
How do regions evolve over time

To install Hannilal follow the following steps , it wont take long.

https://github.com/sentric/hannibal

1)

Download the latest version

$ git clone https://github.com/sentric/hannibal.git

$ cd hannibal

2)

Edit your .profile or /etc/environment your choice

To include the following property

Change the version depending on your HBase version , options are 0.94 , 0.90 , 0.94

export HANNIBAL_HBASE_VERSION=0.94

3)

Copy the hbase-site.xml from your HBase conf to Hanniball conf directory

4) Build the project

It will take sometime as it will download dependencies from internet

$ ./build

After it show message as Success

5) Start the server

$ ./start

It will take sometime for server to start and then you can monitor it at

http://localhost:9000

You can configure the port incase you already have something running there.

Please note that history data about regions is only collected while the application is running, it will need to run for
some time until the region detail graphs fill up.

Happy Hadooping :)

How HBase major compaction works

Compaction is the process in which HBase combines small files (HStoreFiles) into bigger ones.

Its of two types

Minor : When it take FEW number of files which are placed together and make them one.

Major : When it takes all the files in region and make them one.

This post covers the major compaction.

If you want to read about minor compaction , please read other post. How HBase minor compaction works . I suggest you to read that first.

The following properties effect major compaction

hbase.hregion.majorcompaction

The time (in miliseconds) between 'major' compactions of all HStoreFiles in a region. Default: 1 day. Set to 0 to disable automated major compactions.

Default: 86400000

hbase.server.compactchecker.interval.multiplier

This property affects decision to get the number that determines how often (time interval) we scan to see if compaction is necessary.

The interval between checks is hbase.server.compactchecker.interval.multiplier multiplied by hbase.server.thread.wakefrequency.

hbase.server.thread.wakefrequency

Time to sleep in between searches for work (in milliseconds). Used as sleep interval by service threads such as log roller.

Default: 10000

Quoting from ( Discussion specific stuff i have removed )

http://apache-hbase.679495.n3.nabble.com/Major-Compaction-Concerns-tp3642142p3645444.html

Major compactions are triggered by 3 methods: user issued, timed, and size-based.

Even if we disable time based major compaction we can hit size-based compactions where your config is disabling time-based compactions. Minor compactions are issued on a size-based threshold.

The algorithm sees if sum(file[0:i] * ratio) > file[i+1] and includes file[0:i+1] if so.

This is a reverse iteration, so the highest 'i' value is used. If all files match, then you can remove delete markers [which is the difference between a major and minor compaction]. Major compactions aren't a bad or time-intensive thing, it's just delete marker removal.

Minor compactions will usually pick up a couple of the smaller adjacent StoreFiles and rewrite them as one. Minors do not drop deletes or expired cells, only major compactions do this.

Now that you have read what major and minor compaction is , optimizing the above parameters based on cluster profile is necessary which we would see in other post.

Happy Hadooping :)

How HBase minor compaction works

Compaction is the process in which HBase combines small files (HStoreFiles) into bigger ones.

Its of two types

Minor : When it take FEW number of files which are placed together and make them one.

Major : When it takes all the files in region and make them one.

This post covers the minor compaction.

If you want to read about major compaction , please read other post. How HBase major compaction works . I suggest you to read minor compaction first.

Lets see what decides the term FEW in minor compaction

The following properties effect minor compaction

hbase.hstore.compaction.min	Minimum number of StoreFiles per Store to be selected for a compaction to occur (default 2).
hbase.hstore.compaction.max	Maximum number of StoreFiles to compact per minor compaction (default 10).
hbase.hstore.compaction.min.size	Any StoreFile smaller than this setting with automatically be a candidate for compaction.
hbase.hstore.compaction.max.size	Any StoreFile larger than this setting with automatically be excluded from compaction
hbase.store.compaction.ratio	Ratio used in compaction file selection algorithm

The file which would be used for minor compaction is decided based on following logic

Note the size of file

selects a file for compaction when the file size <= sum(smaller_files_size) * hbase.hstore.compaction.ratio.

Quoting example from official book

Consider following configuration settings

    hbase.store.compaction.ratio = 1.0f
    hbase.hstore.compaction.min = 3 (files)
    hbase.hstore.compaction.max = 5 (files)
    hbase.hstore.compaction.min.size = 10 (bytes)
    hbase.hstore.compaction.max.size = 1000 (bytes)

The following StoreFiles exist: 100, 50, 23, 12, and 12 bytes apiece (oldest to newest). With the above parameters, the files that would be selected for minor compaction are 23, 12, and 12.

Why?

Remember the logic

selects a file for compaction when the file size <= sum(smaller_files_size) * hbase.hstore.compaction.ratio.

    100 --> No, because sum(50, 23, 12, 12) * 1.0 = 97.
    50 --> No, because sum(23, 12, 12) * 1.0 = 47.
    23 --> Yes, because sum(12, 12) * 1.0 = 24.
    12 --> Yes, because the previous file has been included, and because this does not exceed the the max-file limit of 5
    12 --> Yes, because the previous file had been included, and because this does not exceed the the max-file limit of 5.

Hope this helps in understanding HBase minor compaction

Hadoop Hadooping :)

HBase Default Configuration list

HBase Default Configuration list

The page below explains the default property files of HBase , what are there purpose.

The documentation says its build from source , so it should be always latest.

http://hbase.apache.org/book/config.files.html

I know you are source control guy , always want to see trunk data.

So here is that direct link

http://svn.apache.org/repos/asf/hbase/trunk/hbase-common/src/main/resources/hbase-default.xml