Building mesos from source

Install the required software for building

Download JDK from Oracle website and set following env variables

export JAVA_HOME=/home/jagat/development/tools/jdk1.6.0_45
export PATH=$PATH:$JAVA_HOME/bin

Install software for build stuff

Copy paste this

sudo apt-get install -y cmake git-core git-svn subversion checkinstall build-essential dh-make debhelper ant ant-optional autoconf automake liblzo2-dev libzip-dev sharutils libfuse-dev reprepro libtool libssl-dev asciidoc xmlto ssh curl

sudo apt-get install -y devscripts

sudo apt-get build-dep pkg-config dh-autoreconf python-dev libcurl4-openssl-dev libboost-all-dev libunwind8-dev


Clone the code

git clone https://git-wip-us.apache.org/repos/asf/mesos.git mesos


Go to directory where code is copied

cd mesos

All of the commands below should finish without errors.

./bootstrap


./configure

make



-----------------

Some of the errors i got and solved

mbolic-functions
checking consistency of all components of python development environment... no
configure: error: in `/home/jagat/development/code/berkley/mesos':
configure: error:
  Could not link test program to Python. Maybe the main Python library has been
  installed in some non-standard library path. If so, pass it to configure,
  via the LDFLAGS environment variable.
  Example: ./configure LDFLAGS="-L/usr/non-standard-path/python/lib"
  ============================================================================
   ERROR!
   You probably have to install the development version of the Python package
   for your distribution.  The exact name of this package varies among them.
  ============================================================================

See `config.log' for more details
jagat@nanak-P570WM:~/development/code/berkley/mesos$ apt-cache search python27



Solution
Install python-dev


sudo apt-get install python-dev



checking for curl_global_init in -lcurl... no
configure: error: cannot find libcurl
  -------------------------------------------------------------------
  You can avoid this with --without-curl, but it will mean executor
  and task resources cannot be downloaded over http.
  -------------------------------------------------------------------



Solution
sudo apt-get instal libcurl4-openssl-dev




checking whether -pthread is sufficient with -shared... yes
checking for backtrace in -lunwind... no
configure: error: failed to determine linker flags for using Java (bad JAVA_HOME or missing support for your architecture?)


Solution
Download JDK from

set env variable as

export JAVA_HOME=/home/jagat/development/tools/jdk1.6.0_45


export PATH=$PATH:$JAVA_HOME/bin


----


And in end it fails wiht this message

cc1plus: all warnings being treated as errors
make[2]: *** [sched/libmesos_no_3rdparty_

la-sched.lo] Error 1
make[2]: Leaving directory `/home/jagat/development/tools/mesos-0.13.0/build/src'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/home/jagat/development/tools/mesos-0.13.0/build/src'
make: *** [check-recursive] Error 

Solution

Read this
http://www.mail-archive.com/dev@mesos.apache.org/msg02267.html

or shortcut

open

mesos/src/Makefile.am:

Replace 

MESOS_CPPFLAGS += -Wall -Werror

With below line

MESOS_CPPFLAGS += -Wall -Werror -Wno-unused-local-typedefs. 

YARN vs Mesos

A very good discussion on the same topic is present on Quora

http://www.quora.com/How-does-YARN-compare-to-Mesos


Mesos is a meta, framework scheduler rather than an application scheduler like YARN
 

Besides the above link following additional (updated) info i found which you might find useful.

There might be many other things as open source community moves very fast and this post also might be very old while you are reading.

With changes in Capacity scheduler now Yarn can support CPU also as resource scheduler. See JIRA YARN-2 for details.

Yarn now has support for cgroups in containers. A very good related blog post

Storm on Yarn can now directly used

Starting 0.6 Spark on Yarn is now offically supported

GSOC project to add security to Mesos related to adding security features to Mesos which its lacking currently and Yarn has that via Kerberos. Wiki on Mesos security website

Lastly  papers

Google Omega
http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/Schwarzkopf.pdf

This paper is based on research done in Amplabs and Google for next generation schedulers on parallel infrastructures.

Mesos
http://bnrg.cs.berkeley.edu/~adj/publications/paper-files/nsdi_mesos.pdf

YARN
http://www.socc2013.org/home/program/a5-vavilapalli.pdf



It classifies the schedulers into following types

Monolithic schedulers use a single, centralized scheduling algorithm for all jobs (our existing
scheduler is one of these).

Two-level
schedulers have a single active resource manager that offers compute resources to multiple parallel, independent “scheduler frameworks”, as in Mesos and Hadoop-on-Demand (HPC)

The paper classifies Yarn as Monolithic scheduler and Mesos onto Two level scheduler.

It is an interesting read and also raises one question for Yarn

I quote

It might appear that YARN is a two-level scheduler, too. In YARN, resource requests from per-job
application masters are sent to a single global scheduler in the resource master , which allocates resources on various machines, subject to application-specified constraints. But the application masters provide job-management services, not scheduling, so YARN is effectively a monolithic scheduler architecture.
At the time of writing, YARN only supports one resource type (fixed-sized memory chunks). Our experience suggests that it will eventually need a rich API to the resource mastin order to cater for diverse application requirements, including multiple resource dimensions, constraints, and placement choices for failure-tolerance.

Although YARN application masters can request resources on particular machines,it is unclear how they acquire and maintain the state needed to make such placement decisions. 

Google seems to be drifting away from Yarn , unlike its counterpart Yahoo

Quoting Hortonworks from

Office Hours: Q&A on YARN in Hadoop 2

Architecturally how does YARN compare with Mesos?
Conceptually YARN and Mesos address similar requirements. They enable organizations to pool and share horizontal compute resources across a multitude of workloads. YARN was architected specifically as an evolution of Hadoop 1.x. YARN thus tightly integrates with HDFS, MapReduce and Hadoop security.