What is Apache Mesos you ask? Well, from their web site at http://mesos.apache.org/ it is
Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other applications on a dynamically shared pool of nodes.
What exactly does that mean? Well, to me, it means that I can deploy applications and certain frameworks into a cluster of resources (VM’s) and it will look after my resource allocation for me. For example, if you have a particularly computationally intensive task, like a Hadoop Map/Reduce job that may require additional resources, it can allocate more to it temporarily so that your job finishes in the most efficient way possible.
Installing Mesos is not too difficult, but the docs are a little sparse. I set up a vanilla Ubuntu-12.04 LTS VM on my localhost to experiment. I will assume the same for you, so you may not need all the packages etc. but please do bear with me.
Disclaimers aside, let’s get cracking on the installation!
The first step will be to download the Apache Mesos tarball distribution. I used wget on my VM to grab the link from a local mirror, but you should check which mirror to use via ttp://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.13.0/
Next up, you will need to prepare your machine to compile and host Mesos. On Ubuntu, you need the following packages:
apt-get update && apt-get install python-dev libunwind7-dev libcppunit-dev openjdk-7-jdk autoconf autopoint libltdl-dev libtool autotools-dev make gawk g++ curl libcurl4-openssl-dev
Once that is complete, unpack your Mesos tarball with:
tar xzvf mesos-0.13.0.tar.gz
Change directory to the newly created mesos directory and run the configure script
cd mesos-0.13.0 ./configure
All the configure options should check out, but if not, make sure that you have all the relevant packages installed beforehand!
Next, in the same directory, compile the code
Depending on how many resources you gave your VM, this could take a while, so go get some milk and cookies…
Once the make is done, you should check everything with
This will run a bunch of unit tests and checks that will ensure that there are no surprises later.
After you have done this, you can also set up a small Mesos cluster and run a job on it as follows:
In your Mesos directory, use
to start the master server. Make a note of the IP and port that the master is running on, so that you can use the web based UI later on!
Open up a browser and point it to http://yourhostname.com:5050 or http://yourIP:5050. As an example, mine is running on http://192.168.10.56:5050
Go back to your VM’s terminal and type
and refresh your browser. You should now notice that a slave has been added to your cluster!
Run the C++ test framework (a sample that just runs five tasks on the cluster) using
It should successfully exit after running five tasks.
You can also try the example python or Java frameworks, with commands like the following:
src/examples/java/test-framework 192.168.10.56:5050 src/examples/python/test-framework 192.168.10.56:5050
If all of that is running OK, you have successfully completed the Mesos setup. Congratulations!
Follow @ApacheMesos on twitter as well as @DaveLester for more information and goodness!