Category Archives: DevOps

An introduction to Apache Mesos

What is Apache Mesos you ask? Well, from their web site at it is

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other applications on a dynamically shared pool of nodes.

What exactly does that mean? Well, to me, it means that I can deploy applications and certain frameworks into a cluster of resources (VM’s) and it will look after my resource allocation for me. For example, if you have a particularly computationally intensive task, like a Hadoop Map/Reduce job that may require additional resources, it can allocate more to it temporarily so that your job finishes in the most efficient way possible.

Installing Mesos is not too difficult, but the docs are a little sparse. I set up a vanilla Ubuntu-12.04 LTS VM on my localhost to experiment. I will assume the same for you, so you may not need all the packages etc. but please do bear with me.

Disclaimers aside, let’s get cracking on the installation!

The first step will be to download the Apache Mesos tarball distribution. I used wget on my VM to grab the link from a local mirror, but you should check which mirror to use via ttp://

Next up, you will need to prepare your machine to compile and host Mesos. On Ubuntu, you need the following packages:

apt-get update && apt-get install python-dev libunwind7-dev libcppunit-dev openjdk-7-jdk autoconf autopoint libltdl-dev libtool autotools-dev make gawk g++ curl libcurl4-openssl-dev

Once that is complete, unpack your Mesos tarball with:

tar xzvf mesos-0.13.0.tar.gz

Change directory to the newly created mesos directory and run the configure script

cd mesos-0.13.0

All the configure options should check out, but if not, make sure that you have all the relevant packages installed beforehand!
Next, in the same directory, compile the code


Depending on how many resources you gave your VM, this could take a while, so go get some milk and cookies…

Once the make is done, you should check everything with

make check

This will run a bunch of unit tests and checks that will ensure that there are no surprises later.

After you have done this, you can also set up a small Mesos cluster and run a job on it as follows:
In your Mesos directory, use


to start the master server. Make a note of the IP and port that the master is running on, so that you can use the web based UI later on!

Open up a browser and point it to or http://yourIP:5050. As an example, mine is running on

Go back to your VM’s terminal and type

bin/ --master=

and refresh your browser. You should now notice that a slave has been added to your cluster!

Run the C++ test framework (a sample that just runs five tasks on the cluster) using

src/test-framework --master=localhost:5050 

It should successfully exit after running five tasks.
You can also try the example python or Java frameworks, with commands like the following:


If all of that is running OK, you have successfully completed the Mesos setup. Congratulations!

Follow @ApacheMesos on twitter as well as @DaveLester for more information and goodness!

How to install Puppet (puppetmaster) server on Debian Squeeze in 2 minutes or less

Reposted from old site – original date: Friday 2 March 2012

echo “deb squeeze-backports main” >> /etc/apt/sources.list.d/backports.list
echo “deb-src squeeze-backports main” >> /etc/apt/sources.list.d/backports.list
apt-get update
apt-get update
apt-get install -y -t squeeze-backports puppet puppet-common puppet-el puppet-testsuite puppetmaster puppetmaster-common vim-puppet vim-nox
kill -9 $(ps aux | grep puppet | grep -v grep | awk ‘{print $2}’)
sed -i “s/START=yes/START=no/g” /etc/default/puppetmaster
cat /etc/default/puppetmaster
apt-get install -y puppetmaster-passenger
mkdir -p /etc/puppet/files
echo “ServerName puppet” >> /etc/apache2/apache2.conf
service apache2 restart
puppet –version
netstat -a | grep 8140



Very dirty site monitor

Reposted from old site – original date: 2009-05-26 11:54:39

I needed a quick script to monitor a bunch of sites quickly and *very* simply. I looked at doing this all in a single thread with a ping, but then decided to go a non blocking route and make the thing a little more thread safe. This was done so that multiple sites that actually could go down could be monitored with relative accuracy, without a single site blocking the rest.

OK, enough flapping, here is the code (Python this time)

import threading
import os, popen2, select, signal
import Queue
import sys, xmpp
import time



jidparams={'jid':'[email protected]/pinger', 'password':'somepass'}
cl=xmpp.Client(jid.getDomain(), debug=[])
con=cl.connect() ##proxy={'host':'','port':'8080','user':'userdude','password':'le password'})
if not con:
    print 'could not connect!'


if not auth:
    print 'could not authenticate!'

def log(s):
    print s
    text = s
    tojid='[email protected]'

class Pinger(threading.Thread):
    def __init__(self, queue, address, *args, **kwargs):
        self.address = address
        self.queue = queue
        threading.Thread.__init__(self, *args, **kwargs)
        self.stop = 0

    def run(self):
        child = popen2.Popen3("ping -i %i %s 21" % (PING_INTERVAL, self.address))
        while 1:
            ready_fds =[child.fromchild], [], [])
            line = child.fromchild.readline()
            if line.find("Destination Host Unreachable") = 0:
                log("host %s is down" % self.address)

            if self.stop:
                os.kill(, signal.SIGTERM)

def main():
    pinglist = ["", "", "", ""]
    threads = []
    queue = Queue.Queue()

    for adr in pinglist:
        threads.append(Pinger(queue, adr))

    for thread in threads:

        while 1:
    except KeyboardInterrupt:

    for thread in threads:
        thread.stop = 1

    for thread in threads:

    # Ok, I've had enough of you stoopid threads.

if __name__ == "__main__":