Category Archives: DevOps

An introduction to Apache Mesos

What is Apache Mesos you ask? Well, from their web site at http://mesos.apache.org/ it is

Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, MPI, Hypertable, Spark, and other applications on a dynamically shared pool of nodes.

What exactly does that mean? Well, to me, it means that I can deploy applications and certain frameworks into a cluster of resources (VM’s) and it will look after my resource allocation for me. For example, if you have a particularly computationally intensive task, like a Hadoop Map/Reduce job that may require additional resources, it can allocate more to it temporarily so that your job finishes in the most efficient way possible.

Installing Mesos is not too difficult, but the docs are a little sparse. I set up a vanilla Ubuntu-12.04 LTS VM on my localhost to experiment. I will assume the same for you, so you may not need all the packages etc. but please do bear with me.

Disclaimers aside, let’s get cracking on the installation!

The first step will be to download the Apache Mesos tarball distribution. I used wget on my VM to grab the link from a local mirror, but you should check which mirror to use via ttp://www.apache.org/dyn/mirrors/mirrors.cgi/mesos/0.13.0/

Next up, you will need to prepare your machine to compile and host Mesos. On Ubuntu, you need the following packages:

apt-get update && apt-get install python-dev libunwind7-dev libcppunit-dev openjdk-7-jdk autoconf autopoint libltdl-dev libtool autotools-dev make gawk g++ curl libcurl4-openssl-dev

Once that is complete, unpack your Mesos tarball with:

tar xzvf mesos-0.13.0.tar.gz

Change directory to the newly created mesos directory and run the configure script

cd mesos-0.13.0
./configure

All the configure options should check out, but if not, make sure that you have all the relevant packages installed beforehand!
Next, in the same directory, compile the code

make

Depending on how many resources you gave your VM, this could take a while, so go get some milk and cookies…

Once the make is done, you should check everything with

make check

This will run a bunch of unit tests and checks that will ensure that there are no surprises later.

After you have done this, you can also set up a small Mesos cluster and run a job on it as follows:
In your Mesos directory, use

bin/mesos-master.sh

to start the master server. Make a note of the IP and port that the master is running on, so that you can use the web based UI later on!

Open up a browser and point it to http://yourhostname.com:5050 or http://yourIP:5050. As an example, mine is running on http://192.168.10.56:5050

Go back to your VM’s terminal and type

bin/mesos-slave.sh --master=192.168.10.56:5050

and refresh your browser. You should now notice that a slave has been added to your cluster!

Run the C++ test framework (a sample that just runs five tasks on the cluster) using

src/test-framework --master=localhost:5050 

It should successfully exit after running five tasks.
You can also try the example python or Java frameworks, with commands like the following:

    src/examples/java/test-framework 192.168.10.56:5050
    src/examples/python/test-framework 192.168.10.56:5050

If all of that is running OK, you have successfully completed the Mesos setup. Congratulations!

Follow @ApacheMesos on twitter as well as @DaveLester for more information and goodness!

How to install Puppet (puppetmaster) server on Debian Squeeze in 2 minutes or less

Reposted from old site – original date: Friday 2 March 2012

echo “deb http://debian.mirror.iweb.ca/debian-backports squeeze-backports main” >> /etc/apt/sources.list.d/backports.list
echo “deb-src http://debian.mirror.iweb.ca/debian-backports squeeze-backports main” >> /etc/apt/sources.list.d/backports.list
apt-get update
apt-get update
apt-get install -y -t squeeze-backports puppet puppet-common puppet-el puppet-testsuite puppetmaster puppetmaster-common vim-puppet vim-nox
kill -9 $(ps aux | grep puppet | grep -v grep | awk ‘{print $2}’)
sed -i “s/START=yes/START=no/g” /etc/default/puppetmaster
cat /etc/default/puppetmaster
apt-get install -y puppetmaster-passenger
mkdir -p /etc/puppet/files
echo “ServerName puppet” >> /etc/apache2/apache2.conf
service apache2 restart
puppet –version
netstat -a | grep 8140

Done.

python_logo_without_textsvg

Very dirty site monitor

Reposted from old site – original date: 2009-05-26 11:54:39

I needed a quick script to monitor a bunch of sites quickly and *very* simply. I looked at doing this all in a single thread with a ping, but then decided to go a non blocking route and make the thing a little more thread safe. This was done so that multiple sites that actually could go down could be monitored with relative accuracy, without a single site blocking the rest.

OK, enough flapping, here is the code (Python this time)

import threading
import os, popen2, select, signal
import Queue
import sys, xmpp
import time

HOST_UP = 1
HOST_DOWN = 0

PING_INTERVAL = 60

jidparams={'jid':'[email protected]/pinger', 'password':'somepass'}
jid=xmpp.protocol.JID(jidparams['jid'])
cl=xmpp.Client(jid.getDomain(), debug=[])
con=cl.connect() ##proxy={'host':'cache.company.co.za','port':'8080','user':'userdude','password':'le password'})
if not con:
    print 'could not connect!'
    sys.exit()

auth=cl.auth(jid.getNode(),jidparams['password'],resource=jid.getResource())

if not auth:
    print 'could not authenticate!'
    sys.exit()

def log(s):
    print s
    text = s
    tojid='[email protected]'
    #cl.SendInitPresence(requestRoster=0)
    id=cl.send(xmpp.protocol.Message(tojid,text))
    time.sleep(1)

class Pinger(threading.Thread):
    def __init__(self, queue, address, *args, **kwargs):
        self.address = address
        self.queue = queue
        threading.Thread.__init__(self, *args, **kwargs)
        self.stop = 0

    def run(self):
        child = popen2.Popen3("ping -i %i %s 21" % (PING_INTERVAL, self.address))
        while 1:
            ready_fds = select.select([child.fromchild], [], [])
            line = child.fromchild.readline()
            if line.find("Destination Host Unreachable") = 0:
                log("host %s is down" % self.address)
            else:
                pass

            if self.stop:
                os.kill(child.pid, signal.SIGTERM)
                break

def main():
    pinglist = ["www.company1.co.za", "it.company.co.za", "something.company.co.za", "172.16.65.208"]
    threads = []
    queue = Queue.Queue()

    for adr in pinglist:
        threads.append(Pinger(queue, adr))

    for thread in threads:
        thread.start()

    try:
        while 1:
            time.sleep(1)
    except KeyboardInterrupt:
        pass

    for thread in threads:
        thread.stop = 1

    for thread in threads:
        thread.join(2.0)

    # Ok, I've had enough of you stoopid threads.
    os._exit(0)

if __name__ == "__main__":
    main()