Tag Archives: code

Android Volley – HTTP async Swiss Army Knife

This serves as a post to help you get started with Android Volley. Volley is used for all sorts of HTTP requests, and supports a whole bang of cool features that will make your life way easier.

It is a relatively simple API to implement, and allows request queuing, which comes in very useful.

The code below is a simple example to make a request to this blog and get a JSON response back, parse it and display it in a simple TextView widget on the device.

package za.co.paulscott.volleytest;

import org.json.JSONObject;

import android.os.Bundle;
import android.support.v4.app.Fragment;
import android.support.v7.app.ActionBarActivity;
import android.util.Log;
import android.view.LayoutInflater;
import android.view.Menu;
import android.view.MenuItem;
import android.view.View;
import android.view.ViewGroup;
import android.widget.TextView;

import com.android.volley.Request;
import com.android.volley.RequestQueue;
import com.android.volley.Response;
import com.android.volley.VolleyError;
import com.android.volley.toolbox.JsonObjectRequest;
import com.android.volley.toolbox.Volley;

public class MainActivity extends ActionBarActivity {

    private TextView txtDisplay;
    protected void onCreate(Bundle savedInstanceState) {

        if (savedInstanceState == null) {
                    .add(R.id.container, new PlaceholderFragment())
        txtDisplay = (TextView) findViewById(R.id.txtDisplay);

		RequestQueue queue = Volley.newRequestQueue(this);
		String url = "http://paulscott.co.za/blog/wp-json.php/posts/100";

		JsonObjectRequest jsObjRequest = new JsonObjectRequest(Request.Method.GET, url, null, new Response.Listener<JSONObject>() {

			public void onResponse(JSONObject response) {
				String txt = response.toString();
				Log.i("volleytest", txt);
		}, new Response.ErrorListener() {

			public void onErrorResponse(VolleyError error) {
				// TODO Auto-generated method stub



    public boolean onCreateOptionsMenu(Menu menu) {
        // Inflate the menu; this adds items to the action bar if it is present.
        getMenuInflater().inflate(R.menu.main, menu);
        return true;

    public boolean onOptionsItemSelected(MenuItem item) {
        // Handle action bar item clicks here. The action bar will
        // automatically handle clicks on the Home/Up button, so long
        // as you specify a parent activity in AndroidManifest.xml.
        int id = item.getItemId();
        if (id == R.id.action_settings) {
            return true;
        return super.onOptionsItemSelected(item);

     * A placeholder fragment containing a simple view.
    public static class PlaceholderFragment extends Fragment {

        public PlaceholderFragment() {

        public View onCreateView(LayoutInflater inflater, ViewGroup container,
                Bundle savedInstanceState) {
            View rootView = inflater.inflate(R.layout.fragment_main, container, false);
            return rootView;


How to start an Android app at boot time

I wanted to have my custom Android ROM boot up and start an application specific to my needs start immediately. This is the way to accomplish it:

In your AndroidManifest.xml document (application part):

<receiver android:enabled="true" android:name=".BootUpReceiver"
                <action android:name="android.intent.action.BOOT_COMPLETED" />
                <category android:name="android.intent.category.DEFAULT" />

You also need to set up a permission with

<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" />

and then create the BootUpReceiver class to handle it

public class BootUpReceiver extends BroadcastReceiver{
        public void onReceive(Context context, Intent intent) {
                Intent i = new Intent(context, MyWhateverActivity.class);  

How to truncate multiple MongoDB collections in a Database

MongoDB has a drop() command that you can use to delete everything in a certain collection, but this also, unfortunately, will drop your indexes and other things too. What I wanted was a way that I could “truncate” the collection (borrowing from MySQL) and retain the indexes etc too.

The following snippet will do that, plus it has a built in “Oops, I changed my mind” safety check in case you need to cancel the collection truncate command.

var dbName = 'myDB';
db.getSiblingDB(dbName).getCollectionNames().forEach(function(collName) {
    // Drop all collections except system ones (indexes/profile)
    if (!collName.startsWith("system.")) {
        // Safety net
        print("WARNING: going to drop ["+dbName+"."+collName+"] in 5s .. hit Ctrl-C if you've changed your mind!");

This would be best saved as a UDF in your mongo shell, and probably made to take a parameter for the db too…

Java HTTPClient

Just a quick post with an example of using the Java HTTPClient to make requests to remote web servers. I did not find much in the way of succint examples, so here is one:

import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
import org.apache.commons.httpclient.params.HttpMethodParams;

import java.io.*;

public class MyClient {

	public MyClient() {

	private String url;
	public String getUrl() {
		return url;

	public void setUrl(String url) {
		this.url = url;

	public byte[] grok() {
		// Create an instance of HttpClient.
		HttpClient client = new HttpClient();

		// Create a method instance.
		GetMethod method = new GetMethod(url);

		// Provide custom retry handler is necessary
				new DefaultHttpMethodRetryHandler(3, false));

		try {
			// Execute the method.
			int statusCode = client.executeMethod(method);

			if (statusCode != HttpStatus.SC_OK) {
				System.err.println("Method failed: " + method.getStatusLine());

			// Read the response body.
			byte[] responseBody = method.getResponseBody();

			// Deal with the response.
			// Use caution: ensure correct character encoding and is not binary
			// data
			return responseBody;

		} catch (HttpException e) {
			System.err.println("Fatal protocol violation: " + e.getMessage());
		} catch (IOException e) {
			System.err.println("Fatal transport error: " + e.getMessage());
		} finally {
			// Release the connection.
		return null;


Hadoop HDFS “abstraction” class in Java

Recently I found myself working with Hadoop’s HDFS, which is the Hadoop file system in a Java project. The docs for working with Hadoop (especially HDFS) are somewhat sparse, as I guess most folks prefer to keep their code pretty secretive. Well, seeing as though the following will never give anyone a business advantage over anyone else, but will probably spare a few folks some sleepless nights, here you go!

First off, you need to start a new Maven project. I just used the simple archetype as I was just messing about really. You then will need to add the following dependencies to your POM.xml file (Oh, using Maven 3…)



You will notice a couple of excludes there. They are really important as some of that code is dead and discontinued.

Next up, we need a class to interact with our hdfs store.

public class HbaseExample {
	private static Configuration hBaseConfig = null;
	private static String hbaseHost = "ubuntu.local";
	private static String zookeeperHost = "ubuntu.local";

	 * Initialization
	static {
		hBaseConfig = HBaseConfiguration.create();
		hBaseConfig.setInt("timeout", 120000);
		hBaseConfig.set("hbase.master", "*" + hbaseHost + ":9000*");
		hBaseConfig.set("hbase.zookeeper.quorum", zookeeperHost);
		hBaseConfig.set("hbase.zookeeper.property.clientPort", "2181");
        new HTablePool(hBaseConfig, 10);

You will see that all is pretty standard and that I have defined a few static properties to hold the values of the zookeeper and Hadoop hosts. Note that these will work off IP addresses, but Hadoop most definitely prefers FQDN’s. The final line in the method is a sticky one. Basically is you do not create a connection pool, your code can take up to 5 seconds to re-initialize the database, which is obviously not cool.

After that, there is nothing too tricksy. I will paste a copy of the whole class next so that you can  check your imports etc as well as have a look at full CRUD on an HDFS store:

package za.co.paulscott.hdfstest;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.HTablePool;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.exceptions.MasterNotRunningException;
import org.apache.hadoop.hbase.exceptions.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.util.Bytes;

public class HbaseExample {
	private static Configuration hBaseConfig = null;
	private static String hbaseHost = "ubuntu.local";
	private static String zookeeperHost = "ubuntu.local";

	 * Initialization
	static {
		hBaseConfig = HBaseConfiguration.create();
		hBaseConfig.setInt("timeout", 120000);
		hBaseConfig.set("hbase.master", "*" + hbaseHost + ":9000*");
		hBaseConfig.set("hbase.zookeeper.quorum", zookeeperHost);
		hBaseConfig.set("hbase.zookeeper.property.clientPort", "2181");
        new HTablePool(hBaseConfig, 10);

	 * Create a table
	public static void creatTable(String tableName, String[] familys)
			throws Exception {
		HBaseAdmin admin = new HBaseAdmin(hBaseConfig);
		boolean exists = false;
		try {
			exists = admin.tableExists(tableName);
		} catch (NullPointerException e) {
			exists = false;
		if (exists = true) {
			System.out.println("table already exists!");
		} else {
			HTableDescriptor tableDesc = new HTableDescriptor(tableName);
			for (int i = 0; i < familys.length; i++) {
				tableDesc.addFamily(new HColumnDescriptor(familys[i]));
			System.out.println("create table " + tableName + " ok.");

	 * Delete a table
	public static void deleteTable(String tableName) throws Exception {
		try {
			HBaseAdmin admin = new HBaseAdmin(hBaseConfig);
			System.out.println("delete table " + tableName + " ok.");
		} catch (MasterNotRunningException e) {
		} catch (ZooKeeperConnectionException e) {

	 * Put (or insert) a row
	public static void addRecord(String tableName, String rowKey,
			String family, String qualifier, String value) throws Exception {
		//System.out.print("Adding record to table:  " + tableName);
		try {
			HTable table = new HTable(hBaseConfig, tableName);
			Put put = new Put(Bytes.toBytes(rowKey));
			put.add(Bytes.toBytes(family), Bytes.toBytes(qualifier),
			System.out.println("insert recored " + rowKey + " to table "
					+ tableName + " ok.");
		} catch (IOException e) {

	 * Delete a row
	public static void delRecord(String tableName, String rowKey)
			throws IOException {
		HTable table = new HTable(hBaseConfig, tableName);
		List list = new ArrayList();
		Delete del = new Delete(rowKey.getBytes());
		System.out.println("del recored " + rowKey + " ok.");

	 * Get a row
	public static void getOneRecord(String tableName, String rowKey)
			throws IOException {
		HTable table = new HTable(hBaseConfig, tableName);
		Get get = new Get(rowKey.getBytes());
		Result rs = table.get(get);
		for (KeyValue kv : rs.raw()) {
			System.out.print(new String(kv.getRow()) + " ");
			System.out.print(new String(kv.getFamily()) + ":");
			System.out.print(new String(kv.getQualifier()) + " ");
			System.out.print(kv.getTimestamp() + " ");
			System.out.println(new String(kv.getValue()));

	 * Scan (or list) a table
	public static void getAllRecord(String tableName) {
		try {
			HTable table = new HTable(hBaseConfig, tableName);
			Scan s = new Scan();
			ResultScanner ss = table.getScanner(s);
			for (Result r : ss) {
				for (KeyValue kv : r.raw()) {
					System.out.print(new String(kv.getRow()) + " ");
					System.out.print(new String(kv.getFamily()) + ":");
					System.out.print(new String(kv.getQualifier()) + " ");
					System.out.print(kv.getTimestamp() + " ");
					System.out.println(new String(kv.getValue()));
		} catch (IOException e) {

	public static void main(String[] agrs) {
		try {
			String tablename = "scores";
			String[] familys = { "grade", "course" };
			HbaseExample.creatTable(tablename, familys);

			// add record paul
			HbaseExample.addRecord(tablename, "paul", "grade", "", "5");
			HbaseExample.addRecord(tablename, "paul", "course", "", "90");
			HbaseExample.addRecord(tablename, "paul", "course", "math", "97");
			HbaseExample.addRecord(tablename, "paul", "course", "art", "87");
			// add record caz
			HbaseExample.addRecord(tablename, "caz", "grade", "", "4");
			HbaseExample.addRecord(tablename, "caz", "course", "math", "89");

			System.out.println("===========get one record========");
			HbaseExample.getOneRecord(tablename, "paul");

			System.out.println("===========show all record========");

			System.out.println("===========del one record========");
			HbaseExample.delRecord(tablename, "caz");

			System.out.println("===========show all records========");
		} catch (Exception e) {


As you can see, there is also a Main method, so to execute this in Eclipse, you can simply run as… Java application, or compile is as a jar (using goal jar:jar) and run it.

NOTE: Make sure that you change the host(s)!

NOTE2: This is an example to connect to a remote hadoop cluster!

NOTE3: This is one too many notes, so go now and have fun!

Laptop tracking system (cheap)

Reposted from old site – original date: Wednesday 7 September 2011

Following on from a discussion on a work forum, I decided to have a stab at writing a super simple laptop tracking system. The main aim of the system will be to track stolen laptops and (hopefully) recover them somehow.

After a little bit of consideration, I decided to have a look at doing this in a client/server way with a MongoDB back end. The following is a simple, but workable system that uses the python bindings to the DBUS messaging system on Linux (tested on Ubuntu 11.04), so I am doubtful that this could be used for anything other than that. That being said, however, the client bit simply needs to send through wifi access points and MAC addresses to get a triangulation on the position of the laptop, so I am pretty sure that this can be achieved with relative ease on other platforms.

Triangulation is done via a Google API and the coordinates as well as the accuracy level is then inserted to the MongoDB back end for mapping or whatever else needs to be done with the data. Features that the client should also probably support include bricking the machine remotely or something to that effect, as well as possibly sending messages or SOS tweets or something to let people know that it is not in its rightful owners posession.

Enough rambling! To the code!


As I said, the code uses python-dbus and the standard json and urllib modules. You can install python-dbus with an apt-get install python-dbus.

import dbus
import json
import urllib

NM = 'org.freedesktop.NetworkManager'
NMP = '/org/freedesktop/NetworkManager'
NMI = NM + '.Device'
PI = 'org.freedesktop.DBus.Properties'

def list_ssids(): 
    bus = dbus.SystemBus()
    nm = bus.get_object(NM,NMP)
    nmi = dbus.Interface(nm,NM)
    # Iterate over the devices queried via the interface
    for dev in nmi.GetDevices():
        # get each and bind a property interface to it
        devo = bus.get_object(NM,dev)
        devpi = dbus.Interface(devo,PI)
        if devpi.Get(NM+'.Device','DeviceType') == 2:
            wdevi = dbus.Interface(devo,NMI + '.Wireless')
            wifi = []
            for ap in wdevi.GetAccessPoints():
                apo = bus.get_object(NM,ap)
                api = dbus.Interface(apo,PI)
                wifi.append({'ssid':''.join(["%c" % b for b in api.Get("org.freedesktop.NetworkManager.AccessPoint", "Ssid", byte_arrays=True)]), 'mac':''.join(["%c" % b for b in api.Get("org.freedesktop.NetworkManager.AccessPoint", "HwAddress", byte_arrays=True)])})
    return wifi

if __name__ == '__main__':
  ap = list_ssids()
  data = json.dumps(ap)
  params = urllib.urlencode({'wifi': data})
  f = urllib.urlopen("", params)

You will need to modify the post URL at the end to somewhere meaningful for yourself of course.

Initially, I did the client using the PHP ext/dbus, but decided against that due to the fact that it is hard to install and nobody really uses it…

The server side is just as simple. You will need a Mongodb instance running on the server and then you need a simple script to catch the POSTs. NOTE: This script is just a concept, so if you are actually going to do something like this, clean it up!

$wifi = $_POST['wifi'];
$wifi = json_decode($wifi);
$request = array( 'version' => '1.1.0', 'host' => 'myurl.com', 'wifi_towers' => $wifi );
$c = curl_init();
curl_setopt( $c, CURLOPT_URL, 'https://www.google.com/loc/json' );
curl_setopt( $c, CURLOPT_POST, 1 );
curl_setopt( $c, CURLOPT_POSTFIELDS, json_encode( $request ) );
curl_setopt( $c, CURLOPT_RETURNTRANSFER, true );
$result = json_decode( curl_exec( $c ) )->location;

$fields = array(
            'laptopid' => urlencode('16cptl-pscott'),

// connect
$m = new Mongo();
// select a database
$db = $m->laptoptrack;
$collection = $db->lappoints;

// add a record
$obj = array( "loc" => array("lon" => floatval($fields['lon']), "lat" => floatval($fields['lat'])), "lon" => floatval($fields['lon']), "lat" => floatval($fields['lat']), "accuracy" => intval($fields['accuracy']), "laptopid" => $fields['laptopid'], "wifi" => json_encode($wifi));
$collection->ensureIndex(array('loc' => "2d"));
$collection->insert($obj, array('safe'=>true)); 

So from the above code, you will see that we create a 2d geospatial index on the Mongodb instance as well. Not sure if this is useful, but it will probably help in speeding up queries like “Gimme all the laptops that Company X owns in area Y” or something – if that is something that you would like to add to your query interface of course…

Also, I am not 100% sure of the legality of storing SSID’s with a location, so check that one first too!

Dead simple, works well.

I would say that the client bit should be on a cron job or something that pings the service every hour or something.

Remember: Mongodb works best on 64bit OS. If you are using a 32bit arch, then you will only be able to store around 2GB data at a time. Depending on the need for historical records etc, keep that in mind…

Most importantly, HAVE FUN!

Why Tweetdeck?

Reposted from old site – original date: Thursday 26 November 2009

I was thinking a bit about stuff last night, and especially the fact that I am under pressure from a lot of folks to move away from Tweetdeck (which I think is awesome) to a web based client like HootSuite,
Seesmic web or any of the other AJAXY things out there (there are now loads to choose from). The bottom line is, yes, they probably do work well, and yes, that is great, but
there are very distinct disadvantages to using a browser based client:

You need an open browser window, “maximised” to be useful.

IMO browser windows take up a LOT of desktop real estate, so not great when you have 6 shells open too, and, and, and…

I realise that Tweetdeck does take up quite a lot of resources, although it seems MUCH better in its latest incarnation, but I really like it.

Code wise, would you rather write a bunch of JavaScript and PHP or something to do a simple AJAX based Twitter client, or would you rather write a
hundred or so lines of python to do the same thing on your desktop? Remembering that in a web environment you have to hack thread-like behaviour, while on desktop
threads are a no-brainer to do background, non-blocking updates etc.

Basically, what I am saying is that RIA (Rich Internet Applications) rock. They bring the best of both worlds to the user in a controlled and resource happy environment. In a web sense, if all my users
were using desktop clients, I would be over the moon, as the only thing I would have to serve is JSON or something similarly lightweight, saving my bandwidth bill and my hardware bill, as I could support
way more folks than a heavier interface could!

The trick though is to save yourself time too. Write simple, cross platform desktop apps that are easily maintainable and package-able. I chose wxPython to demo with below. You don’t need whole whacks of code
you just need a minimal interface to hold together some glue code making use of existing packages. The code below is not pretty, but it is terse and to the point.

If you would like to improve the code, please feel free, but please try keep it less than 100 lines…

So can you write a simple, but useful Twitter client in less than 100 lines? Here is one in 98, including shebang…

#!/usr/bin/env python
import wx
import twitter
import time
import sys 

class RedirectText(object):
    def __init__(self,aWxTextCtrl):
    def write(self,string):

class MyFrame(wx.Frame):

    def __init__(self, *args, **kwds):
        wx.Frame.__init__(self, *args, **kwds)
        self.textUpdate = wx.TextCtrl(self, -1, "", style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE|wx.TE_AUTO_URL|wx.TE_LINEWRAP|wx.TE_WORDWRAP)
        self.updateButton = wx.Button(self, -1, "Update", style=wx.BU_EXACTFIT)
        self.panel = wx.Panel(self, wx.ID_ANY)
        self.log = wx.TextCtrl(self.panel, wx.ID_ANY, "",  style=wx.TE_MULTILINE|wx.TE_WORDWRAP)


        self.Bind(wx.EVT_TEXT_ENTER, self.onUpdate, self.textUpdate)
        self.Bind(wx.EVT_BUTTON, self.onUpdate, self.updateButton)

    def __set_properties(self):
        self.username = 'paulscott56'
        self.password = 'password123'
        self.encoding = None
        self.SetSize(wx.DLG_SZE(self, (205, 234)))

    def __do_layout(self):
        sizer_1 = wx.BoxSizer(wx.VERTICAL)
        sizer_2 = wx.BoxSizer(wx.VERTICAL)
        sizer_3 = wx.BoxSizer(wx.HORIZONTAL)
        sizer_2.Add(self.textUpdate, 1, wx.ALL|wx.EXPAND, 4)
        sizer_2.Add(self.updateButton, 0, wx.ALL|wx.EXPAND, 4)
        sizer_3.Add(self.log, 4, wx.EXPAND|wx.ALIGN_CENTER_HORIZONTAL|wx.ALIGN_CENTER_VERTICAL, 4)
        sizer_2.Add(self.panel, 1, wx.EXPAND, 0)
        sizer_1.Add(sizer_2, 1, wx.EXPAND, 0)

    def onUpdate(self, event): 
        self.api = twitter.Api(username=self.username, password=self.password, input_encoding=self.encoding)
        message = self.textUpdate.GetValue()
            status = self.api.PostUpdate(message)
        except UnicodeDecodeError:
            print "Your message could not be encoded.  Perhaps it contains non-ASCII characters? "
        print "%s just posted: %s" % (status.user.name, status.text)

    def fetchTwitter(self,  event):
        self.api = twitter.Api(username=self.username, password=self.password, input_encoding=self.encoding)
        statuses = self.api.GetFriendsTimeline(user=self.username)
        for s in statuses:
            # make the text thing
            sn = s.user.screen_name
            txt = s.text
            time = s.relative_created_at
            stringthing = sn+": "+txt+" "+time+'rn'
            # redirect text
            print stringthing
        # make a timer here
        timeLapse = 60000
        self.timer = wx.Timer(self)
        self.Bind(wx.EVT_TIMER, self.fetchTwitter, self.timer) 

# end of class MyFrame

if __name__ == "__main__":
    app = wx.PySimpleApp(0)
    Twitrrr = MyFrame(None, -1, "")


Chisimba event based messaging framework

As of January 2009, I have now committed the code to the core classes of Chisimba that
enables an event based messaging framework. This is useful in many ways,
so use it with innovation in mind… :)

The framework creates an observer pattern that can be used pretty
globally. I have added an example to the phpinfo module for a test. It
is really an arbitrary example, but I think it should give you all a
rough idea on how it works. The following example is the same code in
case you miss it, but annotated with many more comments:

The first thing that you need to do is in your init() function (or
wherever else I suppose) you need to create the observer. I have left
this pretty open so that it is super flexible:

$this->eventDispatcher->addObserver ( array ($this, 'modaccess' ) );

is a property from the core, so that will always
be the case. You are calling the addObserver() function, which sets up a
callback function that you specify in an array. This particular example
is calling a callback in the same class, but you can specify any module
you like (such as a class that logs to the db or something for site wide
notes about current activity?). The function that I am calling is called
modaccess, which is defined as:

public function modaccess($notification) {
    if ($notification->getNotificationName () == 'test') {
       log_debug("PHPInfo module was accessed");

The events system creates a referenced $notification object that is
passed through Chisimba to your module and then callback, so that will
always be the argument. There are a couple of properties in
$notification that you can use such as:

getNotificationName() - gets the name of the notifier
getNotificationObject() - get the name of the object that sent the note
getNotificationInfo() - any info that you have added
getNotificationCount() - a count
isNotificationCancelled() - well, is it?

There are some others, but I will doc them properly soon and let you

The function then simply logs the notification to the error_log, but
this could fire off almost anything (db insert, xmpp message, forum
post, etc etc)

The only thing left is to actually post a notification to our messaging
system now, which is done like so:

this->eventDispatcher-post($this, "test");

This then sets the notification name and posts away. This could be done
in any function or in any action as well. as an example…

Any questions, please don’t hesitate to ask!