How to write MCollective Agents to run actions in background

The Marionette Collective, or MCollective in short, is a cutting-edge tech for running system administration tasks and jobs in parallel against a cluster of servers. When the number of servers you have to manage grows, the task of managing them, including keeping the OS and packages installed on them updated, becomes without a doubt a nightmare. MCollective helps you drag yourself out of that nightmare and into a jolly dream, where you are the king, and at your disposal is a powerful tool, by merely wielding which you can control all of your servers in one go. I’ll probably do a bad job of painting MCollective in good light, so I’d recommend you read all about it on MCollective’s main website.

Like every good tool worth its while, MCollective gives you the power to extend it by writing what it calls agents to run custom code on your servers to perform any kind of job. You can read about agents here. MCollective is all in Ruby, so if you know Ruby, which is a pretty little programming language by the way, you can take full advantage of MCollective. Incidentally, a part of my day job revolves around writing MCollective agents to automate all sorts of jobs you can think of performing on a server.

For a while I have been perplexed at the lack of support for being able to run agents in the background. Not every job takes milliseconds to finish itself. Most average-level jobs, in terms of what they have to do, take anywhere from seconds to, even longer, minutes. And since I write an API which uses MCollective agents to execute jobs, I often run into the problem of having the API block while the agent is taking its sweet time to run. As far as I’ve looked, I haven’t found support within MCollective for asynchronously running actions.

So, I got down to thinking, and came up with a solution, which you could call a bit of a hack. But insofar as my experience testing it has been, I’ve yet to face any issues with it.

I’ve an example hosted on GitHub. It’s very straightforward, even if crude. The code is self explanatory. At the heart of it is creating your actions in agents to fork and run as childs, without having the parent wait for the childs to reap, and having one extra action for each agent to fetch the status of the other actions. So, essentially, the approach has to be taken for every agent you have, but only for those actions which you wish to run asynchronously. With agents in place, all you’ll need to do is call the agents thus:

$ mco rpc bg run_bg -I node -j
{"status": 0, "result": "2434"}

and repeatedly fetch the status of the async action thus:

$ mco rpc bg status pid=2434 operation=run_bug -I node -j
{"status": 0, "result": ""}

It’s a solution that works. I’ve tested it over a month and a half now over many servers without any issues. I would like to see people play with it. The code is full of comments which explain how to do what. But if you have any questions, I’d love to entertain them

Mobile Safari Web Inspector on Safari on Mac.

If you tweak HTML and CSS and JavaScript on webpages, then you may appreciate how simple and convenient the plugin that goes by the name of Firebug for Firefox has made your life. I know it has for me. It is simply an indispensable piece of technology that I use every time I have to debug a CSS or HTML or JS issue on a webpage. Other browsers, notably Chrome and Safari, have built-in web inspectors which allow similar features.

I have of late been working on a lot of Mobile versions of websites, particularly those optimized for Mobile Safari and iOS devices using the fascinating jQuery Mobile framework. One of my biggest pet peeves while debugging issues on Mobile websites was not being able to debug the sites live on a mobile browser. More times than I’d care to acknowledge a particular mobile site, when viewed on a normal browser would look fine, but would have slight aberrations when viewed on a mobile browser. Those kind of issues are a real pain in the neck, because you are, more often than not, left to get sucked in an annoying cycle of wildly making changes in the HTML/CSS and refreshing the page on the mobile browser continuously in the hopes that the problem would fix itself.

However, today I bumped into a way to inspect and debug a mobile site on Mobile Safari by using Safari on a Mac computer (I am sure that it may work equally well for Safari on Windows, but I have not had reason nor occasion to try to find out). It not only works so well, but is also very straightforward to set up.

My testing environment includes an iPhone 4S running iOS 6 and a MacBook Pro running OS X Lion 10.7.5. Here are the steps you must follow in order to be able to inspect mobile web pages live via Safari on your Mac:

  • On your iOS device, turn on “Web Inspector” inside the Advanced settings for Safari. You’ll find these under the Settings app.
  • On the Advanced tab under Preferences for Safari on your Mac, you must check the “Show Develop menu in menu bar” setting.
  • Connect your iOS device to your Mac via the data cable.
  • You may now load on Mobile Safari the mobile webpage you want to inspect.
  • On Safari on Mac, open the Develop menu in the menu bar, select the name of your iOS device that will appear in the list, and click on the webpage shown on the pop-up that slides to the right.

The web inspector in Safari will kick in. You may explore the different tabs and functionality available there. The only downside is that no preview of the page is available on the inspector. Instead, the live page open on your Mobile Safari is the preview. You may go through the HTML source displayed inside the inspector, selecting the elements you want to inspect and notice that the corresponding elements are highlighted on the device. What’s more, you can also live edit the source as well as CSS, and they take effect almost immediately. How awesome is that!

Guide: Deploying on IIS7 using PyISAPIe

I spent the last week travailing away, trying to painfully find a way to deploy a based API on IIS7 using PyISAPIe. As frustrations had begun to mount up, I had nearly decided to give up. Being a die-hard Linux and Mac guy, I despise having to work on Windows. Here I was, not only forced to work on Windows, but to find a solution for a problem that left no leaves unturned in its effort to drive me crazy. As if someone decided all this misery wasn’t quite enough, I had to work with a remote desktop session in order to research, tweak, bang my head, and get things to work. Eventually, I cut through massive frustration and despair, managing to find a satisfactory solution. I almost danced in excitment and relief, letting out all sorts of expletives directed at Windows in general and IIS in particular.

To get back to the important question of deploying a script on IIS7 using PyISAPIe, I will make it such that this guide will list down various steps I took, including snippets of relevant code I changed, to tame the beast. I can only hope that what is below will help a poor, miserable soul looking for help as I did (and found none).

I worked with PyISAPIe because I had successfully deployed multiple Django websites on IIS7 on it. The script in question was going to be a part of another Django website (though acting independently). It only made sense to use PyISAPIe for it as well.

First and foremost, I had to install the module on the system. Having had trouble before with IIS with installed through easy_install, I decided to be safe and installed it from source.. Getting to work with PyISAPIe required a small hack (I notice I may make it sound as though it all came down to me in a dream, but in reality, it took me days to figure it out, and clearly after much anguish and pain). In the file Lib\site-packages\web\ lies the following function:

def _is_dev_mode():
    # quick hack to check if the program is running in dev mode.
    if os.environ.has_key('SERVER_SOFTWARE') \
        or os.environ.has_key('PHP_FCGI_CHILDREN') \
        or 'fcgi' in sys.argv or 'fastcgi' in sys.argv \
        or 'mod_wsgi' in sys.argv:
            return False
    return True

In its pristine state, when is imported from a source file through PyISAPIe, an exception is thrown. The exception, while I don’t have the exact message, is about it complaining about sys.argv not having an attribute argv, which reads fishy. Since the function _is_dev_mode() only checks whether is being run in development mode, I thought I didn’t care about it since I wanted everything to run in production mode. I edited the function such that its body would be bypassed, while it returned a False boolean value. It looked like this (the important changes I made are highlighted):

def _is_dev_mode():
    return False
    # quick hack to check if the program is running in dev mode.
    if os.environ.has_key('SERVER_SOFTWARE') \
        or os.environ.has_key('PHP_FCGI_CHILDREN') \
        or 'fcgi' in sys.argv or 'fastcgi' in sys.argv \
        or 'mod_wsgi' in sys.argv:
            return False
    return True

This innocuous little addition did away with the exception.

Next up, I used default Hello World-esque example of found on their site to test the deployment (of course, I went on to use my original API script, which was far too complex to trim down and fit into as an example). I called it (I placed it inside the folder C:\websites\myproject). It looked like this:

  import web
  urls = (
      '/.*', 'hello',
  class hello:
      def GET(self):
          return "Hello, world."
  application = web.application(urls, globals()).wsgifunc()

It was pretty simple. You have to pay particular attention on the call to web.application. I called the wsgifunc() to return a WSGI-compatible function to boot the application. I prefer WSGI.

I set up a website under IIS using the IIS Management Console. Since I was working on a 64-bit server edition of Windows and had chosen to use 32-bit version of Python and all modules, I made sure to enable 32-bit support for the application pool being used for the website. This was important.

I decided to keep the PyISAPIe folder inside the folder where rested. This PyISAPIe folder contained, of import, the PyISAPIe.dll file, and the Http folder. Inside the Http folder, I placed the most important file of all: the That file could be thought of as the starting point for each request that is made, what glues the Request to the proper Handler and code. I worked with the Examples\WSGI\ available as part of PyISAPIe. I tweaked the file to look like this:

from Http.WSGI import RunWSGI
from Http import Env
#from md5 import md5
from hashlib import md5
import imp
import os
import sys
from code import application
ScriptHandlers = {
	"/api/": application,
def RunScript(Path):
  global ScriptHandlers
    # attempt to call an already-loaded request function.
    return ScriptHandlers[Path]()
  except KeyError:
    # uses the script path's md5 hash to ensure a unique
    # name - not the best way to do it, but it keeps
    # undesired characters out of the name that will
    # mess up the loading.
    Name = '__'+md5(Path).hexdigest().upper()
    ScriptHandlers[Path] = \
      imp.load_source(Name, Env.SCRIPT_TRANSLATED).Request
    return ScriptHandlers[Path]()
# URL prefixes to map to the roots of each application.
Apps = {
  "/api/" : lambda P: RunWSGI(application),
# The main request handler.
def Request():
  # Might be better to do some caching here?
  Name = Env.SCRIPT_NAME
  # Apps might be better off as a tuple-of-tuples,
  # but for the sake of representation I leave it
  # as a dict.
  for App, Handler in Apps.items():
    if Name.startswith(App):
      return Handler(Name)
  # Cause 500 error: there should be a 404 handler, eh?
  raise Exception, "Handler not found."

The important bits to note in the above code are the following:

  • I import application from my code module. I set the PATH to include the directory in which the file is so that the import statement does not complain. (I’ve to admit that the idea of import application and feeding it into RunWSGI came to while I was in the loo.)
  • I defined a script handler which matches the URL prefix I want to associate with my script. (In hindsight, this isn’t necessary, as the RunScript() is not being used in this example).
  • In the Apps dictionary, I again route the URL prefix to the lambda function which actually calls the `RunWSGI` function and feeds it application.
  • I also imported the md5 function from the hashlib module instead of the md5 module as originally defined in the file. This was because Python complained about md5 module being deprecated and suggested instead of use hashlib.

And that’s pretty much it. It worked. I couldn’t believe what I saw on the browser in front of me. I danced around my room (while hurling all kinds of expletives).

There’s a caveat though. If you have specific URLs in your script, as I did in my API script, you will have to modify each of those URLs are add the /api/ prefix to them (or whatever URL prefix you set in the Without that, will not match any URLs in the file.

What a nightmare! I hope this guide serves to help others.

Thank you for reading. Good bye!

PS: If you want to avoid using PyISAPIe, there is a simpler way of deploying on IIS. It is documented crudely over here.

Get all public interface IPs on a system using Python

     I recently came across a requirement in a project where I had to, in Python, programmatically extract all available public IPs on available interfaces on the machine the code would run. I looked around and settled with the following snippet of code that uses the built-in, standard socket Python module:

import socket
ip_list = [ip for ip in socket.gethostbyname_ex(socket.gethostname())[2] if not ip.startswith("127.")]

     While this piece of code does find a public IP listening on any of the available interfaces, its restriction lies in not being able to return all public IPs on interfaces: It gives back just one IP.

     This wasn’t clearly sufficient. I looked around again, and this time, found a third-party Python module called pynetinfo. This module could make possible working with different network device settings.

     I rearranged the code around pynetinfo and produced this:

def get_inet_ips():
    import netinfo
  except ImportError:
    return None
    inetIPs = []
    for interface in netinfo.list_active_devs():
      if not interface.startswith('lo'):
        ip = netinfo.get_ip(interface)
    return inetIPs

     The code above loops through all available and active interfaces on the system, fetching and storing their IP in a simple datastructure. That got me all of the IPs available to a machine, excluding the loopback one, which the code was set to discard.

     But that wasn’t it. There was a slight problem. Not all the active interfaces on the system had public IPs. Some had private, local LAN IPs in the and subnets. The code above was returning all the IPs it could find, including public and private ones.

     I then found the netaddr third-party Python module which provided a Pythonic means of manipulating network addresses. I modified my code to use the netaddr module and got the following to boot with:

def get_inet_ips():
    import netinfo
    from netaddr import IPAddress, AddrFormatError
  except ImportError:
    return None
    inetIPs = []
    for interface in netinfo.list_active_devs():
      if not interface.startswith('lo'):
        ip = netinfo.get_ip(interface)
          ip_address = IPAddress(ip)
        except AddrFormatError:
          # If the IP is not private, use it.
          if not ip_address.is_private():
    return inetIPs

     The netaddr.IPAddress.is_private() method in the code above determines whether the given IP is part of any of the defined private networks.

     Admittedly, there is much room for improvement in the code above. I can only hope that if it doesn’t help, then at the very least it serves as an interesting read.

Browsing on the BlackBerry simulator

     I am not a big fan of BlackBerry smart-phones. I realize that there are a lot of people who can’t seemingly exist without access to their emails virtually all the time, and for those lot, BlackBerry, with its prominent push email feature, is perhaps a better fit than any other smart-phone platforms out there. When it comes to me and my smart-phone usage, I would not go so far as to say that I can’t live without my phone. I can. By every measure, I consider myself a hardcore geek, perhaps more hardcore than most others, but I am by no means a gadget freak. While it will be unfair to say that I absolutely abhor typing on small (semi-) keyboards, I don’t also quite enjoy the experience. When it comes down to typing, I would much rather prefer a full-fledged keyboard. That is why to me a compact laptop is many times more important than a fully equipped smart-phone. (For the curious reader, I own a Nokia E72.)

     For a recent mobile website project that I worked on, I had to face a complaint from the client where the layout of certain pages on the site didn’t look quite as expected on BlackBerry devices. Naturally, I didn’t have a BlackBerry handset nor an easy equivalent to test the issue for myself, so I did what anyone stuck in the same corner as I would do: I went over the BlackBerry developer portal online to look for BlackBerry simulators.

     Unlike the Epoch simulator for Symbian/Nokia phones and the iPhone simulator, the BlackBerry simulators were spread out such that for each possible BlackBerry smart-phone model in existence, there was a simulator available for it. And each one of the download was anywhere from 50 to 150 MB in size.

     I chose the simulator for one of the latest BlackBerry handsets, and downloaded it. Like the Epoch simulator, BlackBerry simulators are Windows-specific, in that, they are available in the form of Windows executable binaries. I didn’t have Windows anywhere in my study, so I had to set up a Windows guest inside VMware Fusion in order to set up the simulator. To cut a long, painful story short, I was able to install the simulator after tirelessly downloading a big Java SDK update, without which the installation wouldn’t continue. And then, I powered up the simulator. I was instantly reminded of the never-ending pain I had to suffer through the hands of the Epoch simulator in my previous life where I used to be a Symbian developer. The BlackBerry simulator took ages to start up. I almost cursed out loud because that fact alone opened up old, deep gashes that I had thought I had patched up for good. I was mistaken. Never in my dreams had I thought of having to deal with such monstrosity ever again. And, to my utter, absolute dismay, here I was.

     Eventually, after what seemed to have been ages since I booted up the simulator, I was able to navigate my way to the BlackBerry browser. I let out a deep sigh and thought that I could now finally concentrate on the problem I set out to tackle. But, no! I couldn’t browse on the BlackBerry browser at all. No amount of fiddling with the 3G and WiFi settings all over the BlackBerry OS got browsing working. From what I could tell, both the 3G and WiFi networks were alive, but there was no traffic flowing through. I almost gave up.

     After groping on the Internet with a wince on my face, I was finally able to find out why. Apparently, by default, the BlackBerry simulator are unable to simulate network connections. In order to do this, you have to download and install an additional, supplementary BlackBerry simulator that is called the BlackBerry MDS Simulator. Once this simulator is up and running, your actual BlackBerry simulator will be able to simulate network connections, browse, and do all sorts of network related functions. Who knew!

     As an aside, there’s also the BlackBerry Email Simulator that simulates messaging functionality.

Ranting about Mercurial (hg) (part 1)

     The day I got a grip of git, I fell in love with it. Yes, I am talking about the famous distributed version control application: git. I was a happy Subversion (svn) user before discovering git. The day to day workflow with git and the way git worked made me hate Subversion. I was totally head over heels in love with git.

     I have come to a point where the first thing I do when I am about to start a new project is to start a git repository for that project. Why? Because it is so easy and conveneient to do. I really like the git workflow and the little tools that come with it. I know that people who complain about git often lament about its lack of good, proper GUI front-ends. While the state of GUI front-ends for git has improved incredibly and drastically over the years, I consider myself a happy command-line user of git. All my interactions with git happen on the command-line. And, I will admit, it does not slow me down or hinder my progress or limit me in any way in what I do. I have managed fairly large projects on git from the command-line, and let me tell you, if you are not afraid of working on the command-line, it is extremely manageable. However, I understand that a lot of people run away from the command-line as if it were the plague. For those people, there are some really good git GUI front-ends available.

     I have to admit that for someone coming from no version control background or centralized version control background (such as Subversion or CVS), the learning curve to git could turn out to be a litle steep. I know that it was for me, especially when it came to getting to grips with concepts involving pushing, pulling, rebasing, merging, etc. I don’t think I will be off-base here to think that a good number of people who use version control software for fairly basic tasks are afraid of merge conflicts in code when collaborating code. I know I was always afraid of it, until I actually finally faced a merge conflict for the first time. That was the last time I felt afraid of it, because I was surprised to find out how easy it was to deal with and (possibly) resolve merge conflicts. There’s no rocket science to it.

     I have been plenty impressed with git that I wince when I am forced to use any other version control. In fact, I refuse to use Subversion, as an example, and try my best to convince people to move to git. Why? Because git is just that good. I should probably point the reader to this lovely article that puts down severals reasons that explain why git is better that a number of other popular version control applications out there.

     Recently, at work, I have had to deal with the beast they call Mercurial (hg). Yes, it is another famous distributed version control system out there. I was furious at the decision of my peers to select Mercurial for work-related use, and tried my best to convince them to use git instead, but they didn’t give in to my attempts to persaude them. I had no choice but to unwilling embrace the beast and work with it.

     Instantly, I had big gripes with the way Mercurial worked as well as its general workflow. Where I fell in love with how git tracked content and not files, I hated the fact that Mercurial only tracked files. What I mean by that is if you add a file in a Mercurial repository to be tracked and commit it, and then make any changes to the file, Mercurial would automatically add it to the staging commit. Yes, it wouldn’t let you specify manually which files that were changed should be part of the staging commit, something that git does. A staging commit is a staging area where files that have been changed and have been selected to be part of the next commit are loaded to be ready for commit. With git, you have to manually specify which file to add to the staging commit. And I loved doing it that way. With Mercurial, this was done automatically. This in turn brings me to the second big gripe I had with Mercurial: incremental commits. Git workflow places an indirect emphasis on incremental commits. What are incremental commits? In the version control world, it is considered best practice to perform a commit that is as small as possible and that does not break any existing functionality. So, if for example, you have made changes to ten files to add five different functionalities, that may be independent of each other, but would only like to commit one functionality at a time. That is where incremental commits come in action. With incremental commits, you specify only the files you want to be committed and commit them, instead of committing all the changed files in a bunch and add a general purpose commit message. Incremental commits is a big win for me, and I adore that feature. Because with git you had to manually specify which of the changed files you wanted in the staging commit area, incremental commits came easily and naturally. With Mercurial, where it automatically added all changes to tracked files into the staging commit area, it did not.

     So I asked the obvious question: How do we do incremental commits with Mercurial? Aftering sifting through the logs, I found out that it was indeed possible, but in a very ugly way. The Mercurial commit command takes an extra parameter, -I, that can be used to specify which files explicity to make part of the commit. But that wasn’t it. If you had, say, five files of the ten changed files you wanted to commit, you would have to tag the -I switch behind each of those files, otherwise, Mercurial will fail to include the files in the commit. To this day, this little quirk bites me every now and then.

     I will admit that the one thing I really liked about Mercurial was the built-in web server it came with that could provide a web-based interface to your Mercurial repositories for viewing. This was something that was missing from or quite difficult to do with git. And, this was also one of the biggest reasons why my peers at work decided to go with Mercurial for. They all wanted a convenient way to access the repositories over the web.

To be continued.

Creating click-able hyper-links in CSV files programmatically

If you are programmatically creating CSV files intended for Microsoft Excel users, and cannot figure out how to make URL data in the CSV files behave like click-able hyper-links for your users, this tip is for you.

There is a special format in which URLs should be formatted, that is used by a number of spreadsheet applications to interpret URLs as click-able hyper-links. The format is this:


The URL must be enclosed in quotes for the format to be interpreted properly (without which, at least, Microsoft Excel complains).

Short and useless commit messages

I have been guilty of writing short, pointless, and completely useless commit messages [before]. Stoked by a recent thread on StackOverflow about the worst commit message that one has ever authored, I looked into the commit log for a project I have been working on and subsequently maintaining for the last one year. I had to learn git as it had been decided that git would be used for version control for the project. Git indeed has been used since the onset of the project, but during the first three quarters working on the project, due to my unfamiliarity with working with distributed version control and shameful lack of knowledge of how git worked and a growing misperception that git is overly complicated for beginners to distributed version control, I used git sparingly (in the sense that I did not commit regularly, nor did I use any of its more than simple features) and ham-handedly. Nevertheless, skimming through the commit log from since the very initial commit, I found the following gems:

  • minor changes
  • some brief changes
  • assorted changes
  • lots and lots of changes
  • another big bag of changes
  • lots of changes after a lot of time
  • LOTS of changes. period

Of course, the ever notorious “bug fixes” is also to be found at a lot of places.

On a happier note, since then, I have come to understand and utilise the full potential of git. And I no longer blurt out when authoring commit messages.

Debugging Django applications!

When developing Django applications, there can be many times when you would want to roll up your sleeves and get your hands dirty. You may want to know, for example, why the control isn’t falling into a certain block of code, what values are being returned to the view from the browser or test client, why a certain form is bailing out on you, or any manner of possible problematic scenarios. For these and many more, there are a number of things you can do to help yourself and your code. I am going to describe a couple of those that I frequently employ.

1. Quick and dirty debugging

Yes! It is the quickest, dirtiest, and easiest form of debugging that has been around since who knows when. Like with many other programming languages, it takes the form of Python print statements for Django. It is really helpful when you are running your Django applications off of the Django development server, where the print output make their way to the console from which the development server is run.

2. Python Logging framework

The Logging framework for Python is as easy to use as they come. The use of the logging framework resembles to some extent quick and dirty debugging, but goes much further in terms of the flexibility as well as the levels of sophistication it provides. The simplest and quickest use is to import the logging module in the file carrying the code you want to debug, and use any of the available log message creation methods: debug(), info(), warning(), error(), etc. The output from these methods will make their way to the console where the development server is running.

With the default settings, however, the logged messages do not provide useful information beyond what you tell them to print. Also, you have no control over where the messages end up showing. So, say, if you are running your Django project over Apache with mod_python or mod_wsgi, you may be up a stump trying to locate where the messages go, or you may want to keep the messages aloof in a different file, but will find that the default settings for the logging framework won’t be able to lend you much room to breathe.

However, that is where the Logging framework really shines. It is configurable to a great extent. The docs for the framework give detailed information about the different nuts and bolts of the framework and the different ways in which it can be tuned. For the sake of this article, I will briefly brush over a slightly basic configuration that I use the logging framework in when debugging Django applications. I simply create a basic configuration setting for the logger, and move it into the file for the Django project. It looks like this:

import logging
  format='%(asctime)s %(funcName)s %(levelname)-8s %(message)s',
  datefmt='%a, %d %b %Y %H:%M:%S',
  filename='/tmp/project.log', filemode='w')

Not only does this separate the log messages to the file /tmp/project.log, it also adds useful debugging information to the start of each logged message. In this particular case, the date and time, the name of the function from which the logging method was called, the logging level, and the actual message passed are displayed. All these and much more are thoroughly documented with elaborate examples in the documentation for the logging module.

3. The Python Debugger (pdb)

You may probably already have used the pdb module before for debugging Python scripts. If you have not figured out already, you can just as easily use pdb to debug your Django applications.

If you are like me, you may have got into the habit of writing unit tests prior to writing down Django views that the unit tests test. It is a wonderful habit, but it can get unnerving at times when you are writing your tests first. This is primarily because of the way the unit tests interact with your code: Once written, they run in their entirety without any form of interaction from the user. If any number of tests fail, the fact is made clear at the conclusion of the test runner. What I want to say is that there is no easy way for you to interact when the tests are being run.

With pdb, you can have a moment to sit back and take a sigh of relief. By importing the pdb module, and calling the pdb.set_trace() method right before the point where you want to start to debug your code, you can force the test runner to freeze itself and drop to the familiar, friendly pdb prompt. This helps immensely when you want to find out, for example, why a form that you are unit testing is not validating, what error messages it is receiving, what errors or outputs it should produce so that your tests can simulate those, etc. Once at the pdb prompt, you may use the usual lot of pdb commands to inspect and step through the code.

The use of pdb, however, it not restricted to unit testing. It may equally well be used when serving your Django applications over the development server. However, there is one little detail that needs to be accounted for. When you want to debug your Django applications over the development server with pdb, you must start the development server with these additional switches:

$ python -m pdb runserver --noreload

The -m pdb switch is documented in the documentation for the pdb module. Simply, it makes sure that if the program being debugged burps and crashes (either owning to an error, or when stimulated such as by calling the pdb.set_trace() method), the pdb module automatically falls flat on its face and activates the post-mortem debugger. This is very convenient, because what it means is that the friendly pdb prompt shows up, and you can dissect the code from that point onwards.

The --noreload switch to the runserver command, however, is crucial. The Django development server is designed to automatically reload the Python interpreter if there’s a crash or error of some sort, or to reread all the Django files if there has been a change in any one of those. One fallout of this default behaviour of the development server is that since the Python interpreter is reloaded, all previous context is lost, and therefore, there is no way for pdb to save face. The --noreload switch, therefore, forces the Django server to stray off of its default behaviour.

With the development server running with these switches, all you have to do is make sure you have placed calls to pdb.set_trace() method in your code where you want to break out. And that’s as easy as it gets.

I hope that what I have described finds its way into the useful bucket of my readers. For now, that’s all. I hope you do enjoy, if you did not before, debugging your Django applications after reading this article. Please, stay safe, and good bye!

A guide to configuring two different Django versions for development

Django 0.9x and Django 1.x branches are so far apart that a project built on top of the former, not least when it has been rolled off into production, that porting it to the 1.x branch of Django can easily escalate into a nightmare. If you have to maintain a legacy Django project — I choose to call projects built on anything before 1.x legacy –, you will find that it is easier if you maintain two different versions of Django on your development environment. How do you do that, is the quagmire this post revolves around.

I prefer Apache with mod_python to Django’s simplistic development web-server for Django projects. Initially, the task of setting up mod_python and Apache to your tastes may seem daunting. However, once you get around its not so steep curve, the difficult task becomes second nature.

I am working with Python 2.5 on a project with Django 1.0.2. Django is deployed on the system in the site-wide default directory for third-party Python modules and applications. I also have to support development for a legacy application built on Django 0.97 (a trunk snapshot from a time long ago). My requirement is simple: I want to be able to run both projects simultaneously without having to muck around tweaking configurations besides the one time when I set the projects up.

A garden-variety solution is to change the default Django directory in the Python path to point to the specific Django that is the need of the moment. This approach puts upfront a limitation of not being able to run both projects side by side. It also is annoying because it requires tap-dancing around Django directories when switching between the two versions. It is not a considerable blow to productivity, but if it can be avoided for a better, more efficient solution, there is no reason not to.

The solution I am proposing involves having the recent Django set up as the default, with the legacy Django installed in a different directory—after all, the legacy Django is the odd item of the group. I develop on an OS X environment, where I have the two Django versions set up thus:


You can probably tell which is which.

I have both projects deployed as virtual hosts on Apache, each running off of the root of the web-server without stepping on the feet of the other. They can easily instead be set up to, instead of the root of the web-server, serve from under a unique sub-URL path. That is mostly a matter of preference.

For each virtual host, I have assigned a domain name based off of the name of the project. I should emphatically point out that these domains are set up to resolve to localhost on my system. They may likely have real world twins that are routable over the Internet, but for all intents and purposes, they are local to my development environment. I have done it so, partly out of convenience, and partly because one of the projects hinges on subdomain-based user-to-account mapping (what this means, simply, is that registered users on the project are assigned different accounts that directly relate to subdomains, and can log in to the project only via their respective subdomains) for proper functioning. For the latter, the domain based approach was inevitable.

With that in mind, here are the virtual host settings for the two projects

<VirtualHost *:80>
   ServerAdmin root@localhost
   <Location "/">
      SetHandler python-program
      PythonHandler django.core.handlers.modpython
      SetEnv DJANGO_SETTINGS_MODULE projectone.settings
      PythonPath "['/Users/ayaz/Work/', 
            '/Users/ayaz/Work/projectone/'] + sys.path"
      PythonDebug On

<VirtualHost *:80>
   ServerAdmin root@localhost
   <Location "/">
      SetHandler python-program
      PythonHandler django.core.handlers.modpython
      SetEnv DJANGO_SETTINGS_MODULE legacyproject.settings
      PythonPath "['/Library/Python/2.5/site-packages/django-0.97', 
            '/Users/ayaz/Work/', '/Users/ayaz/Work/legacyproject/'] + sys.path"
      PythonDebug On

While each line of the above is important, the following is the highlight of this post:

   PythonPath "['/Library/Python/2.5/site-packages/django-0.97', 
            '/Users/ayaz/Work/', '/Users/ayaz/Work/legacyproject/'] + sys.path"

I have effectively injected the path to Django 0.97 into Python path. What this helps achieve is that when mod_python loads the Python interpreter to serve an incoming request for the project, Python analyses its path, looking for a module named django. The first successful match is under the /django-0.97 directory, which, if I have not already lulled you into sleep, is where Django 0.97 lives.

For the curious, all possible mod_python directives are documented here.

All this is a one-time headache: you pop in a pill, and forget about it. I can now type in or in the browser to get to either project.

I should mention, still and all, that I have of late come to know of virtualenv, a Virtual Python Environment builder. It may be a more proper solution than what I have proposed, but I have not yet used it myself to say more.

I hope I was able to clearly explain what I had intended to.

Symbian: Folded descriptor contents, and case-insensitivity

UPDATE: I found this page from Symbian docs online that describes folding and collation in detail.

In Symbian jargon, the term “folded” means “case-insensitive“. If you grope throughout the Symbian docs and API looking for the term case-insensitive, you will be hard-pressed to find anything.

If you have dabbled with descriptors enough, you may be familiar with methods like Match(), Find(), Compare(), etc. By default, their operations are case-sensitive. However, there are folded variants of these methods available too, for, you guessed it, case-insensitive mode. They have an ‘F’ attached to the end of their names, such as, MatchF(), FindF(), CompareF(), etc. They work by folding the contents of the descriptors which are their arguments first, and then applying their respective functional semantics on them — or, simply, for MatchF(), it applies a case-insensitive match operation.

It is not every day that you may need to perform any operation against data without worrying about the case-sensitivity of that data, but when you do, and if you find yourself working with Symbian, you may likely quickly find yourself floundering in the API, without much or any help.

That is just Symbian.

Symbian: iCode for ‘Enter’ and ‘c’ keys on Nokia phones.

The following won’t make sense to most of the people reading my blog. However, hopefully, it will help out fellow Symbian developers from pulling off their remaining hair looking for the solution I break out in the space of this post.

Nokia phones have a ‘c‘ key which mostly deletes selected items or text. They also have an ‘Enter‘ key, which is mostly in the form of the somewhat bigger middle button on the navigation pad. Symbian developers looking to catch signals when either of those keys are pressed would be hard-pressed to get a taste of success.

One would think ‘EKeyEnter‘ and ‘EKeyDelete‘ would be the right iCode values to catch the ‘Enter’ and ‘c’ key events, respectively. However, that’s not how it works. Surprising, I know. With the help of jelle on #symbian (, I found out that the iCode for grabbing Enter key event is ‘EKeyDevice3‘. Similarly, ‘EKeyBackspace‘ is the iCode for the ‘c’ key event.

It is as weird as it gets. But being a Symbian developer, you know this is nothing in the face of the greater bizarre things that make up the Symbian OS.

Paste Mode: Pasting text (and indenting it) in Vim

Pasting a block of code copied from elsewhere in Vim is a nightmare, unless you know how to do it right. The autoindent (and, possibly, smartindent) features of Vim go crazy when you yank and slap a block of text. How do you do it right, then? This is not a new trick, but something I did not know about for a long time, so I suppose it is reasonable to assume many others who use Vim may not have come across it either. It is called the “paste mode“. In Vim, you can enable the paste mode by typing “:set paste” on Vim’s command interface. Once the paste mode is enabled, you can paste the block(s) of code you have yanked from elsewhere, and disable the paste mode through the command “:set nopaste” as soon as you are done with pasting text. A bit inconvenient, but it is a handy feature to get familiar with if you use Vim a lot to write code.

Python programmers would run into syntax error problems if they execute source code files in which they pasted code in the paste mode. I am not completely certain on this, but when text is pasted in paste mode, the copied lines are indented with spaces irrespective of the tab character and tab width defined for the open Vim session. If you tend to indent source files by the tab character (something probably any sane Python developer should do), this will definitely create problems. There is an easy way to fix this. Just after you turn off the paste mode, highlight the pasted text by moving the cursor to the first line pasted, pressing “shift + v” to enable “visual” mode in Vim (which is used solely for highlighting blocks of text), and dragging the highlight marker down until all the pasted text is highlighted, and pressing either the right angle bracket (>) or the left (<) to conveniently indent the block of highlighted text right and left according to the tab character and tab width set for the current Vim session. Simply indenting left once, and then right once will get the block of code back to its original indent position, but will additionally (and this is the important bit) indent the block according to the currently defined tab width and tab character settings. Pretty convenient, eh?

Mid-week nightly Perl useless magic and fun!

Last night, I was sitting at my desk, watching text scroll by across an IRC session open in a terminal screen set in black, when my eyes came to halt at Pixelized on ##slackware on FreeNode soliciting help with a particular use of grep. To cut the rather overly technical story short, what Pixelized wanted to do was grok all text in a file between the keywords “X1” and “.bat”, inclusive. Being a long time Perl guy, I quickly threw two Perl one-liners that did pretty much what he had asked for. Both forms work perfectly accurately for my selected data set, but appeared to be slightly misbehaving against Pixelized data set. It was late, and I was already drooping with sleep, carrying a cracking headache, so I never bothered asking him for the data set. Anyhow, I thought I would share the two Perl one-liners just for the heck of it here.

perl -e 'while (<>) { if (/X1/ .. /\.bat/) { s/.*(X1.*)/\1/; s/(.*\.bat).*/\1/; print $_; } }' dump.txt

perl -e '$c = 0; while (<>) { if (/X1/) { $c = 1; s/.*(X1.*)/\1/; } if (/\.bat/) { s/(.*\.bat).*/\1/; print $_; last; } print $_ if $c; }' dump.txt

Unquote HTML-quoted strings in Python

I wasted a lot of time looking for something in Python to decode (if that is the right word, otherwise, unquote) HTML-quoted strings such as those you get back as form values. Despite using the module “urllib“ to retrieve the form values in the first place, I was not aware of the set of utility functions it offers for quoting and unquoting of values. So, here you go: Look at “unquote()“ and “unquote_plus()“.