Guide: Deploying web.py on IIS7 using PyISAPIe

I spent the last week travailing away, trying to painfully find a way to deploy a web.py based API on IIS7 using PyISAPIe. As frustrations had begun to mount up, I had nearly decided to give up. Being a die-hard Linux and Mac guy, I despise having to work on Windows. Here I was, not only forced to work on Windows, but to find a solution for a problem that left no leaves unturned in its effort to drive me crazy. As if someone decided all this misery wasn’t quite enough, I had to work with a remote desktop session in order to research, tweak, bang my head, and get things to work. Eventually, I cut through massive frustration and despair, managing to find a satisfactory solution. I almost danced in excitment and relief, letting out all sorts of expletives directed at Windows in general and IIS in particular.

To get back to the important question of deploying a web.py script on IIS7 using PyISAPIe, I will make it such that this guide will list down various steps I took, including snippets of relevant code I changed, to tame the beast. I can only hope that what is below will help a poor, miserable soul looking for help as I did (and found none).

I worked with PyISAPIe because I had successfully deployed multiple Django websites on IIS7 on it. The script in question was going to be a part of another Django website (though acting independently). It only made sense to use PyISAPIe for it as well.

First and foremost, I had to install the web.py module on the system. Having had trouble before with IIS with web.py installed through easy_install, I decided to be safe and installed it from source.. Getting web.py to work with PyISAPIe required a small hack (I notice I may make it sound as though it all came down to me in a dream, but in reality, it took me days to figure it out, and clearly after much anguish and pain). In the file Lib\site-packages\web\wsgi.py lies the following function:

def _is_dev_mode():
    # quick hack to check if the program is running in dev mode.
    if os.environ.has_key('SERVER_SOFTWARE') \
        or os.environ.has_key('PHP_FCGI_CHILDREN') \
        or 'fcgi' in sys.argv or 'fastcgi' in sys.argv \
        or 'mod_wsgi' in sys.argv:
            return False
    return True

In its pristine state, when web.py is imported from a source file through PyISAPIe, an exception is thrown. The exception, while I don’t have the exact message, is about it complaining about sys.argv not having an attribute argv, which reads fishy. Since the function _is_dev_mode() only checks whether web.py is being run in development mode, I thought I didn’t care about it since I wanted everything to run in production mode. I edited the function such that its body would be bypassed, while it returned a False boolean value. It looked like this (the important changes I made are highlighted):

def _is_dev_mode():
    return False
    # quick hack to check if the program is running in dev mode.
    if os.environ.has_key('SERVER_SOFTWARE') \
        or os.environ.has_key('PHP_FCGI_CHILDREN') \
        or 'fcgi' in sys.argv or 'fastcgi' in sys.argv \
        or 'mod_wsgi' in sys.argv:
            return False
    return True

This innocuous little addition did away with the exception.

Next up, I used default Hello World-esque example of web.py found on their site to test the deployment (of course, I went on to use my original API script, which was far too complex to trim down and fit into as an example). I called it code.py (I placed it inside the folder C:\websites\myproject). It looked like this:

  import web
  urls = (
      '/.*', 'hello',
      )
  class hello:
      def GET(self):
          return "Hello, world."
  application = web.application(urls, globals()).wsgifunc()

It was pretty simple. You have to pay particular attention on the call to web.application. I called the wsgifunc() to return a WSGI-compatible function to boot the application. I prefer WSGI.

I set up a website under IIS using the IIS Management Console. Since I was working on a 64-bit server edition of Windows and had chosen to use 32-bit version of Python and all modules, I made sure to enable 32-bit support for the application pool being used for the website. This was important.

I decided to keep the PyISAPIe folder inside the folder where code.py rested. This PyISAPIe folder contained, of import, the PyISAPIe.dll file, and the Http folder. Inside the Http folder, I placed the most important file of all: the Isapi.py. That file could be thought of as the starting point for each request that is made, what glues the Request to the proper Handler and code. I worked with the Examples\WSGI\Isapi.py available as part of PyISAPIe. I tweaked the file to look like this:

from Http.WSGI import RunWSGI
from Http import Env
#from md5 import md5
from hashlib import md5
import imp
import os
import sys
sys.path.append(r"C:\websites\myproject")
from code import application
ScriptHandlers = {
	"/api/": application,
}
def RunScript(Path):
  global ScriptHandlers
  try:
    # attempt to call an already-loaded request function.
    return ScriptHandlers[Path]()
  except KeyError:
    # uses the script path's md5 hash to ensure a unique
    # name - not the best way to do it, but it keeps
    # undesired characters out of the name that will
    # mess up the loading.
    Name = '__'+md5(Path).hexdigest().upper()
    ScriptHandlers[Path] = \
      imp.load_source(Name, Env.SCRIPT_TRANSLATED).Request
    return ScriptHandlers[Path]()
# URL prefixes to map to the roots of each application.
Apps = {
  "/api/" : lambda P: RunWSGI(application),
}
# The main request handler.
def Request():
  # Might be better to do some caching here?
  Name = Env.SCRIPT_NAME
  # Apps might be better off as a tuple-of-tuples,
  # but for the sake of representation I leave it
  # as a dict.
  for App, Handler in Apps.items():
    if Name.startswith(App):
      return Handler(Name)
  # Cause 500 error: there should be a 404 handler, eh?
  raise Exception, "Handler not found."

The important bits to note in the above code are the following:

  • I import application from my code module. I set the PATH to include the directory in which the file code.py is so that the import statement does not complain. (I’ve to admit that the idea of import application and feeding it into RunWSGI came to while I was in the loo.)
  • I defined a script handler which matches the URL prefix I want to associate with my web.py script. (In hindsight, this isn’t necessary, as the RunScript() is not being used in this example).
  • In the Apps dictionary, I again route the URL prefix to the lambda function which actually calls the `RunWSGI` function and feeds it application.
  • I also imported the md5 function from the hashlib module instead of the md5 module as originally defined in the file. This was because Python complained about md5 module being deprecated and suggested instead of use hashlib.

And that’s pretty much it. It worked. I couldn’t believe what I saw on the browser in front of me. I danced around my room (while hurling all kinds of expletives).

There’s a caveat though. If you have specific URLs in your web.py script, as I did in my API script, you will have to modify each of those URLs are add the /api/ prefix to them (or whatever URL prefix you set in the Isapi.py. Without that, web.py will not match any URLs in the file.

What a nightmare! I hope this guide serves to help others.

Thank you for reading. Good bye!

PS: If you want to avoid using PyISAPIe, there is a simpler way of deploying web.py on IIS. It is documented crudely over here.

Advertisements

Converting a Git repo to Mercurial

     Until today, we had most of the projects on Mercurial repositories hosted on a Windows box in-house. The Mercurial web interface was set up to provide a convenient read-only access to both contributors and non-contributors within. However, the box in question was unreliable, and so was the Internet link it was alive on. And, in this world, there are few things more troublesome than having your source control server unavailable due to an unexpected downtime.

     What’s special about today is that we moved most of our Mercurial repositories to BitBucket.org. While they will remain private, the contributors as well as the internal non-contributors will be given access to them. Aside from having the repositories on a central server that we can mostly always rely on, having a lovely Web interface to browse and control the repositories, we also get useful goodies like issue trackers, wiki pages, and an easy user management interface. It is a win-win situation for everyone.

     One of the projects I was working on had only been on a local Git repository. I started work on it at a time when I found myself deeply in love with Git. Since BitBucket is a Mercurial repository warehouse, I had to find a way to convert or migrate the Git repository into a Mercurial one.

     I looked around on the Web and found a lot of people recommending the use of the HgGit plugin. As I understood it, this plugin made possible, among other things, the workflow that involved working on a Git repository and pushing the changesets to a Mercurial counterpart. However, the process of setting it up seemed rather complex to me. Plus, I didn’t want to keep the Git repository lying around when I was done with the migration. I wanted to be able to migrate the Git repository to a Mercurial one, push it upstream to BitBucket and make any changes in future in the source code by cloning from the Mercurial repository. What HgGit did seemed rather overkill for my needs.

     I then discovered the Mercurial ConvertExtension. This extension did just what I wanted: Convert repositories from a handful different SCMs into Mercurial. The process of converting a Git (or any other repository) to a Mercurial one through ConvertExtension is very straightforward.

     As a first step, you are required to edit your global .hgrc file to enable the extension thus:


[extensions]
hgext.convert=

     You are then required to run the hg convert command on your target Git repository thus:


$ hg convert /path/to/git/repo

     This will migrate the Git repository, and store it in the current directory inside a directory named repo-hg. Once inside the newly created Mercurial repository (and this part is important), you have to run the following command to checkout all the changesets:


$ hg checkout

     You may then push the repository with the usual hg push to BitBucket.

     :)

     PS: This blog post really helped get me going in the right direction.

Browsing on the BlackBerry simulator

     I am not a big fan of BlackBerry smart-phones. I realize that there are a lot of people who can’t seemingly exist without access to their emails virtually all the time, and for those lot, BlackBerry, with its prominent push email feature, is perhaps a better fit than any other smart-phone platforms out there. When it comes to me and my smart-phone usage, I would not go so far as to say that I can’t live without my phone. I can. By every measure, I consider myself a hardcore geek, perhaps more hardcore than most others, but I am by no means a gadget freak. While it will be unfair to say that I absolutely abhor typing on small (semi-) keyboards, I don’t also quite enjoy the experience. When it comes down to typing, I would much rather prefer a full-fledged keyboard. That is why to me a compact laptop is many times more important than a fully equipped smart-phone. (For the curious reader, I own a Nokia E72.)

     For a recent mobile website project that I worked on, I had to face a complaint from the client where the layout of certain pages on the site didn’t look quite as expected on BlackBerry devices. Naturally, I didn’t have a BlackBerry handset nor an easy equivalent to test the issue for myself, so I did what anyone stuck in the same corner as I would do: I went over the BlackBerry developer portal online to look for BlackBerry simulators.

     Unlike the Epoch simulator for Symbian/Nokia phones and the iPhone simulator, the BlackBerry simulators were spread out such that for each possible BlackBerry smart-phone model in existence, there was a simulator available for it. And each one of the download was anywhere from 50 to 150 MB in size.

     I chose the simulator for one of the latest BlackBerry handsets, and downloaded it. Like the Epoch simulator, BlackBerry simulators are Windows-specific, in that, they are available in the form of Windows executable binaries. I didn’t have Windows anywhere in my study, so I had to set up a Windows guest inside VMware Fusion in order to set up the simulator. To cut a long, painful story short, I was able to install the simulator after tirelessly downloading a big Java SDK update, without which the installation wouldn’t continue. And then, I powered up the simulator. I was instantly reminded of the never-ending pain I had to suffer through the hands of the Epoch simulator in my previous life where I used to be a Symbian developer. The BlackBerry simulator took ages to start up. I almost cursed out loud because that fact alone opened up old, deep gashes that I had thought I had patched up for good. I was mistaken. Never in my dreams had I thought of having to deal with such monstrosity ever again. And, to my utter, absolute dismay, here I was.

     Eventually, after what seemed to have been ages since I booted up the simulator, I was able to navigate my way to the BlackBerry browser. I let out a deep sigh and thought that I could now finally concentrate on the problem I set out to tackle. But, no! I couldn’t browse on the BlackBerry browser at all. No amount of fiddling with the 3G and WiFi settings all over the BlackBerry OS got browsing working. From what I could tell, both the 3G and WiFi networks were alive, but there was no traffic flowing through. I almost gave up.

     After groping on the Internet with a wince on my face, I was finally able to find out why. Apparently, by default, the BlackBerry simulator are unable to simulate network connections. In order to do this, you have to download and install an additional, supplementary BlackBerry simulator that is called the BlackBerry MDS Simulator. Once this simulator is up and running, your actual BlackBerry simulator will be able to simulate network connections, browse, and do all sorts of network related functions. Who knew!

     As an aside, there’s also the BlackBerry Email Simulator that simulates messaging functionality.

Creating click-able hyper-links in CSV files programmatically

If you are programmatically creating CSV files intended for Microsoft Excel users, and cannot figure out how to make URL data in the CSV files behave like click-able hyper-links for your users, this tip is for you.

There is a special format in which URLs should be formatted, that is used by a number of spreadsheet applications to interpret URLs as click-able hyper-links. The format is this:

=HYPERLINK("URL")

The URL must be enclosed in quotes for the format to be interpreted properly (without which, at least, Microsoft Excel complains).

Red Alert 2 multiplayer on Windows Vista

Finally, there’s a working patch available for Red Alert 2 that transforms the game to utilise the UDP protocol on LAN instead of IPX (which is no longer supported on Microsoft Windows Vista). The patch requires the modified file, wsock32.dll, to be moved into the root folder of the game. I can hazard a guess that the author of the patch ingeniously hacked the DLL file and overwrote the network routines to use UDP, and since the DLL is available in the same folder as the game binary, it is loaded instead of the original DLL.

I use the patch on Microsoft Windows Vista Home Basic to play Red Alert 2 with a friend on Microsoft Windows XP. He, too, has the patch deployed. Give or take a few not too bothersome glitches sometimes, the game works flawlessly. In our configuration, both the players have the patch deployed. I cannot vouch for whether a mix of UPD and IPX game clients will work.

The patch can be found here.

DeltaCopy: rsync front-end for Windows – incremental file transfer

rsync is the first thing that pops in the mind when there is talk of fast incremental or differential backups or file transfers on and across Linux and Unix. With Cygwin, one may harness on Windows the power, flexibility and ease of incremental file transfers that rsync makes possible. There are no GUI front-ends to it for Windows as far as my knowledge stretches, and to be frank, I have never given rsync on Windows a jab.

However, now, there is DeltaCopy on Windows. Quoting straight from the website:

In general terms, DeltaCopy is an open source, fast incremental backup program …

In technical terms, DeltaCopy is a “Windows Friendly” wrapper around the Rsync program …

I cannot vouch for how well and reliably it works, and whether the interface is intuitive. Being hardly a Windows user, I have less of a reason to depend on GUI front-ends. However, if you are a Windows user and have been looking for an incremental backup solution or even a file transfer program that saves your precious time senselessly put to waste during file transfers, I do think DeltaCopy deserves at least one passing try.

Software patents: How bad they are, and how big companies love them

This post on software patents and copyrights and everything else in between is a means of letting off steam caused by reading news that Apple is taking ideas from commercial softwares being actively sold and trying to get patents for those ideas posing as concepts of their own. Yes: Ideas and concepts Apple has not conceived themselves but would like to legally call their own and demand, if and whenever they like, a royalty from anyone building on those ideas — or, in the worst case scenario, sever competition. Patents are considered evil and bad, and there are good reasons why.

Apple is not the only company who is doing it. Most big companies do it; have done it in the past. It has almost become a trend: big companies openly filching ideas from commercial softwares not their own, and attempting to patent those ideas as their own. For example, here we see Microsoft finally being granted a patent on “Page Up” and “Page Down” keystrokes. As another example, Microsoft owns a patent on the “Tree-View” mode we have come to love in many file-system applications. These are merely examples, and Microsoft and Apple are not the only big companies indulging in such practices.

In the software industry, being able to get a patent for an idea you have conceived that hits of really well is bad enough already that you have big companies going out patenting popular concepts in softwares that aren’t theirs to begin with. Besides giving that big company who did not think of a famous idea but now owns it an unfair advantage to play evil, it severely cripples the ability of other companies in general and developers in particular to be able to build upon that idea in order to build and sell better, bigger products, especially when such an idea is as basic and simple in nature as a window layout — most or all products need to need to build upon that.

One has to understand what a patent (software patent, in particular, in this context) is in order to fully grasp the extent to which they are a threat to, for example and in particular, the software industry. Let’s go over it with a simplistic analogy. I think of a brilliant idea, such as, say, tabbed-window browsing. No-one at this point has thought of it yet. I go out and roll-over a browser which features an implementation of my tabbed-window browsing idea. As set out in the US Copyright Law (and in Copyright Laws elsewhere mostly), any implementation or creation, the moment it is materialised into any tangible form, automatically becomes the property of the individual implementing or creating, and as such, that individual automatically gets the copyrights for it. Now, copyright and patent are two different things. At this point, I have the copyright to my implementation of my idea of tabbed-window browsing — the browser, or at least if we only concentrate on the implementation of the idea, the code that implements the tabbed-window browsing functionality is under my copyright. My idea, however, is not.

Ideas can not be copyrighted. They can be patented though. And that is where patents come in. You cannot copyright an idea, because, according to the US Copyright Law, for anything to be copyrightable, it has to be a work, in a form tangible, of an idea. An idea is not something tangible. That is all fine, but how are patents a threat to the software industry? Let’s imagine, further, that you, a big company with a not-so-great browser product, go out of the way and patent the tabbed-window browsing idea that I thought of. You get the patent, and now you legally own the idea. And, then, you plan to play dirty. Since my browser with the tabbed-window browsing support is gaining popularity at a breath-taking speed, which is more than hurting your browser market penetration, you charge me for patent infringement. Yes. I, with the implementation of the idea of tabbed-window browsing which you now legally own, am infringing on something that you have a patent for. Forget about the moral implications of your getting a patent for what you did not think of, I am committing a crime. And you can drag me to court for it. Easily. At this point, there are two things you can force upon me to choose from to do: Either force all my customers to pay me a royalty fee which in turn I pay back to you, and continue to let my browser remain in business or at least existence, thereby continually paying you a royalty fee for as long as the browser lives; or, force me to pull back my product, and end its life. What is worse perhaps is that that there is barely much of anything I can do about it.

Now, do you see how bad patents are? Fortunately, unlike how copyright applies automatically the moment you create a tangible form of your work, the process to acquire patents is a long and tedious one, which requires filing a patent application at the patent office, waiting for the patent office to approve and grant the patent, and everything in between. However, the brutal fact that you do not have to prove that you are the one in fact who actually thought of an idea in order to get a patent is, most unfortunately, not a requirement for you to get a patent for that idea. That is how big companies manage to run away with patents for ideas that belong to others. Couple that with the threat patents cause I described earlier, and you may see how deadly patents can become if brandished by evil companies to leverage ill-gotten advantages.

As an individual, and not least a software developer, tester, etc, there is little you can do about this, but it helps to know. Let’s turn up a notch for no software patents.