1. Python web development
    1. Google App Engine
    2. Going it alone
      1. WSGI: Code you can run
      2. Ugh. That was hairy.
      3. Other WSGI web servers
      4. What about the first problem?
      5. Templating languages
    3. And finally

Python web development

There are a lot of Python web frameworks. Here are just a smattering of the choices available:

  • django - a nice web framework to let you get on and do it
  • web.py - a mini web framework to let you get started fast
  • TurboGears - a web framework with object orientation
  • Pylons - a web framework to let you use what you need
  • Werkzeug - a web framework, similar to Pylons

All of these frameworks support WSGI, the Python alternative to CGI (although they support CGI too). WSGI is designed to make it easy to put together pieces you like (so-called "middleware").

Google App Engine

Recently, Google have stepped forward with their own Python web framework:

This lets you run your own code on Google's vast cloud of servers, so that you can write web sites which can support thousands or even millions of users while still being able to test and develop new features on your own laptop. And even if you don't want to use Google's servers at all, it's still quite a good introduction to Python web development.

It takes the templating engine from Django and combines it with their own mini web framework that's about as simple as web.py. Instead of a relational database, they include simple access to Google's persistent storage systems (but they keep most of the SQL syntax). This actually makes most things easier.

You can download their freely (Apache-) licensed SDK and have their Tutorial finished in under an hour. What are you waiting for?

Going it alone

Ok, you've decided you'd rather host your app yourself. Maybe you don't trust Google, or you think your app is too simple to need it, or it's not open enough for you, or you just love managing backups, load balancing, databases... Or maybe you just want the learning experience.

Whatever the reason, here's a quick introduction to do-it-yourself Python web development.

First, the simplest web app there is: 

print 'Content-Type: text/plain'
print
print 'Hello, World!'

This is CGI, which means that your web server just runs your program and spits everything it prints out straight at the user's browser. Everything before the first blank line (i.e. the Content-Type line) is an HTTP header, and gets consumed by the browser. In this case it says that this is just a text file, not a normal web page, because we're keeping things simple.

To run this script, you can put the code above into a file called "hello.py" in a directory called "cgi-bin" (that's important), and put this into "server.py": 

from BaseHTTPServer import HTTPServer
from CGIHTTPServer import CGIHTTPRequestHandler

server = HTTPServer(('', 8000), CGIHTTPRequestHandler)
print "Serving: http://localhost:8000/ ..."

# Respond to requests until process is killed
server.serve_forever()

Then when you run "python server.py", your "Hello, World!" page will appear at http://localhost:8000/cgi-bin/hworld.py. For simple sites that won't need to handle much load, that's fine (although the URL is ugly). At some point though, you'll want to run a proper web server, such as Apache. But that's a real pain.

If you do try to use CGI, you'll find your site much easier to debug if you insert this as the first line of each page script after the "#!" line: 

import cgitb; cgitb.enable()

or this line if this is an insecure (public facing) site: 

import cgitb; cgitb.enable(display=0, logdir='/tmp')

Unfortunately, there are at least four big problems with just using CGI. Firstly, as soon as it comes to writing web pages rather than text files, almost everyone gets it wrong. Simple though it may seem, the depressing truth is that writing valid web pages is basically too hard for humans. We'll address how to fix this later.

Secondly, it's really slow. Every time a user tries to view a page on your web site, the web server starts up another copy of Python, which takes time. This also causes the third problem -- that you can't have many people visiting the site before it stops working, because your web server probably can't handle running thousands of copies of Python simultaneously.

Fourthly, it's not composable - you can't just slap on that nice login mechanism you found, together with someone else's compression library, together with... . Now, to be honest, this one tends to get overblown, as most web sites don't make use of this much. So we shan't dwell on it, but it is at least worth mentioning.

But if you don't want to have one file per URL, or don't want to start up a new Python interpreter for every page view, you need to move away from CGI, which is where WSGI comes in.

(For those that need to squeeze out the last drop of speed, SCGI is a good choice too, but you can look that up elsewhere, and you'll need a real web server first).

WSGI: Code you can run

This code is all you need to run your own web site (assuming you have Python 2.5 or later): 

from wsgiref.simple_server import make_server, demo_app

server = make_server('', 8000, demo_app)
print "Serving: http://localhost:8000/ ..."

# Respond to requests until process is killed
server.serve_forever()

(from the docs for wsgiref.simple_server)

Save that in myapp.py, run python myapp.py, and you've got a complete web server, running a demo app that you can visit in a web browser. (Press Ctrl-C or Ctrl-Break to stop it.)

But it's a bit of a cheat, because we imported demo_app, so it gives no real idea of how complicated it would be to write our own. So here's a really complete version: 

from wsgiref.simple_server import make_server

def hello_app(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return ['Hello, World!']

def run():
    server = make_server('', 8000, hello_app)
    print "Serving: http://localhost:8000/ ..."
    server.serve_forever()

if __name__=='__main__':
    run()

The last two lines just run the run() routine, which you should recognise from the code above, except that now it's running our hello_app rather than demo_app. So the only important change is hello_app.

And what is hello_app? Well, it's a normal Python function. It's also a "WSGI application". That means that when a user requests a web page from the server, hello_app gets called with two arguments, environ and start_response. environ contains all of the information that the app might need to know in order to work out what to do. Since hello_app always does the same thing (prints 'Hello, World!'), environ gets entirely ignored.

start_response is another Python function. The WSGI app (hello_app) must call it with the HTTP status code (usually '200 OK' unless something has gone wrong) and the list of headers, [('Content-Type', 'text/plain')], which you may recognise as being the equivalent of print 'Content-Type: text/plain' from our CGI app.

And after calling start_response, all our app has to do is return a list of strings to return to the user, in this case, with return ['Hello, World!']. (Strictly, it can return any iterable, but one step at a time...)

Ugh. That was hairy.

Ok, it was a bit. That's one reason why most people use a web framework, to start to hide all of this gunk, while keeping the ability to let their code run fast.

The other reason is that there are an awful lot of problems involved in web development, and it's useful to re-use the work that other people have done for you. So now you have some slight idea what's happening behind the scenes (which is the key to understanding the web and avoiding painful mistakes), it's probably a good idea to bite the bullet and learn one of the frameworks above.

Other WSGI web servers

Before finishing with WSGI, I should also mention that the web server included with Python that was used above is quite simple and slow. There are better ones included in Python Paste and CherryPy, as well as several others listed at http://www.wsgi.org/wsgi/Servers. Fortunately, you can do your initial development with the default one and upgrade when you need to.

The CherryPy WSGI server is just a single python file (download) which you can distribute along with your app. If you save it as wsgiserver.py, the example above becomes: 

from wsgiserver import CherryPyWSGIServer

def hello_app(environ, start_response):
    start_response('200 OK', [('Content-Type', 'text/plain')])
    return ['Hello, World!']

def run():
    server = CherryPyWSGIServer(('0.0.0.0', 8000), hello_app)
    print "Serving: http://localhost:8000/ ..."
    try:
        server.start()
    except KeyboardInterrupt:
        server.stop()

if __name__=='__main__':
    run()

If you are happy to use the whole of CherryPy, just change the first line to 

from cherrypy.wsgiserver import CherryPyWSGIServer

But if you're going to do that, you might as well use the whole CherryPy "Hello World" example: 

import cherrypy

class HelloWorld(object):
    def index(self):
        return "Hello World!"
    index.exposed = True

cherrypy.quickstart(HelloWorld())

(Yes, it looks completely different, that's what using a framework can do.)

The equivalent Python Paste example can be found in the Python Paste WSGI Tutorial.

What about the first problem?

Oh yes, the first problem.

Writing valid web pages is basically too hard for humans.

The problem with writing valid web pages is that web pages are written in HTML, which is the language* that your web browser speaks, and humans don't. So they make mistakes. Nonetheless, to be a decent web developer, you have to be able to read it. You can learn HTML in a million places on the web, with W3 Schools being a popular recommendation.

* But don't call it a programming language, it's not!

Fortunately, we don't need to try to write valid HTML. We have tools to do it.

Most people can, with the aid of a validator (such as the excellent HTML Validator plugin for Firefox), produce a single valid HTML page. Here's one: 

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" 
  "http://www.w3.org/TR/html4/loose.dtd">

<html lang="en">

  <head>
    <title>My first web page</title>
  </head>

  <body>
    <h1>My first web page</h1>
    <p>Hello, World!</p>
  </body>

</html>

The problem comes when they try to update the text without changing the structure of the web page. So really I should say

Writing valid dynamic web pages is basically too hard for humans.

Specifically, HTML assumes that your webpage has lots of "tags" that look like <this>. And if one little < or > gets in among the normal text like this:

<p>Mathematically speaking, 2 + 2 > 3.</p>

then it gets all confused about what starts where, and you've got an invalid web page.

And the solution to this problem? Templating languages.

Templating languages

There are even more Python templating languages than there are web frameworks. Here are some worth looking at:

...and the django and web.py web frameworks have their own templating languages too. (By default, Jinja mimics Django's syntax).

They all have their strengths, but here are some examples of the first few...

Here's a way to use Tempita to print a valid version of the line above: 

from tempita import HTMLTemplate

t = HTMLTemplate('<p>{{text}}</p>')
print t.substitute(text='Mathematically speaking, 2 + 2 > 3.')

and here's the Jinja equivalent: 

from jinja import Environment

t = Environment(auto_escape=True).from_string('<p>{{text}}</p>')
print t.render(text='Mathematically speaking, 2 + 2 > 3')

and here's the Mako equivalent: 

from mako.template import Template

t = Template("<p>${text}</p>", default_filters=['unicode', 'h'])
print t.render(text='Mathematically speaking, 2 + 2 > 3.')

All basically the same, give or take. The Twiddler equivalent is a bit different: 

from twiddler import Twiddler

t = Twiddler('<p id="text">...</p>')
t['text'].replace('Mathematically speaking, 2 + 2 > 3.')
print t.render()

Twiddler tries to build its own version of the template webpage in memory, so it is probably much slower than the Tempita, Jinja and Mako versions. On the other hand, it doesn't introduce any new syntax, so it may be easier to use with other tools or when working with web designers who are only used to HTML.

In this regard, Twiddler is one of the "XML-aware" templating languages like Genshi and Kid, but a) it's more legible, since it doesn't try to mix programming language constructs with HTML/XML, and b) it can have this feature turned off when appropriate, to behave more like Jinja/Mako, so that it can be used for e.g. outputting "mail-merged" emails.

It's also worth noting that Genshi supports HTML sanitisation, so may be able to do quite well here too.

And finally

Having seen some of the nuts and bolts used to put together a web application, here is a brief step by step guide to using a web framework to put together a simple application. It compares a few popular web frameworks along the way highlighting common features and differences.