The SkunkWeb package and the bootloading process

If you look in the lib directory of a SkunkWeb installation, you will see three subdirectories: SkunkWeb, Services, and pylibs. This arrangement is not quite parallel; SkunkWeb is a package with an __init__.py, and is imported as such, while the other two directories are not, and are themselves added to sys.path. The SkunkWeb package contains the core infrastructure of the SkunkWeb server: bootloading, configuration, and process management. The Services directory contains Python modules that are classified as services; they generally have some dependency on the SkunkWeb package or on some other service. The pylibs directory contains Python modules that SkunkWeb may need, but which could also be used outside of the SkunkWeb environment.

When the server is started, it reads a configuration file (by default, <skunkroot>/etc/sw.conf where <skunkroot> is the installation directory), itself containing Python code, which is then executed in the namespace of a special, somewhat magical object globally available as SkunkWeb.Configuration. (This object pretends to be a module, and is imported, but is actually an instance of the class ConfigLoader.ScopeableConfig). The Configuration object is thereby first populated with attributes; its quasi-magical properties will be discussed a bit later.

When the Configuration object is thusly initialized, it must have as an attribute named services a list of the names of some Python modules. These modules are then imported. They themselves may in turn import SkunkWeb.Configuration and set default values for its attributes, using Configuration's mergeDefaults method. Therefore, the configuration object is built up by first reading in user-defined values, and then setting defaults for any values which haven't yet been defined; almost all of SkunkWeb's configuration variables are defined in services. (A few are defined in the SkunkWeb package itself, and a few more are defined in the AE package in pylibs, in which the component system and cache are implemented.)

When this has occurred, the configuration phase is complete, and the server is ready to spawn its child processes and start serving. First, however, it executes a hook called ServerStart. Although you don't need to know about these hooks unless you are writing services, it is helpful to know a bit about them to understand SkunkWeb's request-handling cycle; if you aren't curious about that, skip the next section.

Hooks

SkunkWeb uses two kinds of hooks (defined in the pylib module hooks.py). The first, hooks.Hook, is a callable list which should contain callable things; each are called in turn until one of them returns something which is not None. It corresponds more or less to the kind of hooks used in Apache and elsewhere. The second, hooks.KeyedHook, is more exotic, although not complicated. When you want to execute a KeyedHook, you pass it a jobName parameter, upon the basis of which it generates a list of functions and then executes them as if they were an ordinary Hook. The function list produced depends on another parameter you specify when adding a function to the hook, the jobGlob. If a KeyedHook's function's jobGlob matches the jobName, with glob-like wildcards allowed, then that function will be selected for the hook, and will be executed if no other executed functions return something other than None first; otherwise, it will be skipped. Because of the glob-style matching, it is possible to add a hook function that will be executed for different jobs that share some particular aspect. (For instance, if you add a function with the jobGlob "*/web/*", then that function will have a chance to be executed for the jobNames "foo/web/gizmo" and "/web/templating", but not for "foo/templating/gizmo".) Through this mechanism, a kind of polymorphism is achieved within a hook architecture, and the same set of hooks can be used for many different jobs, without the different hook functions getting in each others' way.

ServerStart is a simple hook, and is populated with initialization functions by various services. After executing it, the server forks a number of child processes, each of which shares the same configuration, and each child executes another (simple) hook called ChildStart. Both these hooks are defined in the SkunkWeb package, but SkunkWeb's other hooks are defined in its services, principally in requestHandler and web.

Scopeable Configuration

Tantalizing mention has been made of the magical behavior of SkunkWeb.Configuration. This somewhat suspect global object is a chameleon or Zelig; the values of its attributes can be configured to change depending on aspects of the request that SkunkWeb is processing. At the end of the request, the Configuration object is reset to its original condition. The SkunkWeb jargon for this is "scoping". It is implemented through a massive __getattr__ hook, which is implemented in C to make it perform acceptably. When sw.conf is executed, there are several functions which can call "configuration directives" pre-loaded into the local namespace; with these functions one can state the conditions under which the configuration object will return a certain value for a given configuration variable. These most basic of these directives are:

• Scope: this is not a matching directive, but a container for the other directives, all of which must be contained within a Scope to have any effect.
• Port: matches the port on which SkunkWeb received the request, if it came over TCP (note that this is probably different than ServerPort, below).
• IP: tests for equality with the ip address of the interface on which the request came in; only relevant for TCP.
• UNIXPath: does a glob test on the file path of the UNIX socket, if the request came in through one.

The above apply to all jobs performed by requestHandler. After requestHandler knows this information, it "scopes" the Configuration according to it, and thereby finds out what sort of job needs to be performed. If the job is, as it probably is, a web job, there are several more directives that apply:

• Host: performs a glob match on the hostname.
• Location: tests whether the uri of the request begins with the given string.
• File: tests whether the uri of the request matches a regular expression.
• ServerPort: tests whether the SERVER_PORT cgi environmental variable (in other words, the web server's port) is equal to the given port.

There is one more useful directive which has nothing to do with scoping, the Include directive, which enables you to split up your configuration into multiple files.

It is easiest to explain how this works through an example. Imagine that sw.conf contains this line:

Include("/home/skunk/skunkroot/etc/scope.conf")

And that scope.conf contains the following:

# scope.conf
# there can any number of scope directives,
# or you can group the matchers together.
# By the way, the contents of SkunkWeb.constants
# are also preimported into the sw.conf namespace.

# Let requests on 9887 be handled as remote jobs.
Scope(Port(9887,
           job=REMOTE_JOB),
      # define a virtual host
      Host('*froggie.com',
           # you can nest matchers when it makes sense to do so; 
           # put them before the configuration variables you are
           # scoping
           ServerPort(8080,
                      Location("/admin/",
                               # turn on basic-auth password-protection
                               authAuthorizer="auth.BasicAuth",
                               authAuthorizerCtorArgs=('myrealm', '/home/skunk/skunkroot/var/authdb'),
                               authActivated=1),
                      documentRoot="/home/httpd/htdocs/froggie-admin",
                      compileCacheRoot="/home/skunk/skunkroot/cache-froggie-admin",
                      componentCacheRoot="/home/skunk/skunkroot/cache-froggie-admin"),
           documentRoot="/home/httpd/htdocs/froggie.com",
           # N.B.: whenever you scope the documentRoot, you must also
           # scope these cache roots; otherwise, the caches
           # will write over each other, which is not a good
           # thing.
           compileCacheRoot="/home/skunk/skunkroot/cache-froggie.com",
           componentCacheRoot="/home/skunk/skunkroot/cache-froggie.com"))

Assignments occurring outside of a scope block apply to everything, unless a scoped assignment takes precedence.

This is a flexible and powerful system, and one correspondingly vulnerable to abuse. For better or worse, SkunkWeb.Configuration is globally available; it would perhaps have been cleaner, but less convenient and often more verbose, for scoped configuration data to be accessed as an attribute of the CONNECTION. If you reference it directly from within a cached component, the values you extract from it will not be part of the cache key for that component; values therefore can leak from one scope to another if you use the cache thoughtlessly. As a general rule, avoid making the returned value or output of a component dependent on any values found in SkunkWeb.Configuration that might be scoped, if there is any chance that the component might be called from more than one scope; if you need those values, explicitly pass them into the components as arguments. In practice, this is seldom a problem. Also, not every configuration variable can be scoped; usually this is obvious (it doesn't make sense to scope the number of child processes, for example), but in some cases it is not. rewriteRules, for instance, which is used by the rewrite service, was scopeable in SkunkWeb 3.3, but in 3.4 it isn't, by default (because rewriting is now performed before HTTP-specific scoping takes place in the request-handling cycle).

Database Services

Since SkunkWeb uses multiple processes and not multiple threads, connection pooling per se would be difficult or impossible to implement, as it would require sharing open connections between processes. Connections can be cached, however, and the database services make that possible. In addition, they provide another useful feature: they rollback the connection after each request (optional for MySQL), so the connection will always be in a viable state when it is used. (The caches themselves are actually implemented in pylibs, because they are used also by the object-relational mapper PyDO, also authored by Drew Csillag, which is part of the SkunkWeb distribution; SkunkWeb database applications typically use it, as it is extremely convenient.)

These services are all very similar, so an example of how to configure one of them should suffice. Let us say you are using a PostgreSQL database and want to use the standard pgdb driver; your database is called soup, the user is called chef, the password is nincompoop, you want to give this connection the alias lentil, and you are accessing it over TCP on localhost at the default port for PostgreSQL. You would make two changes to sw.conf. First, add 'postgresql' to the list of services, and second, add this line:

# more than one connection alias can be defined in this dictionary
PostgreSQLConnectParams = {'lentil' : 'localhost:soup:chef:nincompoop'}

To get a cached connection from within SkunkWeb:

# dbaccess.pydcmp
# import the pylib which has the cache;
# unfortunately its name differs from the
# service only in capitalization
import PostgreSql
conn=PostgreSql.getConnection('lentil')
# go to town with the connection

(If you are using PyDO, you won't need to deal directly with the cache at all; you'll specify the connection alias when you define your PyDO objects, and the rest is taken care of for you.)

Connections are created lazily, as needed, and retained indefinitely. This means that over time, given enough requests, every process in your server will get its own connection to each database you define in this manner. This is perfectly acceptable in typical situations, even though it means that you may well not be fully utilizing your database connections, and that you can't reduce the number of those connections without also lowering the number of requests that can be processed at once, unfortunately coupling together two factors that are in principle unrelated. Luckily, this problem can usually be neglected, but if you really need to have fewer open database connections, you need a workaround. One possibility is not to cache connections at all, but to compensate for the expense of creating them once per query by wrapping the requests in a data component and caching the results, so that actual database requests are infrequent. Not all data is amenable to being cached, however. An alternative is to use an additional dedicated SkunkWeb installation with a smaller number of active processes to make the actual database calls, and to access that server using SkunkWeb's remote component protocol. That way, a small number of database connections can be fully utilized, servicing any number of other application servers that need access to that data. This scenario is only likely to come about, mind you, if you have a rather hefty, industrial-scale application on your hands; even if you don't, however, you might find the remote services useful for some other purpose.

The remote and remote_client Services

These services make it possible to call components on other SkunkWeb servers. If you want your server to service requests for remote components, add remote to the service list in sw.conf, and add something like the following lines, if they aren't already there:

RemoteListenPorts=['TCP:localhost:9887']
Scope(Port(9887, job=REMOTE_JOB))

This has the effect of causing SkunkWeb to listen on port 9887 and to treat requests arriving there as remote component requests. Of course, you don't need to use 9887, or even a TCP port, if for some peculiar reason you desire to have the "remote" server exist on the same box as the client and communicate over a UNIX socket.

To make a remote call on another service, add "remote_client" to the service list; there is no other configuration. Then to make a remote component call, you do something like this:

<:component "swrc://myremotebox.org:9887/comp/mycomponent.comp"
            x=`3` y=`5`:>

(Note that you need the quotation marks around the component url, or STML will be confused by the colon contained within it. Of course, you can make remote calls from Python as well.) The default port for remote calls is 9887, and can be omitted.

If an exception is thrown at the remote side of a remote call, the exception is marshalled and passed over to the client. The exception that is raised there, however, will be a dynamically created subclass of both the actual remote exception and of remote_client.RemoteException, which gives you some flexibility in how you catch them.

The sessionHandler Service

If you feel that sessions -- bins where you can toss objects that will persist across requests -- are essential or convenient for your application, this service is for you; but I for one would urge you to think over the choice carefully. Sessions have considerable overhead and bring a host of issues with them. Some implementations assume that one server will handle the entire session of a given client, which may be a restrictive assumption; and implementations that do not make that assumption are liable to be more expensive yet. In addition, the naive use of sessions may lead to errors in application logic, if the programmer forgets that the user may use the browser's back button and return to an earlier point in the flow of the program, while the session information has not been rolled back to the same point. The effort involved in compensating for these issues is often in excess of what it would take to design application-specific state management using a conventional database system, and that path, albeit a prosaic and at times dreary one, is recommended in many cases. Still, sessions are nifty at times.

These warnings aside, in order to use sessions in SkunkWeb, you first need to choose what sort of session storage you want to use. There are three storages available by default (all of them given regrettably jaw-breaking names by a programmer -- me -- who had recently emerged from the mind-deadening grip of Java):

• sessionHandler.FSSessionStore.FSSessionStoreImpl, which stores pickle files in a directory in the filesystem;
• sessionHandler.MySQLSessionStore.MySQLSessionStoreImpl, which uses MySQL as a backend; and
• sessionHandler.PostgreSQLSessionStore.PostgreSQLSessionStoreImpl, which uses PostgreSQL.

In sw.conf, you need to define SessionStore to be one of those monstrous names, as a string, and then define other configuration variables, depending on which store you have chosen (see sw.conf and the sessionHandler service itself for documentation). If you are using a database backend, you'll need to create the appropriate table as well. Then add sessionHandler to the list of service in sw.conf to enable it, and restart. When you do so, the CONNECTION object will have a new method, getSession, which returns a dictionary-like object in which you can store any pickleable thing. You use it as follows:

# top-level document: "session.py"
# no html for this example
CONNECTION.setContentType('text/plain')
sess=CONNECTION.getSession(create=0)
if sess:
    # session already exists
    oratory=sess.get('ubu')
    if not oratory:
        print ("hmmmm, who created this session and deplorably "\
               "neglected to initialize it as I expected?")
    else:
        print oratory
else:
    # no session, let's create one
    sess=CONNECTION.getSession(create=1)
    sess['ubu']=("My fellow countrymen, get thee to "\
                 "Katz's Delicatessen and partake thee "\
                 "of the veal before I give you a smack "\
                 "on the behind.")
    print "session initialized with bizarre content."

In most cases, you don't need to worry about saving the session explicitly; it is saved for you after the response is sent. However, if you are making changes to a session and then redirecting back to the same server or another on which the same session is to be accessed, you should save the session first, because another process might restore the session before the first process is done saving it:

sess=CONNECTION.getSession(1)
sess['edible plants']='asparagus, broccoli, endive'
sess.save()
CONNECTION.redirect('http://myserver.com/plants.html')

Sessions time out after a period of inactivity (thirty minutes, by default, controlled by Configuration.SessionTimeout), and sessionHandler periodically checks for stale sessions and clears them out of the session store. Sessions store their ids in cookies, and while it would be possible to embed them in urls as well, this has not been implemented, due to lack of passion and a general conviction that the future lies with cookies.

The auth Service

Unsurprisingly, the auth service provides authorization support, for password-protecting portions of a site. It can also be used to permit guest logins and attach user objects, perhaps with session-like features, to the CONNECTION. Except for the simplest uses, you are very likely to need to extend it with Python code, if you want to authorize against a database, for example. Here is an example which does just that, and furthermore permits guest logins:

# module ubuauth.py

import auth as A
import SkunkWeb.Configuration as C
# this is an imaginary library; assume
# that calls to it return the right things
import ubu_library as U

# this uses cookie-based authorization, with
# a login page.    
class UbuAuth(A.CookieAuthBase, A.RespAuthBase):
    def __init__(self, loginPage):
        A.RespAuthBase.__init__(self, loginPage)
        A.CookieAuthBase.__init__(self,
                                  # cookie name
                                  'ubu_auth',    
                                  # cookie nonce, for 'armoring' the cookie
                                  'Alfred Jarry', 
                                  # path for the cookie
                                  {'path' : '/'}) 

    def validate(self, username, password):
        # get the user with this username, 
        # if it exists.  In this example,
        # the (nonexistent) Users class is
        # a PyDO object.
        user=U.Users.getUnique(username=username)
        # return the actual user object as a true value --
        # will be returned by A.CookieAuthBase.checkCredentials,
        # in our checkCredentials, below, and stored in the
        # CONNECTION object
        if user and user['password']==password:
            return user

    def checkCredentials(self, conn):
        user=A.CookieAuthBase.checkCredentials(self, conn)
        if user:
            # make this object available from CONNECTION
            conn.ubuUser=user
            return 1
        conn.ubuUser=None
        return 0
        
    def authFailed(self, conn):
        conn.remoteUser='guest'
        conn.remotePassword=None
        # let it be configurable (and scopeable)
        # whether guest login is allowed;
        # presumably, you've written a service which
        # defines the default value for this
        if C.UbuRequireValidUser:
            A.RespAuthBase.authFailed(self, conn)

    def login(self, conn, username, password):
        authorized=A.CookieAuthBase.login(self, 
                                          conn, 
                                          username, 
                                          password)
        conn.ubuUser=authorized
        return authorized

And here is the configuration which would enable it:

# auth.conf
# it is presumed that "auth" has been added to the 
# service list, and that this file has been included
# with an Include directive in sw.conf

# allow guest logins by default
UbuRequireValidUser=0

Scope(Location("/",
               authAuthorizer="ubuauth.UbuAuth",
               authAuthorizerCtorArgs=('/login.html',),
               authActivated=1),
      # in this area, disallow guest logins
      Location("/poland/",
               UbuRequireValidUser=1))

Inside components, you would be able to access the user object and customize the pages in accordance with what you find.

For simpler applications, you may not need to write your own authorizer. The auth service provides, in addition to useful mixin classes like the base classes in the above example, fully usable classes for basic authorization, cookie authorization, and session-based authorization against an htpasswd-style password file, and the script <skunkroot>/bin/swpasswd can be used to manage such files.

The rewrite Service

Url rewriting has always been important to SkunkWeb; in the shop where it was developed, development work was done with one language and the website it served was published in two different languages, and special SkunkWeb services were used to translate urls from one language to another. Writing a service to do specific url rewriting tasks is not too hard; for an example, look at the userdir service. But for almost all conceivable purposes, the rewrite service, contributed by Spruce Weber, makes that effort unnecessary. rewrite is SkunkWeb's answer to Apache's mod_rewrite, except that while mod_rewrite is universally acknowledged to be obscure voodoo, and has been called "Engellschall's Bane", SkunkWeb's rewrite is very easy to use and understand, and furthermore, is extremely powerful. You can do much more with it than rewrite urls; you can use it to return responses to the client, and you can modify the connection and hence request in any way you wish. (Note that some of what follows pertains only to SkunkWeb 3.4; earlier versions don't have rewrite conditions, and are slightly different in other respects as well, but rewrite rules from earlier versions are upwards compatible.)

In order to use the service, define Configuration.rewriteRules to be a list of rewrite rules, which come in two forms:

(<regular expression>, <replacement>)

or:

(<rewrite condition>, [<more rules>])

In the first form, the regular expression is used to search CONNECTION.uri. If it matches, what happens next depends on what kind of thing the replacement is. If it has a method named rewrite, that method is called, which can perform any action: rewriting the url, altering the connection or the session dictionary (which you can pretend you never heard about, as the best thing to do is to leave it alone), raising a PreemptiveResponse and thus handling the request, etc. (A number of classes are provided that do commonly needed things along these lines, like performing a redirect, or returning a 404; and you can write your own.) Else, if the replacement is callable or a string, regular expression substitution is performed on the original uri, and the result is assigned to CONNECTION.uri. An additional twist: if the regular expression match contains a groupdict and Configuration.rewriteMatchToArgs is true, then CONNECTION.args is updated with the groupdict. This is an easy way to use path info rather than querystrings to pass request data to scripts.

In the second form, the first element must be callable with the arguments (CONNECTION, sessionDict) (and again, you can ignore the session dictionary). If and only if it returns a true value, the rules in the second element are executed. A class is provided, rewrite.RewriteCond, that performs most of the tests you'll want to perform -- exactly the same tests as you would perform while scoping.

By default, all of your rewrite rules will be applied, one by one. But if you want it to stop after the first match, set Configuration.rewriteApplyAll to a false value.

Here are a few rewrite rules from a real application, with names changed to protect the uninteresting:

from rewrite import Redirect, Status404, RewriteCond

rewriteRules=[
    # for hosts wampum.org, snodgrass.com, and dialectic.net, on port 80:
    (RewriteCond(host=r'wampum|snodgrass|dialectic',
                 port=80), 
     [# put into CONNECTION.args['section'] either
      # artichoke, gnus, or fungus; the rest of the url
      # before the .html extension goes into CONNECTION.args['body']
      (r'^/legacy/(?P<section>artichoke|gnus|fungus)/(?P<body>.*)\.html$', r'/__legacy.py'),
      # similar, but without 'section'
      (r'^/legacy/(?P<body>.*)\.html$', r'/__legacy.py'),
      # a straightforward rewrite: map anything remaining starting with "/legacy/" to "/__legacy/"
      (r'^/legacy/(.*)', r'/__legacy/\1')]),
    (RewriteCond(host=r'nuisance.org', port=80), 
     [
     # put the number at the end of the url into CONNECTION.args['item_id']
     (r'^/(gnus|fungus)/articles/(?P<item_id>\d+)', r'/\1/index.html'),
     # if the item_id is missing from the above, raise a 404
     (r'^/(gnus|fungus)/articles/.*', Status404()),
     # fudge pages. 
     (r'^/torus/(?P<nick>[^/]+)/fudges/(?P<cook_date>\d{8}|current)', r'/torus/fudge.html'),
     # if cook_date is not present, redirect to archive page
     (r'^/torus/([^/]+)/fudge/', Redirect(r'/torus/\1/archive.html')),
     ])]

The product Service

Since SkunkWeb applications consist of a mixture of SkunkWeb top-level documents and components and Python modules, including services, it is desirable to be able to distribute and deploy them all together, which is what the product service is about. A SkunkWeb product is an archive (which can either be a zipfile, a tarfile, or a gzipped tarfile) or a regular directory, in either case containing two subdirectories, one for documents (by default, <product>/docroot) and one for Python modules (by default, <product>/libs). When you are using the product service, all you have to do to load the product is drop this archive into your products directory (Configuration.productDirectory, by default, <skunkroot>/products). The product docroot directory will be automatically mounted underneath your documentRoot in a rather magical way, so that it stays in the same place relative to the documentRoot even if the documentRoot is scoped; the product libs directory will be added to sys.path (by means of an import hook, in the case where the product is an archive).

A product must also contain a MANIFEST file in its top directory, which is a simple properties file like this one:

version = '0.01'
dependencies = ''
services = ''
author = 'Jacob Smullyan'
# default docroot
# default libs
# this SkunkWeb product manifest generated on Mon Jan 27 14:25:42 2003

This file can be used to specify that docroot and libs are to be found in other than the default location; it can also assert dependencies on other SkunkWeb products, and the product will fail to load if the other products are not installed. Furthermore, the MANIFEST can declare that other modules are services, and they will be imported at server startup along with the services listed in Configuration.services.

In order for the product service to work, you may need to make one configuration change to your SkunkWeb installation. SkunkWeb uses a virtual file system api, implemented in the vfs package, to access resources in the document root. The configuration variable documentRootFS holds the reference to the vfs filesystem it employs. As of the latest beta release at the time of writing, 3.4b1, the default filesystem is an instance of vfs.LocalFS, which is just a wrapper around Python's os module. In order for products to be automatically mounted relative to the document root, you'll need to add this to sw.conf:

import vfs
documentRootFS=vfs.MultiFS()
documentRootFS.mount(vfs.LocalFS(), '/')

(A MultiFS, in case you are curious, is a vfs filesystem in which other filesystems can be mounted at arbitrary points. In this case, the product docroots are mounted into the documentRootFS with floating mount points that depend on Configuration.documentRoot, so different virtual hosts can make use of the same products.)

At the moment, there is only one publically available SkunkWeb product (Bloskunk, a web log application), so you are more likely to be a producer than a consumer of them for the time being. In order to create a product from an existing application, you can use the graciously graphical SkunkWeb Product Wizard, <skunkroot>/bin/productwiz (requires Tkinter). It asks where it can find files for the libs and docroot of your application, what the services, dependencies, and metadata are, writes the MANIFEST and packages the product in the format of your choice.

Generally speaking, SkunkWeb products are discouraged from containing top-level documents. The idea behind a product is that a given server instance may contain multiple instances of applications based on a given product, even in different virtual hosts, without having to install its components more than once. As a result, the default uri path for products is the unglamorous /products/<productname>; although this can be changed for all products or on a per-product basis via the configuration variables defaultProductPath and productPaths, respectively (the latter being a mapping of product names to paths), if you stick with the guideline that the docroot of the product only contains includes and components, you don't need to make the product path presentable.