pydap — Server Documentation

Server

Pydap comes with a DAP server based on a modular infrastructure designed to allow easy support for new data formats. The server is based on the Python Web Standard Gateway Interface specification, making it even more flexible.

Deploying a server

The easiest way to deploy a server is running the command:

$ paster create -t dap_server myserver project=My-Server

This will create the following directory structure:

myserver
myserver/data
myserver/myserver.egg-info
myserver/myserver.egg-info/not-zip-safe
myserver/myserver.egg-info/paster_plugins.txt
myserver/server.ini
myserver/template
myserver/template/index.tmpl

Here, myserver/data is a directory from where your files will be served (assuming you have the appropriate plugins), and myserver/template/index.tmpl is a Cheetah template for the directory listing. More interesting, perhaps, is the Paste Deploy configuration file myserver/server.ini, with this content:

[server:main]
use = egg:Paste#http
# Change to 0.0.0.0 to make public
host = 127.0.0.1
port = 8080

[filter-app:main]
use = egg:Paste#httpexceptions
next = cascade

[composit:cascade]
use = egg:Paste#cascade
app1 = static
app2 = pydap
catch = 404

[app:static]
use = egg:Paste#static
document_root = %(here)s/data

[app:pydap]
use = egg:dap
name = My-Server
root = %(here)s/data
verbose = 0
template = %(here)s/template

Now let me explain that configuration file, that describes how the DAP server will be deployed. The first block defines the WSGI server we're using:

[server:main]
use = egg:Paste#http
# Change to 0.0.0.0 to make public
host = 127.0.0.1
port = 8080

The first line in this block, use = egg:Paste#http, tells that we're using a server from in the Paste module, defined by the http entry point. This is a standalone server, useful for testing and debugging our DAP server. In this case, it's configured to listen on the 127.0.0.1 interface at port 8080, so it will only handle local requests. Don't worry if this isn't very clear now, I'll show you how to change this later.

If we wanted to run a plain pydap server, we could replace the remaining blocks with this one:

[app:main]
use = egg:dap
root = %(here)s/data

Here we have our main application, defined in the dap module. It requires a root parameter, pointing to the root directory from where data will be served. Note that here we can use paths relative to the configuration file, using the %(here)s variable.

The two entries, [server:main] and [app:main], are all we need to define our deployment and run a pydap server. The default configuration for pydap is a little more elaborated, though. The first thing you'll probably notice is that instead of an [app:main] entry we have something a little different:

[filter-app:main]
use = egg:Paste#httpexceptions
next = cascade

This blocks defines a WSGI middleware that acts like a filter between the server and another application (or middleware). All it does is capture raised exceptions and format them as nicely formatted HTTP messages. It also chains the cascade application to the server pipeline.

cascade is also a special kind of application: a composite WSGI application. Here it is:

[composit:cascade]
use = egg:Paste#cascade
app1 = static
app2 = pydap
catch = 404

As you can see, it defines two other applications: static and pydap. This applications works the following way: it tries to pass the request to the first application; if a 404 status message is found (the catch parameter), it tries with the next app, and so on. The first app it tries to call, static, is simply a server for static content:

[app:static]
use = egg:Paste#static
document_root = %(here)s/data

Here it is serving files from the same root directory as pydap. With this configuration, it is possible to download the served files. For example, the file at http://test.pydap.org/test.csv can be downloaded from the server. When a DAP client tries to retrieve the DDS at http://test.pydap.org/test.csv.dds, on the other hand, static returns the status 404 Not Found and the request is passed to pydap.

Finally, here's our pydap application:

[app:pydap]
use = egg:dap
name = My-Server
root = %(here)s/data
verbose = 0
template = %(here)s/template

Here we have some additional parameters: a name for our server, a directory for the template file and a verbose option to output debug information to the console.

Standalone server

If all you want is to run a simple, standalone server running on http://localhost:8080, just type:

$ paster serve myserver/server.ini

This is the default behavior of server.ini. It's useful for testing and debugging, but not intended for production use.

Running with CGI

Pydap can easily be run as a CGI script. All you have to do is to change the [server:main] section in the server.ini file to:

[server:main]
use = egg:PasteScript#cgi

Don't forget to remove or comment the host and port definitions. Now create a script with the command:

#!/bin/bash
/path/to/paster serve -q /path/to/myserver/server.ini

Make it executable and put in your cgi-bin directory.

Running with SCGI

The SCGI protocol is a replacement for CGI; it's similar to FastCGI but designed to be easier to implement. This is how the pydap test server runs, together with the Cherokee web server.

To run pydap using SCGI, first we need to install a module called flup:

$ easy_install flup

Now we can use either the threaded or forked version by changing our server.ini file to:

[server:main]
use = egg:PasteScript#flup_scgi_thread
#use = egg:PasteScript#flup_scgi_fork
host = 127.0.0.1
port = 8006

Create an executable script called, say run.sh, with the content:

#!/bin/bash
/path/to/paster serve /path/to/myserver/server.ini

This will start our SCGI server. We still need to connect our web server (Apache, lighttpd or Cherokee) to the SCGI server to make it work.

Configuring Cherokee

This is the relevant configuration used for the pydap test server:

Server test.pydap.org {
    DocumentRoot /var/www/test.pydap.org
    Directory / {
        Handler scgi {
            Server localhost:8006 {
                Interpreter "/var/www/test.pydap.org/run.sh"
            }
        }
    }
}

A nice thing of this setup is that Cherokee will bring the SCGI server up if it's not running.

Configuring lighttpd

Here's the relevant configuration for lighttpd:

server.modules = (..., "mod_scgi", ...)
server.document-root = "/path/to/myserver/"
scgi.server = ("/" => (( 
    "host" => "127.0.0.1",
    "port" => 8006,
)) )

Now simply restart lighttpd and run the script run.sh.

Configuring Apache

If you want to use SCGI together with Apache, first you need to install mod_scgi. You should then enable the module in Apache's configuration file:

LoadModule scgi_module /path/to/mod_scgi.so

To connect to the SCGI server, use the SCGIMount directive:

SCGIMount / 127.0.0.1:8006

Now simply restart Apache and run the script run.sh.

FastCGI

FastCGI is another alternative for CGI. To run pydap using FastCGI, we first need to install the flup module:

$ easy_install flup

As SCGI, flup offers both a threaded and a forking implementation of FastCGI. We define it on our server.ini file:

[server:main]
use = egg:PasteScript#flup_fcgi_thread
#use = egg:PasteScript#flup_fcgi_fork
host = 127.0.0.1
port = 8006

Finally, we meed to create an executable script called, say run.sh, with the content:

#!/bin/bash
/path/to/paster serve /path/to/myserver/server.ini

This will start our FastCGI server. We still need to connect our web server (Apache, lighttpd or Cherokee) to the FastCGI server to make it work.

Configuring Cherokee

This is the relevant part of the configuration for running pydap as FastCGI with Cherokee:

Server test.pydap.org {
    DocumentRoot /var/www/test.pydap.org
    Directory / {
        Handler fcgi {
            Server localhost:8006 {
                Interpreter "/var/www/test.pydap.org/run.sh"
            }
        }
    }
}

Again, with this setup Cherokee will bring the FastCGI server up if it's not running.

Configuring lighttpd

Here's the relevant configuration for lighttpd:

server.modules = (..., "mod_fastcgi", ...)
server.document-root = "/path/to/myserver/"
fastcgi.server = ("/" => (( 
    "host" => "127.0.0.1",
    "port" => 8006,
)) )

Simply restart lighttpd and run the script run.sh to make it work.

Configuring Apache

The first step to run a FastCGI pydap server with Apache is installing mod_fastcgi. Then insert the following directive in your Apache configuration:

SetHandler fastcgi-script
FastCgiExternalServer / -host 127.0.0.1:8006

Now simply restart Apache and run the script run.sh.

IIS

It is possible to run a WSGI application like the pydap server as a ISAPI extension under IIS, using this ISAPI WSGI implementation. I never tried this, but feel free to write me if you need any kind of help.