Handlers

Handlers are special Python modules that convert between a given data format and the data model used by Pydap (defined in the pydap.model module). They are necessary in order to Pydap be able to actually serve a dataset. There are handlers for NetCDF, HDF 4 & 5, Matlab, relational databases, Grib 1 & 2, CSV, Seabird CTD files, and a few more.

Installing data handlers

NetCDF

NetCDF is a format commonly used in oceanography, meteorology and climate science to store data in a machine-independent format. You can install the NetCDF handler using EasyInstall:

$ easy_install pydap.handlers.netcdf

This will take care of the necessary dependencies. You don’t even need to have to NetCDF libraries installed, since the handler will automatically install a pure Python NetCDF library called pupynere.

The NetCDF handler uses a buffered reader that access the data in contiguous blocks from disk, avoiding reading everything into memory at once. You can configure the size of the buffer by specifying a key in the server.ini file:

[app:main]
use = egg:pydap#server
root = %(here)s/data
templates = %(here)s/templates
x-wsgiorg.throw_errors = 0
pydap.handlers.netcdf.buf_size = 10000

In this example, the handler will read 10 thousand values at a time, converting the data and sending to the client before reading more blocks.

NCA

The pydap.handlers.nca is a simple handler for NetCDF aggregation (hence the name). The configuration is extremely simple. As an example, to aggregate model output in different files (say, output1.nc, output2.nc, etc.) along a new axis “ensemble” just create an INI file with the extension .nca:

; output.nca
[dataset]
match = /path/to/output*.nc
axis = ensemble
; below optional metadata:
history = Test for NetCDF aggregator

[ensemble]
values = 1, 2, ...
long_name = Ensemble members

This will assign the values 1, 2, and so on to each ensemble member. The new, aggregated dataset, will be accessed at the location of the INI file:

http://server.example.com/output.nca

Another example: suppose we have monthly data in files data01.nc, data02.nc, ..., data12.nc, and we want to aggregate along the time axis:

[dataset]
match = /path/to/data*.nc
axis = TIME  # existing axis

The handler only works with NetCDF files for now, but in the future it should be changed to work with any other Pydap-supported data format. As all handlers, it can be installed using EasyInstall:

$ easy_install pydap.handlers.nca

CDMS

This is a handler that uses the cdms2.open function from CDAT/CdatLite to read files in any of the self-describing formats netCDF, HDF, GrADS/GRIB (GRIB with a GrADS control file), or PCMDI DRS. It can be installed using EasyInstall:

$ easy_install pydap.handlers.cdms

The handler will automatically install CdatLite, which requires the NetCDF libraries to be installed on the system.

SQL

The SQL handler reads data from a relation database, as the name suggests. It works by reading an INI file with the extension .sql. Below is an example that reads data from a SQLite database:

# please read http://groups.google.com/group/pydap/browse_thread/thread/c7f5c569d661f7f9 before
# setting your password on the DSN
[database]
dsn = "sqlite:///home/roberto/tmp/pydap/data/simple.db"
table = "test"

[dataset]
name = "test_dataset"
owner = "Roberto De Almeida"
contact = "roberto@dealmeida.net"
version = 1.0

    [[NC_GLOBAL]]
    history = "Created by the Pydap SQL handler"

[_id]
col = "id"
long_name = "sequence id"
missing_value = -9999

[lon]
col = "lon"
type = "Float32"
units = "degree_north"
long_name = "longitude"
missing_value = -9999
axis = "X"
valid_range = [-180.0, 180.0]

[lat]
col = "lat"
type = "Float32"
units = "degree_east"
long_name = "latitude"
missing_value = -9999
axis = "Y"

The handler works with SQLite, MySQL, PostgreSQL, Oracle, MSSQL and ODBC databases. To install the handler use easy_install; you should also install the dependencies according to the database used:

$ easy_install pydap.handlers.sql
$ easy_install "pydap.handlers.sql[oracle]"
$ easy_install "pydap.handlers.sql[postgresql]"
$ easy_install "pydap.handlers.sql[mysql]"
$ easy_install "pydap.handlers.sql[mssql]"

Proxy

This is a simple handler intended to serve remote datasets locally. For example, suppose you want to serve this dataset on your Pydap server. The URL of the dataset is:

http://test.opendap.org:8080/dods/dts/D1

So we create an INI file called, say, D1.url:

[dataset]
url = http://test.opendap.org:8080/dods/dts/D1
pass = dds, das, dods

The file specifies that requests for the DDS, DAS and DODS responses should be passed directly to the server (so that the data is downloaded directly from the remote server). Other requests, like for the HTML form or a WMS image are built by Pydap; in this case Pydap acts as an Opendap client, connecting to the remote server and downloading data to fulfill the request.