|
The Dapper package includes a database loader (dapperload.sh)
that updates the server with metadata from netCDF or Opendap
profile files.
Dapper
currently supports the following profile conventions:
The loader is only used to aggregate in-situ data.
Individual netCDF files of any convention can be served through
by using the Dapper netCDF service.
Dapper assumes that each station in the
database is a profile -- that is, each station is
at a fixed (x,y,t) and measurements at the stations consist
of measurements at multiple depth or heights. If each station
is a time series (fixed (x,y,z) and varying t) then
the dataset name must contain the string "_time_".
For example, a dataset named "foo" will
be considered to be a profile datasets, but a dataset named "foo_time_series"
will be a time series.
There are three level of hierarchy for aggregated data.
A dataset contains
an aggregated group of profile files. A database (the
top level) contains a group of datasets.
The database loader allows you to create or remove a database
and dataset, load or unload a set of profile files from a
dataset, and
list the contents of a database or dataset.
Note: the database schema for the releases before
0.9.0 is not compatible with later releases. You
need to drop the database from MySQL and recreate and
reload the database if you loaded data with version 0.5.x.
You can get information on command options to the loader
by executing dapperload.sh without any arguments:
dapperload.sh
Dataset creation/removal
The create command creates a new database
or adds a dataset to an existing database. For example:
dapperload.sh -u <user> -p <pass> create newdatabase newdataset
creates the new dataset newdataset in the
database newdatabase. If newdatabase doesn't
exist it will be created.
The command:
dapperload.sh -u <user> -p <pass> remove newdatabase newdataset
removes newdataset from the database newdatabase.
Adding profiles to a dataset
The load command adds files to a dataset.
The files can be in a JAR or ZIP archive. For example:
dapperload.sh -u <user> -p <pass> \
load newdatabase newdataset /data/file1.nc /data/file2.nc
adds the files /data/file1.nc and /data/file2.nc to the
dataset.
dapperload.sh -u <user> -p <pass> \
load newdatabase newdataset /data/archive.zip
adds all of the files in the archive /data/archive.zip to
Dapper.
Although Dapper can read data from compressed zip or jar
archives, the server will be much slower if compression is
used.
If the files are already in the dataset, metadata
from the file will be updated only if the file has been
modified since the last
time the files were loaded (unless the -f argument
is specified).
You can specify a file containing the files to add by with
the -l argument
:
dapperload.sh -u <user> -p <pass> \
-l files load newdatabase newdataset
All files in a dataset should follow the same netCDF conventions.
Results are unpredictable if the profile files follow different
conventions.
Unloading profiles from a dataset
The unload command removes files from a
dataset. For example:
dapperload.sh -u <user> -p <pass> \
remove newdatabase newdataset /data/file1.nc /data/file2.nc
removes the files /data/file1.nc and /data/file2.nc from
the dataset.
Listing the contents of a database or dataset
The ls command list the files in a dataset
or the datasets in a database. The command:
dapperload.sh -u <user> -p <pass> ls newdatabase
lists all of the datasets in the database newdatabase.
The command:
dapperload.sh -u <user> -p <pass> ls newdatabase newdataset
lists all of the files in the dataset newdataset.
Command reference
Usage: dapperload [OPTIONS] create database dataset_name
dapperload [OPTIONS] remove database dataset_name
dapperload [OPTIONS] load database dataset_name [file [file...]]
dapperload [OPTIONS] unload database dataset_name [file [file...]]
dapperload [OPTIONS] ls database [dataset_name]
Commands:
create create a new database and dataset
remove remove a dataset from a database
load load a set of files into a dapper
database and dataset
unload unload a set of files from a dapper
database and dataset
ls list datasets from a dapper
database or files in a dataset
Options for all commands:
-u [username] database username
-p [password] database password
-h [host] hostname of database server
-P [port] port number of database server
-v print incomprehensible debug information
to log file
Options for load command:
-f force updates of all files
Without this flag the database is only updated with files
that have been modified since the last database update
-l file containing list of files to add
Options for unload command:
-l file containing list of files to unload
|