Science Gateways
About Science Gateways
NERSC assists science teams in building web interfaces to access HPC computers and storage systems. These gateways allow scientists to access data, perform computations and interact with NERSC resources using web-based interfaces to run simulation and analyze data. The goal is to make it easier for scientists to use computing resources at NERSC while creating collaborative tools for sharing data with the rest of the scientific community.
NERSC engages with science teams interested in using these new services, assists with deployment, accepts feedback, and tries to recycle sucessful approaches into methods that other teams can use. Below is a list of current projects and methods. Building blocks are available to NERSC users interested in creating new science gateways. If you would like to participate, or if you have questions, please contact consult@nersc.gov.
A list of science gateways at NERSC can be found in the sidebar to the right.
Gateway Demos
Fully developed examples of science gateways can be found in the sidebar. In order to quickly demonstrate some of the basic capabilities the examples below show some gateway functions labeled by their command line equivalents. If you login to the NERSC web site you can run the quickstart examples in your own account, otherwise they will run in a demo space. These examples don't do a lot but are meant to quickly show some of what is possible through hands on experience.
Gateway Technologies
NERSC provides science teams with the building blocks to create their own science gateways and web interfaces into NERSC. Many of these interfaces are built on web and database technologies.
Web Methods
Science gateways can be configured to provide public unauthenticated access to data sets and services as well as authenticated access if needed. The following features are available to projects that wish to enable gateway access to their data through the web. Other features can be made available on request.
REST based web API (NEWT) to access the NERSC center, including authentication, file management, job submission and accounting interfaces.
- Full-featured backend programming environments in the language of your choice (PHP, Python, Ruby, Java).
- Support for LDAP and Shibboleth authentication.
- Connecting to PostGRESQL/MySQL/NoSQL Databases.
- Modern Web 2.0 interfaces with AJAX front-ends such as Google maps and visualization kits.
- OpenDAP access to large data sets (netcdf and HDF5)
- CVS/SVN/Git code hosting and online browsing.
- Access to NERSC filesystems and HPSS through the NEWT API, grid tools or other custom interfaces
- Direct web access to data in the /project filesystem.
Database Methods
Science gateways can be hosted on or draw data from NERSC's science database nodes. These are specially configured nodes which support MySQL, Postgres, and MongoDB for high performance access. More details on the science gateway database services is provided here. Some exmples of database methods used by gateways are
- Access filecatalogs and other persistently stored collections from your batch jobs
- Conenct a web based gateway to datasets stored in a database (read and read-write)
- Store, search and analyze data objects, e.g. job output, through mapreduce like MongoDB methods
- Expose public read-only data collections through database protocols
Science Gateways in Production
DeepSky - DeepSky Image Database
The Materials Project - Explore Properties of Computed Materials.
The Gauge Connection - QCD Lattice Gauge Data
CXIDB - X-Ray Image Data Bank
20th Century Reanalysis - Reanalysis of 20th Century Climate Data
Dayabay - Dayabay Neutrino Detector Gateway
ESG - Earth System Grid Climate Gateway and Data-node
NEWT - NERSC Web Toolkit for Science Gateway Development
NOVA - NERSC Online VASP Application
Getting Started
A project directory is a good place to host a science gateway. You can publish data through a publicly accessible URL by simply making an appropriate subdirectory like the following:
How to publish your data on NGF to the web:
ssh portal-auth.nersc.gov In the above example, you can replace portal-auth with any other NERSC compute platform that has access to /project. Create a www directory in your project directory:
mkdir /project/projectdirs/yourproject/www
Make sure your project directory and the www directory are world readable:
chmod 755 /project/projectdirs/yourproject/ chmod 755 /project/projectdirs/yourproject/www
Copy your data to this www directory. Any public data will need to be world readable. Add PHP and HTML files to this directory to build custom gateway interfaces to the data Any data under /project/projectdirs/yourproject/www will be publicly accessible through http://portal.nersc.gov/project/yourproject/ To build more sophisticated web apps, include the following in your HTML files to give you access to all NERSC compute and data resources through the NEWT API
<script src="http://newt.nersc.gov/js/jquery-1.4.2.js" />
<script src="http://newt.nersc.gov/js/newt.js" />
How to publish data in HPSS to the web:
Similar to the example above, if you make a "www" directory with appropriate permissions in your HPSS home or project directory, then the data in that directory will be available at a URL of the form http://portal.nersc.gov/archive/home/projects/NAME/www/{FILE|DIR} where NAME is the project directory and FILE|DIR is the name of a file, in which case the file is downloaded, or a directory in which case a directory listing is made. Here is an example: http://portal.nersc.gov/archive/home/projects/incite11/www/1935 Please note that the time to download files from tape may take some time to start as the tape robot finds and mounts the correct tape.
How to move beyond simple gateway functions:
If you are building a web gateway to your science at NERSC, please contact with us at consult@nersc.gov. We are interested in engagaing directly with science teams so that you can build a gateway that meets your specific needs.
