Unidata Science Gateway

Table of Contents

jupyterhub.png

Figure 1: Unidata Python Gallery GOES-16 true color notebook by Brian Blaylock available via the Unidata JupyterHub.

News

10/30/2020, Unidata Science Gateway Summer Internship Case Study Article

UCAR Mentorship and Remote Computing Resources Boost Remote Research for SOARS Student

8/17/2020, JupyterHub Resources Available for Fall 2020 Courses

JupyterHub Resources Available for Fall 2020 Courses

7/29/2020, DOI for Unidata Science Gateway

We have a newly minted DOI (doi:10.5065/688s-2w73) for the Unidata Science Gateway. See the new Acknowledging the Unidata Science Gateway section.

Introducing Students to Scientific Python for Atmospheric Science

Introduction

Welcome to the Unidata Science Gateway on the National Science Foundation Jetstream Cloud. Unidata is a diverse community of education and research institutions with the common goal of sharing geoscience data and the tools to access and visualize that data. For more than 30 years, Unidata has been providing data, software tools, and support to enhance Earth-system education and research. Funded primarily by the National Science Foundation (NSF), Unidata is one of the University Corporation for Atmospheric Research (UCAR)'s Community Programs (UCP).

As part of our current five-year plan (PDF) "Unidata 2024: Next-generation Data Services and Workflows to Advance Geoscience Research and Education", Unidata is exploring cloud computing technologies in the context of accessing, analyzing, and visualizing geoscience data. This site collects Unidata-related technologies hosted on Jetstream; you can make use of these resources either directly in your browser or using one of the client applications described below. Our hope is that by combining the resources available in this gateway – and perhaps coupling them with XSEDE HPC resources – community members can create end-to-end scientific computing workflows in the Jetstream Cloud.

JupyterHub

Unidata JupyterHub

JupyterHub is a technology that can serve programmatic, interactive notebooks to a class of students or for scientific researchers. The Unidata JupyterHub provides users access to the notebooks contained within the Unidata/python-training project and the computing environment required to run them. To gain access to this server, please send a message to support-gateway@unidata.ucar.edu. A free GitHub login is required for access. After logging in, please read the README_FIRST.ipynb notebook that is periodically updated with relevant information concerning this JupyterHub installation.

Also available are dedicated notebook servers for workshops or semester long classes. Please email us at support-gateway@unidata.ucar.edu for these types of requests.

THREDDS Data Server

TDS installation on Jetstream

The Unidata THREDDS Data server (TDS) is a web server that provides metadata and data access for scientific datasets, using a variety of remote data access protocols. A TDS is available on Jetstream at https://tds.scigw.unidata.ucar.edu/thredds/catalog.xml supplying a good portion of the data available on the IDD with a five day archive.

This TDS can be accessed from the IDV or from Python with the netCDF-Python or Siphon APIs or any THREDDS client (e.g., PyCSW).

THREDDS AWS Nexrad Server

THREDDS AWS Nexrad Installation on Jetstream

In collaboration with the NOAA Big Data Project and Amazon Web Services (AWS), Unidata has deployed a THREDDS AWS Nexrad Server on the Jetstream Cloud. More than twenty years of Nexrad Level II data are available from this server. The data reside on AWS, but are served out by a THREDDS Data server running on Jetstream.

AWIPS EDEX

EDEX installation on Jetstream: edex-cloud.unidata.ucar.edu

AWIPS is a meteorological display and analysis system used by the National Weather Service. EDEX is the server-side component of this system. Unidata maintains an EDEX data server on Jetstream to ingest and serve real-time AWIPS data for rendering by the CAVE client and the python-awips data access framework. For CAVE, simply enter edex-cloud.unidata.ucar.edu in the CAVE Connectivity Preferences dialog to connect.

RAMADDA Geoscience CMS

RAMADDA installation on Jetstream

RAMADDA is a geoscience content management system (CMS) originally developed at Unidata and now maintained by Geode Systems. The RAMADDA installation on Jetstream contains IDV bundles that retrieve data from Jetstream data servers.

LDM

LDM installation on Jetstream

The Unidata Local Data Manager (LDM) system is a peer-to-peer network for event-driven data distribution, and is the fundamental component of the Unidata Internet Data (IDD) system. Two internal LDM nodes supply data to our Jetstream resources.

ADDE

ADDE installation on Jetstream: adde://adde.scigw.unidata.ucar.edu:112

ADDE (Abstract Data Distribution Environment) is a data server developed by the University of Wisconsin, Space Science Engineering Center. It primarily serves satellite and radar imagery and point observations to the IDV, McIDAS-V, McIDAS-X. An ADDE server is available at adde.scigw.unidata.ucar.edu over (the usual) port 112.

IDV Resources

The Unidata Integrated Data Viewer (IDV) is a desktop application for viewing data in two and three dimensions. It excels at bringing together geoscience data from disparate sources into one integrated display.

Running the IDV on Jetstream

For those who have a Jetstream allocation (e.g., startup, education, research), an IDV image is available when selecting VMs through the Jetstream Atmosphere dashboard. If you do not have a Jetstream allocation, it is possible to obtain a Jetstream Trial Access account. When searching for images, search for "IDV". You will find the Unidata IDV <version> image. Launching that image will then allow you to "Open Web Desktop" where the IDV will be available pre-loaded with the Jetstream plugin. The catalogs available via the IDV Dashboard, Data Choosers will default to the TDS and RAMADDA servers running on Jetstream enabling data-proximate performance, especially for large datasets. You can also select image sizes that are appropriate to your goals. For example, if working with a data intensive bundle, you could choose an m1.medium VM with 6 vCPUS, 16 GB RAM.

IDV Jetstream Plugin

The IDV Jetstream plugin allows easy access to Jetstream installations of the TDS, RAMADDA and ADDE from the IDV. The plugin URL is https://science-gateway.unidata.ucar.edu/jetstream.jar. See here for how to install an IDV plugin from a URL.

Conference Presentations

Under the Hood

The implementation of this effort is described here: https://github.com/Unidata/science-gateway.

System Status

Internal only Unidata system monitoring link

IDD Relay RTSTATS

IDD Archiver RTSTATS

Contact

If you wish to contact Unidata concerning this science gateway, please contact support-gateway@unidata.ucar.edu.

Acknowledging the Unidata Science Gateway

If you have benefited from the Unidata Science Gateway, please use this DOI for citations in your publications: doi:10.5065/688s-2w73.

Acknowledgments and Bibliography

Acknowledgments

National Science Foundation

Funding for this project was generously provided by the National Science Foundation (Grant NSF-1901712)

XSEDE ECSS

We thank Brian Beck, Maytal Dahan, Jeremy Fischer, Victor Hazlewood, Peg Lindenlaub, Suresh Marru, Lance Moxley, Marlon Pierce, Semir Sarajlic, Craig Alan Stewart, George Wm Turner, Nancy Wilkins-Diehr, Nicole Wolter and Andrea Zonca for their assistance with this effort, which was made possible through the XSEDE Extended Collaborative Support Service (ECSS) program.

Bibliography

References

[1] S. Arms, J. Chastang, M. Grover, J. Thielen, and M. Wilson. Python workshop. In 100th AMS Annual Meeting, Boston, Massachusetts, USA, Jan. 12-16 2020. [ bib | .html ]
[2] S. Arms, J. Chastang, M. Grover, J. Thielen, M. Wilson, and D. Dirks. Introducing students to scientific python for atmospheric science. Bulletin of the American Meteorological Society, 2020. [ bib | DOI | http ]
[3] J. Chastang. Unidata ldm docker container. https://github.com/Unidata/ldm-docker/, 2016. [ bib | http ]
[4] J. Chastang. Unidata on the XSEDE Jetstream cloud. https://github.com/Unidata/science-gateway/, 2017. [ bib | http ]
[5] J. Chastang. Unidata science gateway jupyterhub. In ESIP Summer Meeting 2018, Tucson, AZ USA, 7 2018. Earth Science Information Partners. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[6] J. Chastang. Deploying a Unidata JupyterHub on the NSF Jetstream cloud, lessons learned and challenges going forward. In ESIP Summer Meeting 2019, Tacoma, Washington, USA, 7 2019. Earth Science Information Partners. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[7] J. Chastang. Jupyterhub for atmospheric science research and education on the NSF Jetstream cloud. In 99th AMS Annual Meeting, Phoenix, Arizona, USA, Jan. 6-10 2019. [ bib | DOI | http ]
[8] J. Chastang. A jupyterhub for atmospheric science research and education on the unidata science gateway. In Gateways 2019, San Diego, California, USA, Sep. 23-25 2019. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[9] J. Chastang, W. Fisher, and S. Arms. Unicloud: Docker use at unidata. In Proceedings, 32nd Conference on Environmental Information Processing Technologies, 96th AMS Annual Meeting, New Orleans, LA, 1 2016. American Meteorological Society. See also http://unidata.github.io/Unidata-Dockerfiles/index.html. [ bib | .html ]
Keywords: UPC
[10] J. Chastang and M. K. Ramamurthy. Unidata science gateway on the XSEDE Jetstream cloud. In Gateways 2017, Ann Arbor, Michigan, USA, Oct. 25-25 2017. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[11] J. Chastang and M. K. Ramamurthy. A cloud-based science gateway for the geoscience community. In OGC TC/PC Meetings, Fort Collins, Colorado, USA, Jun.  4-8 2018. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[12] J. Chastang and R. Signell. Met/ocean modeling workflows on XSEDE via HPC & cloud. In ESIP Summer Meeting 2017, Bloomington, IN USA, 7 2017. Earth Science Information Partners. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[13] J. Chastang, R. Signell, and J. L. Fischer. Reducing time to science: Unidata and JupyterHub technology using the Jetstream cloud. In 2017 AGU Fall Meeting, New Orleans, Louisiana, USA, Dec. 11-15 2017. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[14] J. Chastang, R. Signell, and J. L. Fischer. A Unidata JupyterHub server: An online PyAOS resource for students and educators. In 98th AMS Annual Meeting, Austin, Texas, USA, Jan. 22-26 2018. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[15] J. Chastang, T. Yoksas, and M. K. Ramamurthy. Geoscientific data distribution in the XSEDE Jetstream cloud. In Proceedings, 33nd Conference on Environmental Information Processing Technologies, 97th AMS Annual Meeting, Seattle, WA USA, 1 2017. American Meteorological Society. See also https://science-gateway.unidata.ucar.edu. [ bib | .html ]
[16] Julien Chastang. Demonstration of unidata science gateway, 4 2020. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[17] M. K. Ramamurthy. Unidata 2018: Transforming geoscience through innovative data services. Technical report, Unidata Program Center, UCAR Community Programs, University Corporation for Atmospheric Research, 3300 Mitchell Ln, Boulder, CO 80301 USA, 10 2013. [ bib | .pdf ]
[18] M. K. Ramamurthy. A cloud-based science gateway for the geoscience community. In ESIP Summer Meeting 2018, Tucson, AZ USA, 7 2018. Earth Science Information Partners. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[19] M. K. Ramamurthy and J. Chastang. A cloud-based science gateway for the geoscience community with end-to-end workflows on the Jetstream cloud system. In 2018 EGU Spring Meeting, Vienna, Austria, Apr.  8-13 2018. See also https://science-gateway.unidata.ucar.edu. [ bib ]
[20] M. K. Ramamurthy and J. Chastang. A cloud-based science gateway for the geoscience community with end-to-end workflows on the Jetstream cloud system. In IWSG 2018 - 10th International Workshop on Science Gateways, Edinburgh, Scotland, Jun.  13-15 2018. See also https://science-gateway.unidata.ucar.edu. [ bib ]
[21] M. K. Ramamurthy and J. Chastang. Unidata: A cloud-based science gateway for the atmospheric sciences. In Gateways 2018, Austin, Texas, USA, Sept.  25-27 2018. See also https://science-gateway.unidata.ucar.edu. [ bib ]
[22] M. K. Ramamurthy and J. Chastang. A cloud-based science gateway for enabling open and reproducible science. In 99th AMS Annual Meeting, Phoenix, Arizona, USA, Jan. 6-10 2019. [ bib | http ]
[23] M. K. Ramamurthy, J. Chastang, R. M. May, and M. James. Unidata and data-proximate analysis and visualization in the cloud. In ESIP Summer Meeting 2017, Bloomington, Indiana, USA, 7 2017. Earth Science Information Partners. See also https://science-gateway.unidata.ucar.edu. [ bib | DOI | http ]
[24] S. Sarajlic, J. Chastang, S. Marru, J. Fischer, and M. Lowe. Scaling jupyterhub using kubernetes on jetstream cloud: Platform as a service for research and educational initiatives in the atmospheric sciences. In Proceedings of the Practice and Experience on Advanced Research Computing, PEARC ’18, New York, NY, USA, 2018. Association for Computing Machinery. Recipient of Best Poster Award. [ bib | DOI | http ]
Keywords: OpenStack, JupyterHub, Kubernetes, Cloud Computing, Workforce Development, Unidata, Magnum
[25] Craig A. Stewart, Timothy M. Cockerill, Ian Foster, David Hancock, Nirav Merchant, Edwin Skidmore, Daniel Stanzione, James Taylor, Steven Tuecke, George Turner, Matthew Vaughn, and Niall I. Gaffney. Jetstream: A self-provisioned, scalable science and engineering cloud environment. In Proceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure, XSEDE '15, pages 29:1--29:8, New York, New York, USA, 2015. ACM. [ bib | DOI | http ]
Keywords: atmosphere, big data, cloud computing, long tail of science
[26] John Towns, Timothy Cockerill, Maytal Dahan, Ian Foster, Kelly Gaither, Andrew Grimshaw, Victor Hazlewood, Scott Lathrop, Dave Lifka, Gregory D. Peterson, Ralph Roskies, J. Ray Scott, and Nancy Wilkins-Diehr. XSEDE: Accelerating scientific discovery. Computing in Science & Engineering, 16(5):62--74, 2014. [ bib | DOI ]
[27] K. Wilcox and J. Chastang. Unidata thredds docker container. https://github.com/Unidata/thredds-docker/, 2015. [ bib | http ]
[28] Nancy Wilkins-Diehr, Sergiu Sanielevici, Jay Alameda, John Cazes, Lonnie Crosby, Marlon Pierce, and Ralph Roskies. An overview of the XSEDE extended collaborative support program, volume 595 of Communications in Computer and Information Science, pages 3--13. Springer Verlag, Germany, 2016. [ bib | DOI ]
The Extreme Science and Engineering Discovery Environment (XSEDE) is a flagship cyberinfrastructure project funded by the US National Science Foundation (NSF). XSEDE’s Extended Collaborative Support Services (ECSS) program is a significant component of the XSEDE effort, dedicated to extended engagements with our user community which transform their research. We describe the organization, operation and some highlights of the program in this submission.
[29] A. Zonca, R. P. Signell, J. Chastang, J. Fischer, J. M. Lowe, and R. S. Sinkovits. Deploy kubernetes and jupyterhub on xsede jetstream. In Gateways 2020 Proceedings, 2020. [ bib ]
Keywords: OpenStack, JupyterHub, Kubernetes, Cloud Computing, Unidata, Magnum

Date: 2020-10-30 10:37:04 MDT

Validate