Data Management
For over 20 years PNNL has been a leader in data management. By managing large volumes of data, our data integration team designs excellent data products, tools, and dashboards to manage data flow.
For example, led by PNNL, the U.S. Department of Energy (DOE) Office of Science's Atmospheric Radiation Measurement (ARM) Climate Research Facility collects raw data from over 350 unique instruments and 1,500 datastreams and processes it to meet the user's specific science needs. ARM sends 18 TB of data per month to the ARM Archive—that's over 1,000 iPhones worth of storage combined. ARM manages tasks across several institutions, national laboratories, and major universities
For the DOE's Wind Energy Technologies Office, PNNL manages the Atmosphere to Electrons Initiative's Data Archive and Portal—an initiative designed to optimize the performance of wind power plants and lower the levelized cost of energy. The Data Archive and Portal collects, stores, catalogs, preserves, and disseminates results from experimental and computational efforts.
Our data integration team manages large volumes of data in traditional high-performance mass storage systems and highly scalable cloud storage. We streamline data retention policies based on project needs, and we design flexible, secure, and easy-to-manage custom data storage for open, embargoed, or proprietary datasets.
We design flexible security systems to accommodate open, embargoed, or proprietary datasets. System security enables data access by roles, projects, and policies; this also applies to portal security and secure data transmissions. Security management includes other resources, such as compute services and storage systems.
We integrate front-end web interfaces with content management systems for greater control over project management content. These interfaces combine content management systems, scientific online collaboration environments, data and metadata services, and facilities for formal project governance. The web front end also can be designed for multi-project-distributed organizations and integrated with back-end data services. Web portals have been developed to manage data flow processes supporting real-time data with a transfer rate of 100 Mb/second—well over the amount of daily cell phone data used by the average American.
The data is available to the community via the portal for instant validation, allowing submitters to reduce their storage requirements and enabling the necessary operational tools to display data quality information. This is done using carefully planned workflows and data management tools that ensure data integrity afford efficient monitoring and ultimately increasing ease of use.
The ARM Data Integrator is a suite of tools, C libraries, structures, and interfaces developed to simplify the development of algorithms to analyze time-series data and decrease the costs associated with such development. Photo Credit: ARM
Enlarge Image
Our software developers build ingest processes parsing and structuring raw files into standard formats. These processes ensure minimum quality checks, apply calibrations, and convert data into engineering units. We have experience working with instrument mentors and domain scientists to create derived data products and obtain measurements that otherwise cannot be easily observed.