RPC Introduction

Models

• Geomagnetic Activity Index

• STORM Time Empirical Ionospheric Correction Model

• Wang Sheeley solar wind speed, IMF model

• Relativistic Electron Forecast Model

RPC Links

• Overview
  
  
• Process
  
  
• New RPC Submissions

Questions and comments concerning these pages or the Rapid Prototyping Center can be addressed to the

Go to SEC index page

RPC Process

The three basic aspects, or stages, of the RPC are illustrated in Figure 1: Stage 1, the initial evaluation to determine the suitability of a model or data stream to enter the RPC; Stage 2, the evaluation and modification in an operational environment; and Stage 3, the final graduation or transition to operational use. The requirements to progress through each of these stages are outlined below. Although each product transition will be unique, the anticipated maximum time within the RPC is six months. An evaluation of each product under development will be performed monthly to assess the likelihood that the product will become operational and to identify the main issues to be resolved.

Beginning at Stage 1, the external models and new data streams are evaluated by the SEC staff, with representation from the Systems, Operations, and Research Divisions. Upon acceptance into the RPC at Stage 2, operational evaluation occurs with specific requirements at each step of the process. During this stage test products are generated and made available on the Web and in the forecast center. In this way, feedback is obtained from the SEC forecasters and from outside users. Prior to graduation in Stage 3, the documentation and user manuals are completed and the operational products are finalized.

Stage 1:
Procedures for Models Under Initial Evaluation

The basic acceptance requirements are the following:

1. Model/data must be relevant to the operational mission of NOAA Space Weather Operations. Modelers/data providers must demonstrate that the proposed product will provide one or both of the following:
   • Information currently unavailable that is likely to improve services
   • Substantive improvement on information currently being used operationally as judged, for example, by skill-score comparison
     
     
2. Models must be compatible with the hardware, software, and data resources available at SEC.
   • SEC will provide the following information to help a proposer infer the appropriateness of his/her model or data stream for RPC matriculation:
     • Hardware, software, and visualization resources available through SEC
     • Data and model resources available for model use
   • Modelers will provide the following information:
     • Data and/or other input needed to run the model
     • Required CPU usage, RAM usage, and disk storage
     • Visualization tools used
     • Required network connectivity
     • A straw-man Concept of Operations to include information such as frequency of anticipated operational use (hourly, daily, weekly, etc.)
     • Additional special considerations
     • Verification testing that has been performed and the results
     
     
3. Model must run reliably using test data provided by SEC. SEC will provide archived, time-series data to allow the modeler to develop and test the interface to the real-time data. The modeler will then provide SEC with the model output, including a description of any incompatibilities between the model and the data. The data provided by SEC could be available on the Web, and may include simulated data with a minimum amount of gaps, erroneous values, etc.

4. Model/data products must be valid. The model/data must operate without unanticipated interruptions, and "bad" values must be flagged. Where possible, model output or data will be compared by the researcher and by the RPC staff with other measurements or model output to indicate that the results are valid. This will be a consideration during the pre-entrance evaluation and throughout the transition process.

5. Model/data must be documented.
   • Model input requirements: data type, time resolution, software and hardware requirements
   • Model output format or data format
   • Description of output/data: units, resolution, known limitations, all the raw variables that could be used
   • Range of validity: What does it do well, where does it break down? Is there a range where you do not expect it to be valid, where hasn't it been tested?
   • It is desirable that the model be documented internally (not just the part with which SEC will interface).
   • Larger models should be modular to support model modification as well as to allow output from intermediate stages to be extracted for operational products.

6. A brief development plan must be provided. SEC will contribute to this plan to best direct the model development toward operational use. The modeler will provide a plan for the model development within the RPC to meet the final operational requirements. This will include a description of the model/data comparisons that will be made, the possible products that will be available to the forecasters, and the suggested steps that will be taken to improve the reliability and the accuracy while in the RPC. This plan should also include an estimate of the staff support required to implement the model and the time required to adequately train the staff in the maintenance of the model and to properly interpret the model results.

Stage 2:
Procedures for Models and Data That Have Entered the RPC

Models/data submitted to RPC proceed through a sequence of decision points along the path to operational acceptance. The four main decision points are outlined below. However, some of these points may not apply in all cases. For example, models dealing with predictions of sunspot maximum cannot be expected to undergo verification in real time, since the time scales involved are so long that only testing against historical data is practical. Data streams taken into the RPC may be evaluated using similar data sets if such exist or using historical and real-time data to assess the consistency and accuracy.

Each entrant into the RPC will follow a customized plan through the four basic decision points. That is, a statement of criteria will be developed for each step. Success in graduating from the RPC consists of passing through all pre-defined decision points. Failure may occur at any step, resulting in either recycling to a previous step or elimination from further consideration. At each step, an estimate will be made of the resources necessary to continue. If, for example, a major modification becomes necessary during one of the steps that would require substantial future costs, it may be decided to reject the model altogether at that point, rather than to reinsert it at the beginning of the current step. Some flexibility in adjusting the decision criteria may be appropriate during the execution of the adopted development plan, but final acceptance of any such proposed changes will rest with the RPC staff.

The following main decision points are identified within the RPC:

1. Bring model up on local or remote RPC system:
   • Self-contained models to run on SEC workstations and peripherals alone:
     • Port code to local SEC systems (model/data provider responsibility with SEC support).
     • Preliminary benchmarking, assessment of resources consumed
     • Bring up visualization in at least a diagnostic mode.
   • Models run remotely that need an interface and post-processing at SEC:
     • Set up links with remote system to SEC system and resolve interface issues including data ingest, simulation output, run-time diagnostics.
     • Bring up post-processing tools and visualization software on SEC system.
     • Arrange for long-term access.

2. Verification with historical data (no external dissemination of experimental products):
   • Begin with best cases, closest to model concept; proceed on to more general difficult cases.
   • Collect statistics on performance.
   • Establish measures of accuracy.
   • Establish epochs or broad classes of applications (e.g., stratify performance according to phase of solar cycle).
   • Compile statistics on resources consumed.

3. Verification with real-time data (internal use of experimental products, but also begin external dissemination, e.g. through a New Products Web site, to show progress and to encourage outside comment):
   • Establish links to real-time data for model input and develop strategy for filling in missing data, correcting errors, etc.
   • Develop and implement preliminary operational visualization tools.
   • Issue preliminary assessment of "value" in terms of internal consistency/accuracy, operator assessments of day-to-day utility, likelihood that model/data will see further application/development.

4. Bringing system fully online:
   • Finalize Concept of Operations for full implementation.
   • Produce documentation, define training for operational use.
   • Finalize full set of production visualization tools and diagnostics.
   • Document verification, statistical performance, etc, within scope of predetermined criteria.
   • Finalize outside links (WWW).
   • Document record of operator questions, difficulties, etc.
   • Produce statement of scope, known limitations (e.g. verification truth table).
   • Produce clear statement of ownership of final RPC product, what can/cannot be freely accessed and modified, and what part remains the property of the applicant.

Stage 3:
Disposition of Models and Data After Transition to Operational Use

SEC will have full rights to run and distribute output from all models that graduate from the RPC. In particular:

• Access to the operational version of model code will be through SEC staff only.
• Routine access to the model output will be through the standard SEC products.
• All upgrades to the operational version of the model will be implemented through the RPC.
• The modeler will have no further obligations to SEC.

In addition, model verification such as skill-score will be made publicly available, as with any operational model at SEC.