Introduction:
This page will help get started with the basics and provide what you need to know to run NEMS regression tests.
Before continuing, some information:
- Links to the right provide navigation to specific sections.
- This page is for internal NCEP users, i.e., those who can access the R&D (Zeus) or CCS machines (Cirrus/Stratus), and have a Subversion account.
- If you are working on Jet or Bluefire, you can find more information here.
- This page assumes you are new to using NEMS and running NEMS regression tests.
- If at any time you are confused and cannot find the information that you need please, email for help.
|
|
Purpose of Regression Tests:
Regression testing attempts to uncover errors by partially retesting a modified program, and its intent is to provide a general assurance that no additional errors were introduced into the model.
What do we test?
- Bit-identical result reproducibility
- Different domain decomposition
- Threading
- Different physics options
- Timing (only NMM-B, but will eventually also be done for GFS)
- Different compilations
- And many more...
Currently, we do not test forecast verification (scores and skills).
The NEMS regression test suite is separated into two sets, FULL and STANDARD. The STANDARD set should be run before any repository commit. It takes about 1.5 hours to finish and consists of 8 NMM-B tests, 7 GFS tests, and 1 GEN test. Those tests are highlighted in red in the regression test details section below. The FULL set is run from a cron once a week on Saturday to make sure that no code was broken from any recent commits. It takes about 7 hours to finish and has 51 tests and 6 different compilations (ESMF3, ESMF5, NMM-B only, NMM-B with TRAPS ON, and GFS only).
Return to top
NEMS Regression Test Details:
Currently, there are three models that can be run under the NEMS framework: the Global Forecast System (GFS), Global Ensemble Forecast System (GEFS), and the B-grid version of the Nonhydrostatic Multiscale Model (NMM-B), none of which are the operational versions.
There are several compilations in the FULL set of tests, including the following:
- Compiled with ESMF3, all cores and GOCART.
- Compiled with ESMF5, all cores.
- Compiled with ESMF3, only NMM-B.
- Compiled with ESMF3, only NMM-B with traps on.
- Compiled with ESMF3, only GFS.
Original NEMS regression test data is located in directories similarly to the following schematic:
The method to run the regression tests for these models is the same, but each model has its own set of regression tests. Following is a basic schematic to visualize the process.
Below, you will find more specific information about the various regression tests that can be run in NEMS, all of which are run under the first compilation mentioned in the above list (ESMF3, all cores, GOCART). The tests highlighted in red are run in the STANDARD set. Based on trunk version 22934.
NMM-B:
Unless otherwise stated, all NMM-B regression tests use 6x5 compute tasks, 1 thread, and opnl (operational) physics.
1 - Baseline global to compare with previous trunk version: free fcst, pure binary input.
2 - Baseline global with NEMSIO input: free fcst, NEMSIO input.
3 - Global restart run: restart, pure binary input.
4 - Global restart run with NEMSIO input: restart, NEMSIO input.
5 - Global with different domain decomposition: 3x5 compute tasks with single thread, opnl physics, free fcst, pure binary input.
6 - Global with multiple threading: 6x5 compute tasks with 2 threads, opnl physics, free fcst, pure binary input.
7 - Global with GFS physics: GFS physics, free fcst, pure binary input.
8 - Baseline regional to compare with previous trunk version: free fcst, pure binary input.
9 - Baseline regional with NEMSIO input: free fcst, NEMSIO input.
10 - Regional restart run: restart, pure binary input.
11 - Regional restart run with NEMSIO input: restart, NEMSIO input.
12 - Regional with different domain decomposition: 3x5 compute tasks with single thread, opnl physics, free fcst, pure binary input.
13 - Regional with multiple threading: 6x5 compute tasks with 2 threads, opnl physics, free fcst, pure binary input.
14 - Regional with GFS physics: GFS physics, free fcst, pure binary input.
15 - Regional with selected GFS physics schemes: selected GFS physics, free fcst, pure binary input.
16 - Regional with precipitation adjustment on: free fcst, pure binary input.
17 - Regional writing time series: free fcst, pure binary input.
18 - Regional with static nesting: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, pure binary input, compute tasks:- upper parent: 2x3
- child 1: 4x8
- child 2: 2x4
- grandchild: 7x10
19 - Regional restart with static nesting: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, pure binary input, compute tasks:- upper parent: 2x3
- child 1: 4x8
- child 2: 2x4
- grandchild: 7x10
20 - Using RRTM radiation (static nests) : regional parent with two children and one grandchild, single thread, test physics, free fcst, pure binary input, compute tasks:
- upper parent: 2x3
- child 1: 4x8
- child 2: 2x4
- grandchild: 7x10
21 - Using RRTM radiation (restart static nests): regional parent with two children and one grandchild, single thread, test physics, free fcst, pure binary input, compute tasks:- upper parent: 2x3
- child 1: 4x8
- child 2: 2x4
- grandchild: 7x10
22 - Regional static nests with digital filter: regional parent with one child and one grandchild, single thread, opnl physics, free fcst, NEMSIO binary input, compute tasks:- upper parent: 2x2
- child 1: 3x5
- grandchild: 6x7
23 - Regional moving nests: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, NEMSIO and pure binary input, compute tasks:- upper parent: 8x8
- child 1: 2x6
- child 2: 2x6
- grandchild: 5x6
24 - Regional moving nests with generational task assignments: regional parent with two children and one grandchild, single thread, opnl physics, free fcst, NEMSIO and pure binary input, compute tasks:- upper parent: 7x16
- child 1: 4x6
- child 2: 4x6
- grandchild: 5x10
GFS:
GFS in NEMS is not the same as the operational GFS model. There are big differences between the two, although their results are almost the same due to using the same science component. All of the GFS tests run on T62, 64 levels on global domain, 4 soil layers, diabatic run on reduced grid. Restart files are created every 24 hours. All tests exclude tropospheric/volcanic aerosol effects except for test #13.
1 - Baseline to compare with previous trunk version: Runs on 32 pe, 30 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 6 hours.
2- Compare adiabatic results with previous trunk version: Runs on 32 pe, 30 forecast tasks, with single thread, quilting, no digital filter.
3 - With 2-copy option: Runs on 32 pe, 30 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics have (allocated) their own copy of model variables, restarts from 24 hours, integrates to 48 hours, output at every 6 hours.
4 - With different decomposition: Runs on 60 pe, 58 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 6 hours.
5 - With multiple threads: Runs on 16 pe, 12 forecast tasks and 2 write group/2 processors per write group, with 2 threads, quilting, no digital filter, dynamics and physics share one copy of model variables, restarts from 24 hours, integrates to 48 hours, output at every 6 hours.
6 - 32 processors, 1 thread, with frequent output: Runs on 32 pe, with single thread, output every 4 timesteps.
7 - With single processor: Runs on 1 pe with single thread.
8 - With single processor and no quilting: Runs on 1 pe with single thread, no quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 6 hours.
9 - With multiple threads & output at every 2 time steps: Runs on 16 pe with 16 tasks (16 forecast, 1 write), 2 threads, no quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 2 time steps.
10 - With 48 processors: With single thread, no quilting, no digital filter, dynamics and physics share one copy of model variables, restarts from 24 hours, integrates to 48 hours, output every 2 time steps.
11 - With 32 processors and digital filter: Runs on 32 pe with 32 tasks (30 forecast, 2 write), with single thread, quilting, digital filter on reduced grid, dynamics and physics share one copy of model variables, integrates 24 hours, output every 6 hours.
12 - With 16 processors and digital filter: Runs on 16 pe with 16 tasks (12 forecast, 2x2 write), 2 threads, quilting, digital filter on reduced grid, dynamics and physics share one copy of model variables, restarts from 24 hours, integrates to 48 hours, output every 6 hours.
13 - With 1 processor and digital filter: Runs on 1 pe with single thread, no quilting, digital filter on reduced grid, dynamics and physics share one copy of model variables, integrates 24 hours, output every 6 hours.
14 - With OPAC climo scheme: Identical to the baseline (test #1) with two exceptions:- integrates 48 hours instead of 24 hours
- includes tropospheric and volcanic aerosol effects from the OPAC climo scheme.
15 - With NDSL (Non-iteration Dimensional-split Semi-Lagrangian): Runs on 16 pe with 16 compute tasks (12 forecast, 2x2 write), 2 threads, quiliting, digital filter on reduced grid.
16 - With digital filter, Hyb 2 loop: Runs on 16 pe with 16 compute tasks (12 forecast, 2x2 write), 2 threads, quilting, HYB 2 loop digital filter on reduced grid.
17 - With digital filter restart, Hyb 2 loop: Runs on 16 pe with 16 compute tasks (12 forecast, 2x2 write), 2 threads, quilting, HYB 2 loop digital filter on reduced grid.
18 - With digital filter and nst, Hyb 2 loop: Runs on 16 pe with 16 compute tasks (12 forecast, 2x2 write), 2 threads, quilting, HYB 2 loop digital filter on reduced grid.
19 - With digital filter, Hyb 1 loop: Runs on 16 pe with 16 compute tasks (12 forecast, 2x2 write), 2 threads, quilting, HYB digital filter on reduced grid.
GEFS:
Concurrency GEFS: T190 resolution, 28 vertical levels, run on 4 ensemble members (one member for the control with no initial perturbation added, 3 ensemble run members with initial perturbations), using 64 CPU tasks on 1 node, runs 24 hours, and at every 6 hours couples all ensemble members together, adds the stochastic perturbations to represent the model uncertainties, the re-sets up the initial conditions for the next 6 hours.
Generic
1 - GEN: 16 pe, 1 node, 1 member
FIM
Compare FIM results with previous trunk version, only FIM compiled: 40 compute tasks, 1 thread, 1 node.
Tests for post:
1 - NMM-B baseline global to compare with previous trunk version: free fcst, pure binary input.
2 - Baseline GFS to compare with previous trunk version: Runs on 32 pe, 30 forecast tasks and 2 write tasks, with single thread, quilting, no digital filter, dynamics and physics share one copy of model variables, integrates 24 hours, output at every 6 hours.
3 - GFS with GOCART: Runs with 64 compute tasks, reproduces the Q4Y12 NGAC production except for G2 products.
Following are the lists of tests included in the other compilations under the FULL set of tests:
Compiled with ESMF 5.2.0rp1, all cores
- Compare NMMB-global results with previous trunk version ESMF5
- Compare GFS results with previous trunk version ESMF5
- Concurrency GEFS, stochastic perturbations, 4 members, T190L28. ESMF5
Compiled with ESMF3, only NMM-B
- Compare NMMB-global results with previous trunk version, only NMM compiled
Compiled with ESMF3, only NMM-B with traps on
- Compare NMMB-global results with previous trunk version, TRAPS on
Compiled with ESMF3, only GFS
- Compare GFS results with previous trunk version, only GFS compiled
Return to top
Directories and Regression Test Scripts:
For Zeus directories:
* - 1 or 2.
** - global, meso, nems, etc.
Recommended Directories In Which To Store NEMS:
'Userid' is given to you when you obtain a Cirrus/Stratus account.
/gpfs/t3/global/save/userid (Cirrus/Stratus)
/scratch*/portfolios/NCEPDEV/**/save/Firstname.Lastname (Zeus)
Location Of Regression Test Run Scripts:
/gpfs/t3/global/save/userid/trunk/job/regression_tests (Cirrus/Stratus)
/scratch*/portfolios/NCEPDEV/**/save/Firstname.Lastname/trunk/job/regression_tests (Zeus)
Files in Regression Test Directory:
.../job/regression_tests Sampling of scripts used in the NEMS regression tests
| Chem_Registry.rc | Used to control basic properties of the GOCART and StratChem Grid Components; specifically, it- selects which constituents to simulate.
- selects the number of bins for each constituent.
- specifies variable names and units for each constituent.
|
| Compile.log | Results of the compilation. |
| MAPL.rc | Minimum MAPL resource file. |
| RT.sh | Script to run regression tests; performs the following:- compiles NMM-B core.
- runs NMM-B regression tests.
- cleans and compiles GFS core.
- runs GFS tests.
|
| RegressionTests.log | Results of the regression tests. |
| rt_nmm.sh | Makes configure and run files for NMM-B. |
| nmm_ll | Loadleveler script to run NMM-B regression tests. |
| rt_gfs.sh | For stand-alone GFS tests, makes configure and run files for GFS. |
| gfs_fcst_run | Loadleveler script to run GFS and GEFS regression tests. |
|
Return to top
Running the Standard Set of Regression Tests:
The following provides information about how to run the NEMS regression tests.
This is relatively simple to do, although it will take a few hours to complete. The 'STANDARD' set is most likely what you would run, and is required before a commit to the NEMS trunk. The 'FULL' set is on a cron, set to run every Saturday, and contains the full list of regression tests and various compilations.
| Action | Active Directory | Command |
| Run 'STANDARD' set of regression tests | /gpfs/t3/global/save/userid/trunk/job/regression_tests (Cirrus/Stratus)
/scratch*/portfolios/NCEPDEV/**/save/Firstname.Lastname/trunk/job/regression_tests (Zeus)
* - 1 or 2
** - global, meso, nems, etc. | 'RT.sh' |
| Create baseline(s) | Same as above | Use one argument (i.e. nmm, gfs, gen, or all)
e.g. 'RT.sh nmm' |
| Run 'FULL' set of regression tests | Same as above | 'RT.sh full test' |
Details on whether or not each test finished successfully will be found in RegressionTest.log.
Return to top
Understanding Output:
Output from the NEMS regression tests is written to your /ptmp directory (i.e. /gpfs/s4/scrub/ptmp/userid, /gpfs/c4/scrub/ptmp/userid, or /scratch*/portfolios/NCEPDEV/**/ptmp/Firstname.Lastname (* - 1 or 2, ** - global, meso, nems, etc)). NOTE: To view output from regression tests and not have it erased upon completion, make sure the following line is commented out, 'rm -rf ${RUNDIR_ROOT}'.
NMM-B output files are written in a gridded format readable by GrADS (.ctl). GFS files are written in Grib format and can easily be processed into the same format as the NMM-B files. See the above tab, 'GFS Post' for more information regarding how to process GFS output.
For a description of the various output variables, please reference 'Output'.
Return to top
Notes:
CLEAN UP:
Please be aware that files in the /ptmp directory are scrubbed after they are 10 days old. If you want to keep specific files, they should be moved to another directory that is not scrubbed on a regular basis (i.e. /global/save, /meso/save, /u/userid, etc.)
PATH:
Some scripts assume that "." is included in the users PATH. Check for
this if unexpected errors occur. (The error msg isn't always clear).
Return to top
|
|
www.nws.noaa.gov
