EMC > NEMS > RUNNING NEMS

NOAA Environmental Modeling System

Running NEMS

This page is meant to guide you in setting up and running experiments for the different models within NEMS. If you are interested in starting out with regression tests, please refer to 'Getting Started' . Bear in mind that, currently, there is no simplified way of running the NMM-B or GFS individually. It may require changes to the regression test script (RT.sh), the ability to recognize other files involved, creation of your own run scripts, and the creation of your own compatible input files.

NAVIGATE THIS PAGE Libraries & Dependencies Check Out and Build How To Run NMM-B (CCS) How to Run NMM-B Nests (CCS) How To Run GFS (CCS) How To Run FIM Porting Information

Libraries & Dependencies

Currently, NEMS requires the following external libraries to be installed on the system in order to run properly.

ESMF versions 3.1.0rp2 and 5.2.0rp1 are supported. Available from the ESMF web site.
netcdf - available from the UNIDATA website.
The following libraries are from NCEP and can be downloaded from the NEMS trunk (located in EMC's Subversion repository) in /trunk/util/syslib.tar:

bacio
w3
sp
nemsio

The following will be used only when write_dopost is set to .true. in the configure file:

g2 - /nwprod/lib
g2tmpl - /nwprod/util/sorc/g2tmpl.fd
sigio, sfcio - /nwprod/lib
unipost - /climate/save/wx20wa/esmf/nems/unipost_r14561
png, jasper - /usrx/local/64bit/lib; jasper is the library to make jpeg2000 packing

The last two libraries listed above may already be installed on your system (Linux), but may not be by default. PNG, in turn, requires zlib, but that is almost always installed, at least on Linux. They are required for compilation of the g2 (GRIB2) library.

Check Out and Build NEMS:

If you are interested in just running regression tests, please refer to 'Regression Tests'. Otherwise, continue on below.

Process to checkout and build NEMS:

Action	Active Directory	Command
Check out NEMS	/gpfs/t3/global/save/userid (Cirrus/Stratus) /scratch/portfolios/NCEPDEV//save/Firstname.Lastname* (Zeus) * - 1 or 2 ** - global, meso, etc.	svn checkout https://webarchive.library.unt.edu/web/20130304013300/https://svnemc.ncep.noaa.gov/projects/nems/trunk
Set special compile options in the appropriate configure file (default settings are used for regression tests)	.../trunk/src/conf
Set the version of the ESMF library that will be used (if running RT.sh, this will be done automatically): ESMF_Version_Number = 3, use ESMF 3.1.0rp2 library ESMF_Version_Number = 5.2, use ESMF 5.2.0rp1 library	.../trunk/src	esmf_version ESMF_Version_Number
Compile and build the core(s) needed (/job/regression_tests/RT.sh will build the default settings - NMM and GFS with GOCART)	.../trunk/src	See below...

Commands used to compile various NEMS cores (outside of /job/regression_tests/RT.sh):

gmake nmm - for NMM core only
gmake gfs - for GFS core only (without GOCART)
gmake gfs GOCART_MODE=full - for GFS core only (with GOCART)
gmake nmm_gfs - for both NMM and GFS cores (without GOCART)
gmake nmm_gfs GOCART_MODE=full - for both NMM and GFS cores (with GOCART)
gmake fim - for FIM core only
gmake fim_gfs - for both FIM and GFS cores only (without GOCART)
gmake fim_gfs GOCART_MODE=full - for both FIM and GFS cores only (with GOCART)
gmake gen - for GEN core only
gmake gen_nmm_gfs GOCART_MODE=full - for GEN, NMM and GFS cores (with GOCART)
gmake nmm_post - for NMM core, run with post on quilt
gmake gfs_post - for GFS core, run with post on quilt
gmake nmm_gfs_gen_post GOCART_MODE=full - for GEN, NMM and GFS cores with post (with GOCART)

The executable, NEMS.x, will appear in /exe.

How To Run NMM-B (on CCS)

There are several methods by which one can run the NMM-B. Click on a [+/-] for more information.

Modifying RT.sh, using regression test answers, and creating your own run script to run only the experiment(s) needed. [+/-]
Modifying RT.sh, creating your own input files, and creating your own run script to run only the experiment(s) needed. If you want to use your own input data you will have to also compile and run NPS [+/-].
Using the NEMS Launcher. [+/-]

In addition to the steps above, there is an evolving draft of a how-to for NEMS NMM-B here.

Methods #1 & #2: Using regression test answers/your own input

Modifying RT.sh is the recommended method by which to run the NEMS NMM-B.

Basic Instructions for Methods #1 & #2:

For this process we will use the first NMM-B regression test (baseline global NMM-B to compare to previous trunk version). This may simplify the process if you choose to run other regression tests individually or create your own run scripts.
As mentioned earlier, if you want to use your own input files, you will need to compile and run NPS before following the above steps or running your own scripts.

Check out the top of the NEMS trunk via Subversion (https://webarchive.library.unt.edu/web/20130304013300/https://svnemc.ncep.noaa.gov/projects/nems/trunk) for the latest version of the NEMS GFS. You can compile using the built-in compile options in /jobs/regression_tests/RT.sh or by using the instructions provided here or found in the README file included in the code.
FYI: For an NMM-B regression test, RT.sh will call rt_nmm.sh.

This script will create a new script called nmm_ll (which you submit to LoadLeveler) and a configure_file to go along with the settings you want or that are included in each individual regression test.

Copy RT.sh to a new file name (e.g. RT_new.sh, mycopy.sh) and use the new file for the rest of this process.
Keep the 'Clean and compile...' section around line 346 of the original RT.sh. All other 'Clean and compile...' sections can be commented out/removed.
It is okay to comment out/remove the following sections before the actual regression tests:

'Check if running regression test or creating baselines.' - For this case, no baselines are being created.
'If two arguments are provided...' - This section is utilized for those wanting to run the full regression test suite.

You may also want to comment out the following:

'rm' statements in the 'finalize' section
'gmake clean' and 'esmf version 3' lines at the end of the script
'if' statement in in the test section for CB_arg and the corresponding 'fi'
portions of the 'clean and compile' section if you have already cleaned and compiled the code

Make sure you keep the 'exit' at the end of the file.
Run RT.sh or whatever your new filename may be.

If you want output closest to operational output, several changes to the configure file (nmm_nests_conf.IN) need to be made.

Configure File Settings for 'Operational' Output:

NMM-B output files are written in a gridded format readable by GrADS (.ctl). To understand NMM-B output, please refer to the 'Output' tab. [close]

How To Run NMM-B Nests(on CCS)

The NMM-B has telescoping static and moving nest capability. All domains, whether the uppermost parent or any nest, are functionally equivalent and thus each needs its own configure file. Currently all nesting is 1-way which means there is no internal exchange of data between a parent and its children, i.e., the only inter-generational exchange of data is the parents' sending new boundary data to each child at the end of every parent timestep.

The following instructions are also available in the README.NMM file.

1-way nesting: [+/-]

Set 'nest_mode' to '1-way' in all configure files. The value of 'generation' is not relevant and can be ignored.
The uppermost parent's configure file:

The variable 'num_domains_total' must be set in this domain's configure file. This is the total number of domains in the run which includes the upper parent plus all nests. This variable does not need to be set in any other configure files (if it is set in others it is not read).
Set the value for 'my_domain_id' which must always be 1 for the uppermost parent.
Set the value for 'my_parent_id' to -999.
Set 'n_children' to the number of child nests associated with the uppermost parent. This does not include any nests inside the first generation of child nests because all interactions with nesting involve only a parent and its first generation of children.
Set 'my_domain_moves' to false.
If one-way nesting is to be used then specify '1-way' for the variable called nest_mode.

Static nest configure files:

In each nest's configure file set 'my_domain_id' to a unique integer greater than 1. The user is free to choose these integer identifiers in any way desired except that all domain IDs must ultimately form a monotonic sequence. In other words if the run contains 2 first generation nests and one of those nests contains a nest then the three nests may use any integer value between 2 and 4 as their domain ID so that the final IDs are 1,2,3, and 4 but never something like 1,2,4,5.
Set the value for 'my_parent_id' to the integer ID that was given to this nest's parent domain.
Set 'n_children' to the number of child nests inside of this nest but not counting any deeper nests inside of those children.
Set 'i_parent_start' and 'j_parent_start' to the I and J indices on the nest's parent grid H point that coincide with the nest's SW corner H point. This implies that any nest's SW corner must lie directly on a parent grid H point.
Set 'parent_child_space_ratio' to the ratio of the parent's grid increment to the child's. Make this an integer. Although the code was originally written to allow this ratio to be any real number, tests with real values have not been done yet.
Set 'input_ready' to true if input data has already been produced for this nest. Set it to false if input data has not been produced and the user wants the parent to generate the nest's input data. NPS-generated input data is naturally preferable.
Set 'my_domain_moves' to false.

Moving nest configure files. See the example in job/configfile_moving_nmm.

Follow all instructions in 3(a)-(f).
Set 'my_domain_moves' to true.
Set 'ratio_sfc_files' to the ratio of the uppermost parent's grid increment to this moving nest's. Again this should be an integer. The use of moving nests requires the user to generate eight different surface-related static data files for each different resolution of nest in the run. If there are two moving nests with parent_child_space_ratio=3 then a set of the following eight files must be pre-generated: ALBASE_ij_3, FIS_ij_3, ISLTYP_ij_3, IVGTYP_ij_3, MXSNAL_ij_3, SM_ij_3, TG_ij_3, and VEGFRC_ij_3. These are the base albedo, sfc geopotential, soil type, vegetation type, maximum snow albedo, sea mask, deep underground temperature, and vegetation fraction, respectively, at the 3x nests' resolution but which span the entire upper parent domain. This data must be present as the nests move across the parent's domain. Then assume one of the 3x moving nests contains a 3x moving nest inside it. In the configure file for the inner nest the value of ratio_sfc_files would be 9 and the eight sfc datafiles would contain 9x data that spans the entire upper parent's domain. Note that the final integer in these files' names must be the value of ratio_sfc_files.
Set the values of 'nrows_p_upd_w', 'nrows_p_upd_e', 'nrows_p_upd_s', and 'nrows_p_upd_n' to 2. This is the number of rows around the edge of the nest domain that must be updated by the parent after the nest moves. The nest does not use its own data in these rows for updating itself because V is not present on the north/east sides and some variables needed in the integration part of the domain are not computed in these rows.
For now any child of a moving nest must be a moving nest and not a static nest. A static child would be tied to the parent and so when the parent moves then its child would move with it. A moving child is not tied to its parent and so when the parent moves then its child does not move with it and the child continues to determine its own motion independently.
The file called moving_nest_shift.txt must be present in the working directory. It holds critical specifications regarding variables in the Solver internal state when nests move. An explanation is given at the beginning of that file.

Task assignment:

When 1-way nesting is used then the user assigns forecast (compute) tasks and write (quilt) tasks uniquely for each domain in that domain's configure file. The I by J layout of forecast tasks are specified with configure variables inpes and jnpes, respectively. Any number of groups of write tasks can be assigned with the variable called write_groups. More than one write group should be used if the integration might move from one output time to the next before write tasks have finished with the earlier output. The number of tasks in each write group is assigned with the variable called write_tasks_per_group. The sum of inpes*jnpes+write_groups*write_tasks_per_group for all domains must equal the number of tasks that are assigned to the run in the runscript. This task assignment lets the user fine-tune the balance of work being done on all domains to minimize the time that any parent or child waits for the other thus leading to all compute tasks being busy virtually all the time as all domains integrate their forecasts simultaneously.

Configure file names:

The run script will copy each configure file to configure_file_01, configure_file_02, etc. where the final integers on the filenames form a monotonic sequence. The uppermost parent's configure file must be used for configure_file_01 but the user is not required to make the remaining files' names contain the same integer as their corresponding configure files' domain IDs.

NMM-B output files are written in a gridded format readable by GrADS (.ctl). To understand NMM-B output, please refer to the 'Output' tab. [close]

How To Run GFS (on CCS)

Currently, this process is similar to using Method #1 or #2 for the NMM-B, using a GFS regression test instead of NMM-B to start. Following the steps below may simplify the process if you choose to run other regression tests individually.

Modifying RT.sh, using regression test answers, and creating your own run script to run only the experiment(s) needed. [+/-]
Modifying RT.sh, gathering your own input files, and creating your own run script to run only the experiment(s) needed. COMING SOON [+/-]

Method #1: Using regression test answers

Modifying RT.sh is the simplest method by which to run the NEMS GFS. For this process we will use the first GFS regression test (baseline to compare with previous trunk version). This may simplify things if you choose to run other regression tests individually or create your own run scripts.

Check out the top of the NEMS trunk via Subversion (https://webarchive.library.unt.edu/web/20130304013300/https://svnemc.ncep.noaa.gov/projects/nems/trunk) for the latest version of the NEMS GFS. You can compile using the built-in compile options in /jobs/regression_tests/RT.sh or by using the instructions provided here or found in the README file included in the code. This example will utilize the built-in compile options from RT.sh.
FYI: For a GFS regression test, RT.sh will call rt_gfs.sh.

This script will create a new script called gfs_ll (which you submit to LoadLeveler) and a configure_file to go along with the settings you want or that are included in each individual regression test.

Copy RT.sh to a new file name (e.g. RT_new.sh, mycopy.sh) and use the new file for the rest of this process.
Keep the 'Clean and compile...' section around line 346 of the original RT.sh. All other 'Clean and compile...' sections can be commented out/removed.
It is okay to comment out/remove the following sections before your regression test, in addition to any regression tests not being used:

'Check if running regression test or creating baselines.' - For this case, no baselines are being created.
'If two arguments are provided...' - This section is utilized for those wanting to run the full regression test suite.

You may also want to comment out/remove the following:

'rm' statements in the 'finalize' section
'gmake clean' and 'esmf version 3' lines at the end of the script
'if' statement in in the test section for CB_arg and the corresponding 'fi'
portions of the 'clean and compile' section if you have already cleaned and compiled the code

Make sure you keep the 'exit' at the end of the file. Click here for an example of a working script.
Run RT.sh or whatever your new filename may be.

To understand GFS output, please refer to the GFS output descriptions after running the GFS post. [close]

How to Run FIM (coming soon)

Porting Information

Following is a procedure for defining functional equivalence on different environments:

Select key fields from the import state F_i (may include a selection of initial cases).
Run control run C of application on control environment.
Perturb each F_i in the last significant digit.

Nominally after the 6th decimal digit for 32-bit reals.
For each value F_i(x), construct a distribution.
Gaussian distribution with width being | F_i(x)/1.e6|.
Randomly select perturbation from that distribution.

Run perturbed run P from perturbed import state on control environment.
Select key fields from the export states C_j (control run) and P_j (perturbed run) - may include a selection of output times.
Determine area weight w(x)
Compute the butterfly vector .
Butterfly vector will have units of original export fields.
Now run control run but on different environment D.
Compute difference vector like butterfly vector but using D instead of P.
Compute dimensionless difference butterfly number vector .
For a bit-identical requirement, v_j must be zero.
For a functionally equivalent requirement, v_j must be largely less than 10.