IFPS Smart Initialization

Lesson Objectives

This tutorial describes the relationship between the ifpServer and the various Smart Initialization modules, the basic structure and how to modify the Smart Initialization files, the configuration files related to the Smart Initialization and how to diagnose Smart Initialization problems.

Upon completion of this module you will be able to:

  • Describe the relationship between the ifpServer and Smart Initialization.
  • Describe the basic structure of the Smart Initialization file.
  • Describe how to modify entries in Smart Initialization files.
  • Describe the parts of the serverConfig.py and localConfig.py files related to Smart Initialization.
  • Describe how to diagnose Smart Initialization problems.
    •

    Table of Contents

    1. Overview

    7.1. From The Command Line

    2. Basic SmartInit File Structure

    7.2. Relation To The ifpServer

    2.1. The Header

    7.2.1. INITMODULES

    2.2. The Levels

    7.2.2. INITSKIPS

    2.3. Function Set

    7.2.3. D2DDIIRS

    2.4. The Tail

    8. Troubleshooting 

    2.5. Working with SCALAR, VECTOR, WEATHER, and DISCRETE Data

    8.1. Log Files

    3. Modifying Entries in SmartInit Files

    8.2. Using ncdump to Diagnose Problems

    4. Adding Entries in SmartInit Files

    8.3. Specific Troubleshooting Examples

    5. Add New Models

    9. Summary

    6. Coding Examples

    10. Review Questions

    7. Running SmartInit

    11. Appendix

    1. Overview.

    Smart Initialization is closely coupled with the ifpServer. Throughout this module, the words Smart Initialization will be abbreviated to just SmartInit.  The ifpServer is aware of D2D model data changes and therefore knows when IFP data grids can be created.  The IFP data are the surface derived grids according to the definition in the BASE serverConfig.py file and the unique grids you may have defined in your localConfig.py file.  The Fcst defined grids are displayed in the Grid Manager above the blue line and the IFP model grids not editable below the blue line.  In addition, model fields from D2D can also be displayed in the Grid Manager below the blue line.  The following two figures show the Weather Element Browser displaying some of the IFP and D2D data types.  Go here to view the various weather element fields and available levels for each of the models in the BASE directory.

    The grid database types displayed in the Grid Manager can be IFP or D2D as shown in the following two figures:

    Sample Entries in The Weather Element Browser Showing IFP Data

    Sample Entries in The Weather Element Browser Showing D2D Data

    When the ifpServer first starts, or new D2D model data is detected, the ifpServer will examine the netCDF files inventory and the serverConfig/localConfig configuration files for the INITMODULES definition. If it finds a match with the newly arrived D2D model, it will spawn a process to run the appropriate model SmartInit which calculates the new IFPS grids from the model data. It then goes to sleep for 241 seconds (4 minutes and 1 second) and repeats the process. If new grids come in during the sleep period, GFE won't see them until the next time it does the scan.  The sleep time is non-configurable and subject to change in the future to support other more frequent datasets such as satellite and radar.

    When the ifpServer wakes up it updates it's internal list of available databases. This list remains constant until the next time the scan is made. During this scan the ifpServer also prunes the number of databases down to the user specified limit (which most likely is two). So, it is possible some databases such as ADAS10 may come in too quick for the default number of databases. In this case, you need to increase the number of databases retained for ADAS10 to something like 4 or more.  It is not recommended you set a purge time.

    It is also possible there may be a time period you may see a partially populated database. Some grid element calculations require several grids. If you try to populate the Fcst database with some model data such as the AVN and there are some blank grids, the problem may be in missing D2D model grids. For example, the T, Td, and Wx grids are all dependent on the following grids from the AVN D2D model in AWIPS:

    t for levels ["MB850", "MB700", "MB500", "MB400", "MB300"]
    gh for levels ["MB850", "MB700", "MB500", "MB400", "MB300"]

    If any one of these grids is missing then ifpInit will not generate a T grid for that time. Both Td, and Wx are dependent on the T grid, so, if there is no T grid, there will be no Td or Wx grids. Check and see if these native t and gh D2D grids are all in the Weather Element Browser for the time you are missing grids.  Likely, one of them will be missing.

    The algorithms to derive surface sensible weather elements in the early versions of the GFESuite were "hard-coded" and could not be changed by the user.  This process was changed to the SmartInit technique where each algorithm is coded in Python and able to be modified.  Information about how you can modify the initialization algorithms can be found in Section 3.

    When a SmartInit process is started, it will run a particular class file, either one supplied with the release, or one you have added via a local SmartInit.  The software will examine all of the functions within that class looking for function names beginning with calc***.  These functions define the output weather element, e.g., calcT will derive T, and also the dependencies.  The dependencies are determined from the argument list for each function.

    The SmartInit software then discerns the dependencies for all of the calc*** functions and will determine what must run first, second, third, etc.  Since this is done automatically, the person making changes to the SmartInit in BASE need not be concerned about it.  The software also examines the inventory for the dependent grids and the inventory for the output grids to determine if the output grid is already present and no calculations are needed, or if the output grid needs to be created by running the algorithms.  When all the possible dependencies and algorithms have run, the process exits until the next D2D model update.

    Since the SmartInit algorithms are written in Python and its extension, Numerical Python, you will need to have a knowledge of Python if you want to make changes to the initialization algorithms.

    The baseline SmartInit files are on px1 in the /awips/GFESuite/primary/etc/BASE directory.  As you should well know, no modifications should be made to any of the files in the BASE directory. User-customization to the baseline SmartInit files are placed in the /awips/GFESuite/primary/etc/SITE directory. As the case throughout IFPS, the SITE  files are not overwritten during installs and upgrades.

     

    Question

    Which of the following statements are true about the relationship between the ifpServer and SmartInit?

    A.About every four minutes, the ifpServer will look for new model data and then will spawn a process to run SmartInit for that model. 

    B.It is possible for new model data to come in, but, not get processed immediately since the ifpServer has not detected it.

    C.It is possible there may be a time period you may see a partially populated database because some elements require other elements or levels which may not have come in at the same time.

    D.All of the above statements are correct.

     

    Question

    All of the SmartInits are started by cron.

    True False

     

    Question

    The algorithms to derive surface sensible weather elements are "hard-coded" and can not be changed by the user.

    True False

     

    2. Basic SmartInit File Structure.

    The basic structure of a SmartInit file contains several parts which will be discussed in the following sections:

    2.1. The Header.

    The header contains the following parts:

    from Init import *

    class modelForecaster(derivedFromForecaster):
        def __init__(self):
            derivedFromForecaster.__init__(self, "sourceDb", "destDb")

    This is basically defining a new class called "modelForecaster", which is derived from "derivedFromForecaster".  The __init__ function is the constructor to the class, which calls the base class (class it has been defined from) with three arguments, self, "sourceDb", and "destDb".  The source db is the name of the D2D database, such as ETA, NGM, MRF.  The destination database is the name of the output database, such as Eta, MRF, AVN.  If there is an underscore in the destination database, then the format is the modelname_optionaltype such as ETA_SB for Slider Bar.  If the destDb argument is not given, then it defaults to the same name as the sourceDb.

    An example of the header in the NGM and Eta are shown below:

    For the NGM:

    from Init import *

    ##--------------------------------------------------------------------------
    ## Module that calculates surface weather elements from NGM model
    ## output.
    ##
    ##--------------------------------------------------------------------------
    class NGMForecaster(Forecaster):
    ��� def __init__(self):
    ������� Forecaster.__init__(self, "NGM")

    For the Eta:

    from Init import *

    ##--------------------------------------------------------------------------
    ## Module that calculates surface weather elements from Eta model
    ## output.
    ##
    ##--------------------------------------------------------------------------
    class EtaForecaster(Forecaster):
    ��� def __init__(self):
    ������� Forecaster.__init__(self, "ETA", "Eta")

    In these examples, there is a new class called NGMForecaster and EtaForecaster.  The source databases which are the name of the D2D database, are NGM and ETA respectively.  The destination database is the name of the output database. In the case of the Eta initialization, the destination database is called Eta.

    Your ifpInit script needs to be consistent with the naming convention in your Weather Element Browser.  For example, if the model name is WSETA in the Weather Element Browser, the following line in your new SmartInit will not be correct:

    Forecaster.__init__(self, "ETA", "Eta").

    In this example, the "ETA" is the name of the D2D database and the "Eta" string is for the IFP database.

    A WsEta script should look like the following:

    Forecaster.__init__(self, "WSETA", "WsEta")

    You will also need to make sure you have defined the 'WsEta' database in your localConfig.py.

     

    Question

    Which of the following statements are true about the header part of the SmartInit file?

    A.The source db is the name of the D2D database, such as ETA, NGM, MRF. 

    B.The destination database is the name of the output database, such as Eta, MRF, AVN.

    C.Your ifpInit script needs to be consistent with the naming convention in your Weather Element Browser.

    D.If the destDb argument is not given, then it defaults to the same name as the sourceDb

    E.All of the above statements are correct.

     

    2.2. The Levels.

    There usually is a function called levels() which defines a set of vertical pressure levels used when accessing cube data from the models.  These levels are used to create vertical soundings.  The levels() return a list of model dependent level values.  A complete example is shown below:

    For the NGM:

    ��� def levels(self):
    ������� return ["MB1000", "MB950", "MB850", "MB700", "MB500", "MB400", "MB300"]

    For the Eta:

    ��� def levels(self):
    ������� return ["MB1000", "MB950", "MB900","MB850","MB800","MB750",
    ��������������� "MB700","MB650","MB600","MB550", "MB500",
    ��������������� "MB450", "MB400", "MB350"]

    Note:  Make sure the levels you specify are actually available in the model.

    When you try to incorporate additional model levels into your SmartInit, you need to make sure these extra levels actually contain grids.  In AWIPS and IFPS, the number of model defined grids vary from the actual number created by the model at NCEP because not all grids are not being sent over the SBN for some models. A good way to check this is to use your GFE Weather Element Browser to verify all of the levels you have defined contain grids. If even one is missing, then SmartInit will not run the calculation, because it is waiting for data that will never show up.

    It is planned sometime this summer more GFS vertical levels will become available in AWIPS.  You can see what levels are currently available by running the following ncdump commands:

    ncdump -v ghLevels [filename]
    ncdump -v tLevels [filename]
    ncdump -v rhLevels [filename]

    Additional information about the ncdump command and it's optional switches can be found in Section 8.2 below.

    Sometime ago, new netCDF template files were installed at most sites in anticipation of the new 50 mb vertical resolution grids.  When these grids arrive in the field, sites with the new templates will see the grids in D2D.  At sites with the old templates, the new grids will fall on the floor until the new template is properly installed.  It is expected new templates will be reinstalled at all sites sometime in June (OB3.2?).  When the new grids begin flowing, new ifpInit scripts will be required to take advantage of the better vertical resolution.

     

    Question

    Which of the following statements are true about the level() function in the SmartInit file?

    A.The levels() function defines a set of vertical pressure levels. 

    B.If a grid is missing for a level, SmartInit will not run the calculation.

    C.You can verify all of the levels you have defined contain grids via the Weather Element Browser .

    D.All of the above statements are correct.

     

    2.3. Function Set.

    Following the levels() function are the named set of functions.  A function consists of a header line, followed by a block of indented statements.  The indented statements become the function's body; the code Python executes each time the function is called.  You use the def statement to create a function.  The header specifies a function name along with a list of arguments (sometimes called parameters), which are assigned to the objects passed in parentheses at the point of call in the form:

    def <name>(arg1, arg2, ...argN):
          <statement>
         return <value>

    To invoke a function, simply use the name of the function followed by its arguments enclosed in parentheses.  You can use a tuple to return multiple values from a function.

    The Python return statement can show up in function bodies. It ends the function call and sends a result back to the caller.  It consists of an object expression giving the function's results.

    Here is an example of applying these terms to the more commonly used calc function in SmartInit.

    For SmartInit to determine what to run, the calculation function MUST BEGIN WITH calc in it's name.  The remainder of the name of the function after "calc" is the parameter name to create.  For example, if RH is your weather element name, the name of the function to calculate RH would be calcRH().

    When a SmartInit process is started, it will run a particular class file, either one supplied in the baseline, or one you have added.  The software will examine all of the functions within that class looking for function names beginning with calc***.  These functions define the output weather element, e.g., calcT will derive T, and also the dependencies.  The dependencies are determined from the argument list for each function.

    The SmartInit software then figures out the dependencies for all of the calc*** functions and will determine what must run first, second, third, etc.  Since this is done automatically, the programmer need not be concerned about it.  The software also examines the inventory for the dependent grids and the inventory for the output grids to determine if the output grid is already present and no calculations are needed, or if the output grid needs to be created by running the algorithms.

    The typical format for a calc function is shown below.  This example calculates surface temperature from the Eta model:

        def calcT(self, t_FHAG2, t_BL3060, p_SFC, stopo, topo):
            dpdz = 287.04 * t_FHAG2 / (p_SFC / 100 * 9.8) # meters / millibar
            # 45milibars is halfway between 30 and 60
            dpdz = dpdz * 45 # meters between p_SFC and t_BL3060
            lapse = (t_FHAG2 - t_BL3060) / dpdz # degrees / meter
            lapse = clip(lapse, lapse, 0.012)
            t = t_FHAG2 + lapse * (stopo - topo)
            return self.KtoF(t)

    The following contains more details about the above coding:

    1. The following grids from the D2D model database are passed into this routine, 2 meter temperatures (t_FHAG2), the boundary layer temperatures from the 30-60 hPa levels (t_BL3060), the surface pressure in Pascals (p_SFC), surface topography from the model (stopo), and the high resolution topographical data (topo).

        def calcT(self, t_FHAG2, t_BL3060, p_SFC, stopo, topo):

    For the most accurate information, you should use the ncdump command described in Section 9.2 and examine the model data files directly.

    2. The lapse rate is calculated, but,  first the number of meters between the surface and 30-60 hPa level is determined.

            dpdz = 287.04 * t_FHAG2 / (p_SFC / 100 * 9.8) # meters / millibar
            # 45milibars is halfway between 30 and 60
            dpdz = dpdz * 45 # meters between p_SFC and t_BL3060

    3. The lapse rate can now be calculated:
            lapse = (t_FHAG2 - t_BL3060) / dpdz # degrees / meter

    4. Ensure the lapse rate can't get too steep by clipping:
            lapse = clip(lapse, lapse, 0.012)

    5. Calculate the surface temperature, which is the model 2m temperature modified by the lapse rate and the difference between the model surface and the real surface elevation.
            t = t_FHAG2 + lapse * (stopo - topo)

    6. Perform unit conversion and return.
            return self.KtoF(t)

    When all the possible dependencies and algorithms have run, the process exits until the next D2D model update.

    The name of the function is always calc*** where *** is the weather element name and level.  If you are creating weather elements for surface data then the weather element name by itself is sufficient.  If you are creating weather elements for upper air or non-surface data, then the name of the calc function is:  calc***_***, such as calcWind_3K() for the Wind at 3000 feet.

    Note:  In the argument list for the calc*** functions, the first argument is always self.  The remainder of the arguments represent gridded data.  The format of the specification can be one of the following formats:

    Format Example Purpose
    EditAreaName Colorado The name of an edit Area. It is probably best to use polygon edit areas instead of queries (untested with queries). The value of the parameter, in this case Colorado, will be a Boolean grid suitable for use as a mask for numeric functions.
    parmName_level t_FHAG2 Refers to a single grid for the parmName and the level. The example accesses the temperature grid from the model at the FHAG2 (fixed height above ground 2m level)
    parmName_c rh_c Refers to a cube of data for the parmName. The "_c" indicates the cube. The number of layers in the cube depend upon the levels() function contents. For example, if the levels() contain:
      def levels(self):
            return ["MB950","MB900","MB850","MB800","MB750",
                    "MB700","MB650","MB600","MB550", "MB500",
                    "MB450", "MB400", "MB350"]
    then the cube will contain 13 levels.  Access of individual levels can be done using indexing within the function.
    topo topo Refers to the high-resolution surface topography field, in units of meters above MSL.
    stopo stopo Refers to the model topography field, in units of meters above MSL.
    ctime ctime Time from the source database grid currently being calculated, as a time range tuple (startTime, endTime), in seconds since January 1, 1970 at 0000z.
    mtime mtime Time in the destination database grid currently being calculated, as a time range tuple (startTime, endTime), in seconds since January 1, 1970 at 0000z.
    stime stime Number of seconds from the model basetime currently being calculated, in seconds since the model analysis time.
    parmName FzLevel Refers to the weather element in the OUTPUT database, not the INPUT D2D database.

    When a calc method fails, the grid it was supposed to create is set to the python None object.  So, any other calc method which needs T will then also fail because it is now getting a None instead of a T grid.

    You can place additional functions (e.g., utility) functions anywhere in the file after the constructor (__init__) and before the tail end of the file.   An example of a utility function could be one to calculate Td from T and RH as shown below:

       def getTd(self, t, rh):
            # input/output in degrees K.
            desat = clip(t, 0, 373.15)
            desat = where(less(t, 173.15), 3.777647e-05, t)
            desat = exp(26.660820 - desat * 0.0091379024 - 6106.3960 / desat)
            desat = desat * rh / 100.0
            desat = 26.66082 - log(desat)
            td = (desat - sqrt(desat*desat-223.1986))/0.0182758048
            td = where(greater(td, t), t, td)
            return td

    Thee are several "utility" functions in Init.py located in the /awips/GFESuite/primary/etc/BASE directory on px1.  The following are some examples:

    Calculate the wet bulb (C) based on temperature (C) and RH (%)
    Calculate parcel temp (K) based on Theta-E (K), pressure (mb) and temp (K)
    Calculate dewpoint (C) based on temp (C) and RH (%)
    Calculate Theta-E (K) based on pressure (mb), temp (C) and dewpoint (C)
    Calculate temp at LCL (K) based on temp (C) and dewpoint (C)
    Calculate mixing ratio (g/kg) based on dewpoint (C) and pressure (mb)
    Calculate vapor pressure (mb) from dewpoint (C)
    Calculate saturation vapor pressure based on temp (K)
    Calculate the potential temp (K) based on temp (K) and pressure (mb)
    Converts the value from m/s to kts
    Converts the value from ft to m
    Converts the value from F to K
    Converts the value from K to F

     

    Question

    Which of the following statements are true about the calc*** function in the SmartInit file?

    A.You use the def statement to create the function. 

    B.The calculation function must begin with calc in it's name, followed by parameter name to create such as Sky.

    C.The first argument for the calc*** functions is always self with the remainder of the arguments representing gridded data..

    D.All of the above statements are correct.

     

    Question

    When a calc method fails, the grid it was supposed to create is set to the python None object, so, any other calc method which needs T will then also fail because it is now getting a None instead of a T grid.

    True False

     

    2.4. The Tail.

    The tail end of the SmartInit file contains a definition of main() and must be similar to the following:

    def main():
        modelForecaster().run()

    if __name__ == "__main__":
        main()

    Here is an example of the tail to the Eta file.

    Note: The name of the class within the main() function must match the name of the class you have defined in the header:

    def main():
        EtaForecaster().run()

    if __name__ == "__main__":
        main()

     

    Question

    The name of the class within the main() function does not need to  match the name of the class you have defined in the header.

    True False

     

    2.5. Working with SCALAR, VECTOR, WEATHER, and DISCRETE Data.

    This section describes accessing scalar, vector, and weather data through numerical python.

    SCALAR

    When passing in a scalar weather element, you will either get a grid, or a cube.   The grid is a numeric 2-d grid (x,y), the cube is a numeric 3-d grid (z,x,y).

    VECTOR

    For a single vector grid (at a single level), you get a tuple.  The first element is a grid of magnitude, the second is a grid of direction. To access the magnitude grid, use this syntax: wind_SFC[0], and for direction, use this syntax: wind_SFC[1].   Once you access either the magnitude or direction grid, they are treated like a scalar grid.

    There are several "utility" functions in Init.py (located in your /awips/GFESuite/primary/etc/BASE directory on px1) that can help you when working with vector data. The self._getUV( mag, dir) call will convert a magnitude/direction grids into a returned tuple of u and v.  The u component is [0] and the v component is [1]. The self._getMD(u,v) function converts a grid in u and v components into a tuple of magnitude and direction.  The magnitude component is [0] and the direction component is [1].

    WEATHER

    Weather is much more complicated and can be a big performance problem. A tuple is provided.  The first element is a grid, which contains the indexes into the key. The second element is a sequence of all of the keys. The keys are the ugly strings associated with a WeatherKey. To access the grid:

    Wx[0]

    To access the sequence, use:

    Wx[1]

    To access a particular entry in the sequence, use:

    Wx[1][3], would give your the 4th key.

    Normally you don't access the weather grid in your calculations, but if you need to, you have generally created a weather grid first.  In SmartInit, all of the possible weather keys that can be created are known and set up in a table with those keys.  You only need to simply poke in the correct index for the key.  This is more efficient than searching the keys for each grid point.

    The following is part of a SmartInit log file complaining about the Wx grid calculation.

    ----------------------------------------------------------------- 

    03:36:32.018 Init.py 824 EVENT: Calc: Wx (20030605_0000, 20030605_0100) 
    03:36:38.305 Init.py 824 PROBLEM: Error while running method Wx 
    Traceback (most recent call last): 
    File "/awips/GFESuite/etc/BASE/Init.py", line 838, in __runMethod 
    self._ifpio.store(parm, cache['mtime'][0], cache[we][0]) 
    File "/awips/GFESuite/etc/BASE/Init.py", line 1014, in store 
    newwe[newwe.getTimeRange(time[0])] = grid 
    File "/awips/GFESuite/etc/BASE/ifpc.py", line 17, in __setitem__ 
    val = ifpcc.IFPWE___setitem__(self.this,arg0,arg1) 
    Exception: Invalid WeatherKey found in Grid. Key Position is 1 Key is:
    <Invalid>:<Invalid>:<Invalid>:<Invalid>:<Invalid>Save Grid Data failed 
    ------------------------------------------------------------------- 

    Typically the calcWx() declares a set of keys and then assigns the grid values appropriately. In this example, there is a key in the SmartInit algorithm not defined in the localWxConfig. For example, if you no longer have Snow in your weather configuration the SmartInit routine for the AVN can predict snow, and even if it doesn't predict snow, that is enough to make the SmartInit bomb.

    Here is an example of a portion of the calcWx() from the AVN.py SmartInit:

    # Now apply a different algorithm for each type 
    key = ['<NoCov>:<NoWx>:<NoInten>:<NoVis>:', 
    "Wide:S:-:<NoVis>:", "Wide:R:-:<NoVis>:", 
    "Wide:S:-:<NoVis>:^Wide:R:-:<NoVis>:", 
    'Wide:ZR:-:<NoVis>:', 'Wide:IP:-:<NoVis>:', 
    'Wide:ZR:-:<NoVis>:^Wide:IP:-:<NoVis>:', 
    "Sct:SW:-:<NoVis>:", "Sct:RW:-:<NoVis>:", 
    "Sct:SW:-:<NoVis>:^Sct:RW:-:<NoVis>:", 
    "Chc:ZR:-:<NoVis>:", 'Chc:IP:-:<NoVis>:', 
    'Chc:ZR:-:<NoVis>:^Chc:IP:-:<NoVis>:']

    As you can see, it is defining ZR, SW, IP.  If you delete these, you will need to override the calcWx() method in each of the baseline provided SmartInits to remove those keys. Of course, the rest of the calcWx() method will also have to change.   For example, the current value for widespread rain is 2 (since it is key #2 in the key list). If you remove the first key (Wide Snow), then the rest of calcWx() where you assign 2, must be changed to assign 1, which will be the new key for wide rain.

    DISCRETE

    Discrete is much more complicated than the simple scalar and vector case, and like weather, can be a big performance problem. A tuple is provided. The first element is a grid, which contains the indexes into the key. The second element is a sequence of all of the keys.   The keys are the discrete key values associated with the weather element.  To access the grid:

    DK[0]

    To access the sequence, use:

    DK[1]

    To access a particular entry in the sequence, use:

    DK[1][3], would give your the 4th key.

    Normally you don't access the discrete grid in your calculations, but if you need to, you have generally created a discrete grid first.  In SmartInit, all of the possible weather keys that can be created are known and set up in a table with those keys.  You only need to simply poke in the correct index for the key.  This is more efficient than searching the keys for each grid point.

     

    Question

    Which of the following statements are true about accessing scalar, vector, and weather data through numerical python?

    A.Discrete and Weather are more complicated than the simple scalar and vector cases, so, they can cause a big performance problem. 

    B.When you access a single vector grid (at a single level), you get a tuple where the first element is a grid of magnitude and the second is a grid of direction.

    C.To access either weather or discrete, you get a tuple where the first element is a grid, which contains the indexes into the key and the second element is a sequence of all of the keys.

    D.All of the above statements are correct.

     

    3. Modifying Entries in SmartInit Files.

    The basic steps to modify an existing algorithm are:

    Step 1. Create a new file in the /awips/GFESuite/primary/etc/SITE directory on px1, named similarly, but not identical to, the class you are modifying.

    For example, if you are modifying the Eta model initialization (stored in file named Eta.py), you should name your new file MyEta.py to indicate this is your modifications to the baseline Eta.

    Step 2. Make your changes.  The format of the file should follow the standard structure described above.   In particular, it needs to include the following items:

    Step 3. Modify (create first if necessary) your server localConfig.py file to override the INITMODULES section of the serverConfig.py.

    The following is an example of overriding the Eta derivation of Snow Amount. The original Eta.py file contains the following information (only part of the file is shown).  The snow amount function calculates the snow ratio which varies depending upon temperature, and then assigns the snow amount based on the snow ratio and QPF where the snow level (freezing level - 1000

    from Init import *

    class EtaForecaster(Forecaster):
        def __init__(self):
           Forecaster.__init__(self,"ETA", "Eta")

        def levels(self):
            return ["MB950","MB900","MB850","MB800","MB750",
                    "MB700","MB650","MB600","MB550", "MB500",
                    "MB450", "MB400", "MB350"]

        def calcSnowAmt(self, T, FzLevel, QPF, topo):
            m1 = less(T, 9)
            m2 = greater_equal(T, 30)
            snowr = T * -0.5 + 22.5
            snowr = where(m1, 20, snowr)
            snowr = where(m2, 0, snowr)
            snowamt = where(less_equal(FzLevel - 1000, topo*3.048), snowr * QPF, 0)
            return snowamt

    def main():
        EtaForecaster().run()

    if __name__ == "__main__":
        main()

    Here is the derived MyEta file overriding the calcSnowAmt() function:

    from Eta import *

    class MyEtaForecaster(EtaForecaster):
        def __init__(self):
            EtaForecaster.__init__(self)

        def calcSnowAmt(self, T, QPF):
            m2 = less_equal(T, 32)
            snowamt = where(m2, 10.0 * QPF, 0)
            return snowamt

    def main():
        MyEtaForecaster().run()

    if __name__ == "__main__":
        main()

    The algorithm was changed to have a fixed 10:1 snow ratio anytime the temperature is below 32.  The freezing level is no longer used in this revision.  Of course, you can completely rewrite the algorithm, use different arguments, etc.

    Note: The name of the function, calcSnowAmt() in this case is identical to the name in the original file.  This is important!

    To change entries in the BASE model initialization (only use the 2m Temps for AVN), you will need to either create a SITE level file called MyAVN.py or make an entry in an existing MyAVN.py file. In the file you will need the following entries: 

    from AVN import * 

    class MyAVNForecaster(AVNForecaster): 
    def __init__(self): 
    AVNForecaster.__init__(self) 

    def calcT(self, t_FHAG2): 
    return self.KtoF(t_FHAG2)     

    def main(): 
    MyAVNForecaster().run() 

    if __name__ == "__main__": 
    main() 

    In these two example, you need to tell the ifpServer to use your version of the SmartInit rather than the default. You do this by modifying the INITMODULES section in your localConfig.py file in the /awips/GFESuite/etc/SITE directory on px1.  You tell the ifpServer to not use the BASE SmartInit (del) and use your new SmartInit instead.  In the above two examples, your INITMODULES entries would be:

    del serverConfig.INITMODULES["Eta"] 
    serverConfig.INITMODULES["MyEta"] = ["Eta"] 

    del serverConfig.INITMODULES["AVN"] 
    serverConfig.INITMODULES["MyAVN"] = ["AVN"]

    Since there is already an INITMODULES entry in the serverConfig.py file, you prefix it with serverConfig.

     

    Question

    Which of the following statements are true about modifying entries in SmartInit files?

    A.The contents and format of the file should follow the standard structure containing at least a header, the replacement function, and a tail. 

    B.If you are modifying an existing function, the name of the function such as calcSnowAmt(), needs to be identical to the name in the original file.

    C.The name of the class within the main() function in the tail must match the name of the class you have defined in the header.

    D.All of the above statements are correct.

     

    4. Adding Entries in SmartInit Files.

    Adding a new algorithm is just about the same as modifying an existing algorithm.  You will do similar steps:

    Step 1. Create a new file in the /awips/GFESuite/primary/etc/SITE directory on px1, named similarly, but not identical to, the class you are modifying.  For example, if you are modifying the Eta model initialization (stored in file named Eta.py), you should name your new file MyEta.py to indicate this is your modifications to the baseline Eta.

    Step 2. Make your changes.  The format of the file should follow the standard structure described above.   In particular, it needs to include the following items:

    Step 3. Add your new function into the newly created file.

    Define how the additional fields are to be computed.  If you have not already overridden SmartInit calculations before, it is safe to copy these files into your /awips/GFESuite/primary/etc/SITE directory as is. If you do this, ensure you use the same filename for each model.

    If you already have SITE files for SmartInit. Simply cut and paste the functions you wish to use into your existing file for each model in the awips/GFESuite/primary/etc/SITE directory.

    Step 4. Modify (create first if necessary) your localConfig.py file to override the INITMODULES section of the serverConfig.py.

    Step 5. Modify (create first if necessary) your localConfig.py file to add the new weather elements you want the model initialization to populate.

    Step 6. Test your new SmartInit files with this command:

    ./ifpInit -a {filename} (do not include the final '.py')

    If there are errors, fix your SmartInit files until all the fields you chose are being produced correctly.

    Here is an example of a MyEta.py initialization file creating a new weather element called RH for the Eta model.  It does nothing more than taking the model RH FHAG2 field and storing it into the Eta RH weather element, after ensuring the field ranges between 0 and 100%:

    from Eta import *

    class MyEtaForecaster(EtaForecaster):
        def __init__(self):
            EtaForecaster.__init__(self)

        def calcRH(self, rh_FHAG2):
            return clip(rh_FHAG2, 0, 100)

    def main():
        MyEtaForecaster().run()

    if __name__ == "__main__":
        main()

    Here is another complete example of creating WaveHeight and Surface Wind from the GWW model.  The wave height logic catches very high values and assumes they are missing data and resets them to zero height.  There is also a conversion from meters to feet.  The wind logic also catches very high values and assumes they are missing data and resets the winds to calm.  There is also a conversion from meters/second to knots:

    from Init import *
    class GWWForecaster(Forecaster):
        def __init__(self):
            Forecaster.__init__(self, "GWW", "GWW")

        def calcWaveHeight(self, htsgw_SFC):
            grid = where(greater(htsgw_SFC, 50), 0.0, htsgw_SFC/3.048)
            return clip(grid, 0, 100)

        def calcWind(self, wind_SFC):
            mag = where(greater(wind_SFC[0], 50), 0, wind_SFC[0]*1.94)
            dir = where(greater(wind_SFC[0], 50), 0, wind_SFC[1])
            dir = clip(dir, 0, 359.5)
            return (mag, dir)

    def main():
        GWWForecaster().run()

    if __name__ == "__main__":
        main()

    5. Adding New Models.

    Adding new models, and all of the algorithms to derive the elements isn't much different from adding new algorithms or modifying an existing algorithm. The basic steps are:

    Step 1. Create a new file.

    Name the file appropriately (and not the same name as any file in the /awips/GFESuite/primary/etc/BASE directory) and place it  in the /awips/GFESuite/primary/etc/SITE directory on px1.

    For example, if you are going to create a GWW model initialization, then the name should probably be GWW.py.

    Step 2. Put in your information.

    The format of the file should follow the standard structure described above.   In particular, it needs to include the following items:

    Step 3. Modify the INITMODULES section in your localConfig.py file.

    If you do not already have one, you may need to first create a localConfig.py file.

    Step 4. Modify your localConfig.py file to define the new database and all of the weather elements associated with the database.

    Your IFP database can be called anything.  To simplify things, it should be called by the model name listed under Source in the IFPS Weather Element Browser such as the MM5 is based on what is coded in the D2D netCDF file located in the /data/fxa/Grid/SBN/netCDF/.... directory.  It is this model name ifpInit needs to know.  It is the destination db from SmartInit and the one containing the parms you have defined.  The source db doesn't need to be defined, but, is what D2DDIRS points to.  The model variable in the netCDF file defines the name of the source db.

    To clarify things, consider the following naming conventions:

    Name your SmartInit module to be the model name such as MM5.py.

    Name your variables to match the model name.

    The constructor contains the source and destination database names. The source is MM5, since that is what is in the "model" netCDF variable.  Call the destination database anything, but, to be conventional, call it MM5.

    Note: The destination database must be defined in localConfig. If you decide to call it MM5, the database definition line in localConfig has to have "MM5" in it.

    The Python variable names can be called anything.  You could call them A and B and it wouldn't make any difference, as long as you refer to them in the dbs= line by their Python variable names. For conventions we usually name them the same as the database name ("MM5").

    The INITMODULES is coded in the form of a dictionary entry of the destination SmartInit module with the source database model name.

    Here is an example of the entries in your localConfig.py file and what is needed in your SmartInit file if you are adding a new model such as the MM5:

    LocalConfig File Entries:

    ###########################
    ### Add MM5 database ###

    #Location of the netCDF file.

    serverConfig.D2DDIRS.append('/data/local/Grid/MM5')

    #Tell the ifpServer to run the init script.
    #The first name is the GFE database, the second name is the AWIPS name. They don't have to be the same, although they often are.

    serverConfig.INITMODULES["MM5"] = ["MM5"]

    #The format is: name / format / type / single / official / numVer / purgeAge

    MM5 = ('MM5', GRID, '', NO, NO, 2, 0)

    #Define the new parm groupings which relate weather elements to time constraints.

    MM5Parms = [([Temp, Td, Wind, RH, QPF], TC1),([MaxT], MaxTTC),([MinT], MinTTC)]

    #Relate the parm groupings to the database attributes and then append those to a list of new databases.

    dbs = [(MM5, MM5Parms)]

    SmartInit file (MM5.py):

    class MM5Forecaster(Forecaster):
    def __init__(self):
    Forecaster.__init__(self, "MM5", "MM5")

    blah...blah...blah...

    def main():
    MM5Forecaster().run()

    The following example shows the required entries in your localConfig.py file pertaining to adding a new model database. In this example it is the Eta12:

    ####################################### 
    ### Create the Eta12 database###
    # Add Eta12 to the database
    serverConfig.D2DDIRS.append('/data/local/ETA12')

    # Run the init script
    serverConfig.INITMODULES["gjtETA12"] = ["ETA12"]

    # Define the number of models to keep
    ETA12 = ('ETA12', GRID, '', NO, NO, 2, 0)

    # Add the Eta12 parms
    Eta12Parms = [([Temp, Wind, QPF, RH, PoP1hr], TC1),([MaxT], MaxTTC), ([MinT], MinTTC)]

    dbs = [(ETA12, Eta12Parms)]

    #######################################

     

    Question

    Which of the following statements are correct about your SmartInit naming conventions?

    A.Your IFP database can be called anything.  To simplify things, it should be called the model name such as MM5. 

    B.The source database doesn't need to be defined, but, is what D2DDIRS points to.

    C.The model variable in the netCDF file defines the name of the source database.

    D.All of the above statements are correct.

     

    Question

    The model name listed under Source in the IFPS Weather Element Browser such as the MM5 is based on what is coded in the D2D netCDF file located in the /data/fxa/Grid/SBN/netCDF/.... directory.  It is this model name ifpInit needs to know.

    True False

     

    6. Coding Examples.

    The following are specific coding examples highlighted in red.

    To ensure the field ranges are between certain values (clip).  In this example, the RH is between 0 and 100%:

        def calcRH(self, rh_FHAG2):

            return clip(rh_FHAG2, 0, 100)

    To catch very high values of wave height, assume they are missing data, reset the wave heights to zero height and convert them from meters to feet.

        def calcWaveHeight(self, htsgw_SFC):

            grid = where(greater(htsgw_SFC, 50), 0.0, htsgw_SFC/3.048)

            return clip(grid, 0, 100)

    You can derive the winds at the top of the mixing layer (i.e., MixHgt) by first calculating the mixing height.  Then, calculate the winds at this height, per grid point.  If you look at the ifpInit method that calculates the freeWind, you will see the method calculates wind at 3000 feet.  Substitute the 3000 foot value with the MixingHeight grid and you will obtain the wind at the mixing height.

    There is also the coding (clip) ensuring the field ranges are between certain values.

    To catch very high values of wind, assume they are missing data, reset the winds to calm and convert them from meters/second to knots.:

        def calcWind(self, wind_SFC):
            mag = where(greater(wind_SFC[0], 50), 0, wind_SFC[0]*1.94)
            dir = where(greater(wind_SFC[0], 50), 0, wind_SFC[1])
            dir = clip(dir, 0, 359.5)
            return (mag, dir)

    Here is some simple code to insert in your model SmartInit to calculate the 850 mb T (degrees C), 850-700 mb thickness and the 1000-850 mb thickness:

    def calcT850(self, t_MB850):
    return t_MB850 - 273.15

    def calcThick850T700(self, gh_MB850, gh_MB700):
    return gh_MB700 - gh_MB850

    def calcThick1000T850(self, gh_MB1000, gh_MB850):
    return gh_MB850 - gh_MB1000

    To reference edit areas in the model init scripts, simply include the name of the edit area in the calc***() function. For  example, here is a calcT() that takes an edit area name of AK005:

    def calcT(t_c, gh_c, AK005):

    7. Running SmartInit.

    7.1. From the Command Line.

    Normally you don't need to run SmartInit from the command line since SmartInit is automatically started by the ifpServer when new D2D model data arrives.  This is because you have the INITMODULES line in the serverConfig or your localConfig line.

    Although you should never need to run the ifpInit manually, since the ifpServer monitors the received data on D2D and automatically queues SmartInit to run, you may want to run SmartInit from the command line, when you are developing new algorithms or modifying existing ones before you add the SmartInit to your localConfig.py file. 

    The proper syntax for ifpInit is the following:

    ifpInit [-h host] [-p port] [-t modelTime] [-a]  algorithmFile
     
     
    Switch Optional? Purpose
    -h host yes Defines the host upon which the ifpServer is running.  Normally this switch is not needed and the software will determine where the ifpServer is running.
    -p port yes Defines the RPC port upon which the ifpServer is running.  Normally this switch is not needed and the software will determine where the ifpServer is running.
    -t modelTime yes Specifies the model run time in the format of yyyymmdd_hhmm.  If not specified, then run using the latest model data. 
    -a yes Specifies to create all of the possible data grids, which will overwrite existing previously calculated grids.  Normally by default, only those grids that haven't yet been created will be attempted to be calculated.
    algorithmFile no Mandatory argument specifying the name of the SmartInit module, such as Eta, or MyEta.

    Note: The -h and -p switches are not necessary for normal running of ifpInit.  However, if you wish to connect to a different server, then you will need to specify the -h and -p switches. If environment variables ${CDSHOST} or ${CDSPORT} are defined, then the default server and port will be determined from the environment variables, unless overridden with the user specified -h and -p switches.

     You can run ifpInit manually by following these simple steps:

    1. Go to the /awips/GFESuite/bin directory on any lx, as any user
    2. Type ifpInit -a <modelname>

    The -a switch forces all time steps to be recalculated.  This gets around the problem ifpInit doesn't detect the additional data.

    The algorithm file name to ifpInit is the name of the python *module* containing your code, not the file name.  For example, use the name MyEta instead of MyEta.py (without the .py).  So, your command would look like the following:

    ./ifpInit -a MyEta

    If you use the .py extension, you will see the following common error when trying to run a SmartInit from the command line:

    ####################################### 
    ./ifpInit -a gjtETA12.py 
    20:59:22.060 Init.py 859 PROBLEM: Can't load module for gjtETA12.py 
    Traceback (most recent call last): 
    File "/awips/GFESuite/etc/BASE/Init.py", line 862, in main 
    exec("import " + Options['model'] + "; mod = " + Options['model']) 
    File "<string>", line 1, in ? 
    File "imputil.py", line 66, in _import_hook 
    File "imputil.py", line 66, in _import_hook 
    ImportError: No module named py 
    ####################################### 

    The clue is the last line which states "No module named py".

    You can check to see if you have overrides for SmartInit algorithms by comparing with the baseline version.  Go to the /awips/GFESuite/bin directory on lx1, and run the SmartInit in BASE by using the following command to see the Eta:

    ./ifpInit -a Eta

     

    Question

    Which of the following statements are true about the proper syntax for ifpInit?

    A.If the -t switch is not specified, then the latest model data are used. 

    B.If the -a switch is not specified, only grids that haven't been created will be calculated.

    C.Generally, the -h (host) and -p (port) switches are not needed.

    D.All of the above statements are correct.

     

    Question

    You only need to specify ifpInit and the algorithm file name to run ifpInit.

    True False

     

    Question

    You can run any SmartInit, baseline or locally defined, as any user.

    True False

     

    7.2. Relation to The ifpServer.

    There are three serverConfig variables, INITMODULES, INITSKIPS, and D2DDIRS, in your serverConfig or localConfig controlling the operation of the various SmartInit programs.

    7.2.1. INITMODULES.

    The INITMODULES defines your IFP databases and which model is used to generate them.  To get the server to run the model SmartInit automatically, add the model to INITMODULES.

    The INITMODULES contains the mapping of dictionary key (modulename) to value (database model name).  In simple terms, the first entry is the name of the database and the second is the name of the ifpInit script to run. Most of the time the names are the same, but, they can be different.  Make sure these names are spelled correctly.   Every time you make a change to your localServer.py file, you will also need to bounce the server.

    If you can manually run an ifpInit for a model and see them in the Weather Element Browser, then you can make these SmartInits kick off automatically by adding the INITMODULES line in your localConfig.py. For example, the entries will look something like the following:

    serverConfig.INITMODULES['GLERL'] = ['D2DGERLDatabaseName']
    serverConfig.INITMODULES['Eta12'] = ['D2DEta12DatabaseName']

    In the above lines substitute the database name you see in the GFE's Volume Browser (switched to the D2D mode).  Then restart the server.

    Because of the INITMODULES setup, SmartInit does not need to be run from a cron job or from the command line. It should run automatically as new data comes into the associated D2D directory.  The following figure shown the INITMODULES section defined in the serverConfig.py file in the BASE directory.

    Example of the INITMODULES part of serverConfig.py

    This is standard Python syntax for a dictionary, with the keys on the left of the colon and the values as a list on the right side. For example, the line:

    "MesoEta" : ["MESOETAU", "MESOETAS"]

    indicates the MesoEta initialization module will be executed whenever changes occur in the MESOETAU and MESOETAS D2D databases. The MesoEta is more complicated than most models since it arrives in two completely separate netCDF files and resolution. The syntax for the "Eta" model is more straightforward:

    "Eta" : ["ETA"]

    which indicates the Eta initialization module (named Eta.py and found in the /awips/GFESuite/primary/etc/BASE) directory will be executed whenever the D2D ETA model changes. Remember the initialization module is on the left, and the D2D model name files are on the right.

    7.2.2. INITSKIPS.

    If you don't want to initialize certain model runs, you can use the INITSKIPS feature.  By default, the Smart Initialization will run for each model run of a particular model. This behavior can be changed by listing the model name and the hours to skip. For example, here is a section in serverConfig.py:

    INITSKIPS = {
    "RUC" : [1,2,4,5,7,8,10,11,13,14,16,17,19,20,22,23],
    }

    The above line indicates all runs of the RUC will be ignored, except every 3 hours. The default for serverConfig is to run all of the available model runs, except for the RUC, which is run only every 3 hours.

    INITSKIPS only works by the hour, it will not work for 15 min grids.

    7.2.3. D2DDIRS.

    The D2DDIRS definition in your localConfig file determines which "D2D" databases show up in your GFE.

    The following figure shows the section in the BASE serverConfig.py file identifying the location of the D2D model data in the netCDF directories.

    Example of the netCDF Part of serverConfig.py File

    Question

    Which of the following statements are correct about the INITMODULES entry in the local or serverConfig files?

    A.This entry causes the SmartInit to run automatically as new data flow into AWIPS . 

    B.INITMODULES requires two entries; the database name and the name of the ifpInit script to run.

    C.You can use INITMODULES to rename a SmartInit (del entry).

    D.All of the above statements are correct.

     

    8. Troubleshooting.

    Use the following table to quickly find your problem and a possible solution:

    Blank Grids After Populate

    Model SmartInit Stops after AWIPS Upgrade

    Overflow Error

    Global name 'topo' is not defined Error

    No database Error

    Unknown PID Error

    MesoEta Stopped being Generated

    No MSAS in GFE

     

    There have been reports of additional files (other than the default number) in a model's database directory.  Some of the files may be dated and some contain all 0's for the date and time. In this case, you should delete the dated files.

    8.1. Log Files.

    When a SmartInit is run, it creates a log file with the same name (For example, MyMesoEta). These log files are contained in the /awips/GFESuite/primary/data/logfiles/yyyymmdd directory on px1 and provide clues if element grids do not get created. 

    The following figure shows the various SmartInit files created on this day.

    Logfile Directory from px1 Showing Various SmartInit Log Files

    For example, consider the following entry in an MM5 log file:

    "Exception: Unknown PID: pc_SFC:SEW_GRID_D2D_MM5_20020829_0000".

    This entry means there isn't a field named pc_SFC for the MM5, although there may be a p_SFC field.

    If you are receiving a local or baseline model in AWIPS, but, the data are not getting into any GFE D2D, check the D2DDIRS entry in you localConfig.py file and look at the related netCDF file to see if the header information is present.  If these are correct, there has to be some explanation why the ifpServer is not recognizing your file as a valid D2D grid file.

    Look in the ifpServer log files searching for the model name and see if there is any evidence the server tried to process it.  Hopefully, there is an error message indicating the problem.

    The following figure shows sample entries in the locally modified MesoEta SmartInit log file.

    Example of MyMesoEta SmartInit Log File

    This log file contains detailesysinformation about the many fields calculated ranging from the standard Wind, T, Td, Sky and Wx fields to added fields such as various thicknesses to severe weather parameters such as SWEAT, K, TT, EHI, CAPE and DCI

    To obtain more information in the ifpServer log file on why the ifpServer does not consider the file valid, add the following line to your etc/BASE/ifpServer.logPref file.

    all D2DFile.C tty debug = on

    You will then need to restart the server.

    You will need to comment this out after finding the problem because the ifpServer finds invalid files such as the template ones on a routine basis. 

    If the SmartInit isn't running on its own, but, does get into GFE when manually run by the ifpInit command, there should be a log file in the logfiles directory with the name of your SmartInit.  If ifpInit tried and failed there should be some clues left behind.

    Possible solutions may be:

    1. One of the grids is not available in the netCDF file.  This would cause ifpInit to never fire.

    2. Your localConfig is not set up with the databases named properly.  This is a very common problem when setting up ifpInit for a new model.

    For example, the localConfig.py file init INITMODULES entry would look something like this:

    import serverConfig

    serverConfig.INITMODULES['yourWsEtaInitFileName'] = ['TheD2DWsEtaDatabaseName']

    Remember not to include the .py extension to your init file name.  The D2D database name should be the one you see in the GFE's Volume Browser when loading the D2D version of the model.

    The following figures show specific model log file entries for the GWW in the ifpServer log file.

    GWW Entries in ifpServer Log File

    The following figures show specific model log file entries for the locally modified MesoEta in the ifpServer log file.

    MesoEta Entries in ifpServer Log File

    The following figures show specific model log file entries for the MRF in the ifpServer log file.

    MRF Entries in ifpServer Log File

    The following figures show specific model log file entries for the locally modified RUC in the ifpServer log file.

    RUC Entries in ifpServer Log File

    In each of these cases, you can see the starting of IfpInit for each model as new data arrives in AWIPS, the creation of new databases, the deleting of old database and the skipping of the RUC SmartInits specified in the serverConfig.py file.

    Question

    Which of the following statements are correct about the logfiles?

    A.When a SmartInit runs, it creates a logfile with the same name. 

    B.You can modify the ifpServer.logPref file to obtain additional information on why the ifpServer considers a file invalid.

    C.The logfiles should provide clues as to why element grids do not get created.

    D.All of the above statements are correct.

     

    Question

    The most common problems causing ifpInit to not run is missing grids in the netCDF files and incorrect naming of the databases in your localConfig.py file.

    True False

     

    8.2. Using ncdump to Diagnose Problems

    The program ncdump can be used to view the contents of netCDF files.  Information such as dimension names and sizes, variable names, types and shapes, attribute names and values and the values of data for all variables or selected variables in a netCDF file can be obtained.  This information can be used to diagnose SmartInit problems.  Optional switches are:

    -c This is the most suitable option to use for a brief look at the structure and contents of a netCDF file.  It shows the values of coordinate variables (variables that are also dimensions) as well as the declarations of all dimensions, variables, and attribute values.

    -h Show only the header information in the output such as the declarations of dimensions, variables, and attributes, but, no data values for any variables.  The output is identical to using the -c option except the values of coordinate variables are not included.

    -v var1, ..., varn The output will include data values for the specified variables, in addition to the declarations of all dimensions, variables and attributes.  One or more variables must be specified by name in the comma-delimited list following this option.  The list must be a single argument to the command, hence, cannot contain blanks or other white space characters.  The named variables must be valid netCDF variables in the input file.  The default listing without this option, and in the absence of the -c or -h options, is to include data values for all variables in the output.

    For example,

    To look at the structure of the data in the netCDF file MesoEta, you need to go to the MesoEta directory.  It is located in the /data/fxa/Grid/SBN/netCDF/CONUS212 directory.  In the MesoEta directory is a listing of the latest file in the yyyymmdd_hhmm format.  Type the following using the date format for the file name.

    ncdump -c yyyymmdd_hhmm where yyyymmdd_hhmm is the name of the file.

    To output data for only the variable tLevels for the MesoEta:

    ncdump -v tLevels yyyymmdd_hhmm where yyyymmdd_hhmm is the name of the file in the MesoEta directory.

    Weather Element not in D2D Model File.

    If you add a new model such as the MM5 and substitute the initialization algorithms from another model, some of the new fields may not get created because the calc functions are referring to D2D weather element(s) that do not exist in the D2D MM5 netCDF file. There should be some error messages in the MM5 ifpInit log file complaining about variables that can't be found such as the following entry:

    Exception: Unknown PID: pop12c1_SFC:SEW_GRID_D2D_MM5ensprob12km_20040324_0000

    This error message from the MM5 SmartInit indicates the "pop12c1" weather element is not in the D2D MM5* file, yet SmartInit is trying to refer to it. You can verify if this element is really there by using the ncdump program on the MM5 netCDF file by typing the following:

    ncdump -h 20040324_0000 where 20040324_0000 is the name of the netCDF file in the MM5 directory.

    Compare the resulting list to what the ifpInit routine is expecting.  If it isn't there, then you will need to change the SmartInit algorithms to not use "pop12c1".

    Listing of Model levels.

    If you run the following command on your netCDF files (ncdump is included with the GFE):

    ncdump -v tLevels yourFileName

    This should print out something like this:

    [snip]

    data:

     tLevels =
      "FHAG 2    ",
      "MB 1000   ",
      "MB 950    ",
      "MB 900    ",
      "MB 850    ",
      "MB 800    ",
      "MB 750    ",
      "MB 700    ",
      "MB 650    ",
      "MB 600    ",
      "MB 550    ",

    		"MB 500    ",
      "MB 450    ",
      "MB 400    ",
      "MB 350    ",
      "MB 300    ",
      "MB 250    ",
      "MB 200    ",
      "MB 150    ",
      "MB 100    ",
      "BL 0 30   ",
      "BL 30 60  ",
      "BL 60 90  ",
      "BL 90 120 ",
      "BL 120 150" ;
    }

    If you see a guidance field (T) defined at an unusual level such as "1HAG2", instead of "FHAG2", as specified in the netCDF template file shown above, the problem may be internal to GFE or an AWIPS issue.

    The GFE uses the values/names found in the netCDF file. If the netCDF template needs to be changed, this is an awips issue.  If the GFE init file needs to change, this is a GFE issue). It is probably easier to change the GFE name.

     

    Question

    Which of the following statements are correct about the use of the ncdump program?

    A.The -c switch is useful to look at the structure and contents of a netCDF file. 

    B.The -v switch is useful to look at specific variables such as the levels for which temperature are available.

    C.The entry after the switch is the name of the netCDF file and not the model name.

    D.All of the above statements are correct.

     

     

    8.3. Specific Troubleshooting Examples.

    Blank Grids After Populate.

    As mentioned previously, if you try to populate the Fcst database with some model data such as the AVN and there are some blank grids, the problem may be in missing D2D model grids.  If a weather element is not being calculated and there are no syntax or programming errors in your SmartInit file, one of the grids is likely missing.

    For example, the T, Td, and Wx grids are all dependent on the following grids from the AVN D2D model in AWIPS:

    t for levels ["MB850", "MB700", "MB500", "MB400", "MB300"]
    gh for levels ["MB850", "MB700", "MB500", "MB400", "MB300"]

    If any one of these grids is missing then ifpInit will not generate a T grid for that time. The ifpInit program refuses to just make up data, so, it needs to wait until all of it is in.  Both Td, and Wx are dependent on the T grid, so, if there is no T grid, there will be no Td or Wx grids. Check and see if these t and gh D2D grids are all there for the time you are missing grids.  Likely, one of them will be missing.

    You need to check to make sure your new levels actually have data in them. It has been know to happen for AWIPS data files to set aside space for levels but never actually populate them (like the 1000MB AVN).

    You need to make sure all the grids declared in ifpInit methods are available.

    In the past, there has been some confusion on the model parameters available in AWIPS. It is important to realize the contents of each model can vary.  If there is a p_SFC parameter for the Eta, that doesn't mean there will be same parameter in a different model.  You can see what parameters are available, and more importantly the naming convention,  by going to the Weather Element Browser, selecting D2D and the various models.  Experiment with the different fields and the levels to find all of the combinations available.  The biggest problem with the AVN SmartInit is it simply doesn't have enough vertical resolution and does not have the full complement of fields sent on the AWIPS SBN to produce good results compared to the levels available in the actual model.

    If there's an error in the MODELNAME.py file, an entry should be in the associated ifpInit log file.

    Model SmartInit Stops after AWIPS Upgrade.

    If a SmartInit such as MSAS suddenly stops working after an upgrade to AWIPS, the problem is likely related to the data ingest in AWIPS D2D changing the model definition in the MSAS netCDF file.  This causes previous definitions for INITMODULES and localConfig.py to be invalid and therefore need to be changed.

    Check the model variable name in the netCDF file using the ncdump command described in Section 8.2 above. 

    In the case of the MSAS, the steps would be:

    Step 1: Go to the /usr/local/netcdf/bin directory on any lx.

    Step2: Type: ncdump -v model /data/fxa/Grid/FSL/netCDF/MSAS/template 

    There are two places where changes may be needed:

    1. Your localConfig.py file - Check the INITMODULES definition.  If it contains something like "MAPSNationalSurfaceAnalaysis", change it to "MSAS".

    2. Your local MyMSAS.py (or the BASE MSAS SmartInit module) file - Check the source database.  If it indicates something like  "MAPSNationalSurfaceAnalaysis", change it to "MSAS".

    Global name 'topo' is not defined Error.

    The following MSAS Init script experienced a problem with the CalcT method:

    #########################################################################
    from Init import *

    class MSASForecaster(Forecaster):
    def __init__(self):
    Forecaster.__init__(self, "MAPSNationalSurfaceAnalysis", "MSAS")

    # You may see small errors with T particularly in high terrain
    # Need a better eqn for T
    def calcT(self, pot_SFC):
    pot_f = self.KtoF(pot_SFC)
    p = -0.4 * topo + 1013
    return pot_f * power((p/1013), .287)

    ....Snip....

    def main():
    MSASForecaster().run()

    if __name__ == "__main__":
    main()

    When the calcT algorithm crashed the following error with the topo variable was written to the MSASInit log file:

    13:17:59.218 Init.py 670 EVENT: Calc: T (20021204_1200, 20021204_1300)
    13:17:59.244 Init.py 670 PROBLEM: Error while running method T
    Traceback (most recent call last):
    File "/awips/GFESuite/etc/BASE/Init.py", line 680, in __runMethod
    rval = apply(mthd, tuple(gargs))
    File "/awips/GFESuite/etc/SITE/MSAS.py", line 11, in calcT
    p = -0.4 * topo + 1013
    NameError: global name 'topo' is not defined

    In this case, the topo field is a local variable within the calcT method (i.e. there is no 'self.' in front of it). The topo needs to be included as a method argument like this:

    def calcT(self, pot_SFC, topo):

    No database Error.

    Using the following command to run the ifpInit for the workstation Eta:

    ifpInit -t 20030326_1200 -a WSETA

    Produced the following error:

    "Init.py 270 PROBLEM: no databases for WSETA"

    Diagnostic procedures revealed the following:

    1. There is a netCDF file in AWIPS corresponding to the date.

    2. The only thing edited is the WSETA.py init file to adjust some field calculations.

    3. The localConfig.py was not been touched recently.

    4. It was working yesterday.

    The ifpInit message "No databases for ..." mean SmartInit can't find the databases. You should verify the following:

    1. Can you see the WSETA grids from the GFE?

    From the Weather Element Browser select D2D and see if that database and those grids can be viewed. If you can't see them, there's something different about the WSETA model file. It's likely your localConfig.py file will need to be updated.

    2. If you can see them, then something is wrong with your WSETA.py module. Perhaps some name changed on the AWIPS side? What changed since yesterday.

    If the netCDF file does not have the right geographical information in it, then the ifpServer won't take the database as valid (even if D2D can display it).

    Another way to quickly see what databases ifpServer is seeing, is to run the ifpServerStats -d program, the D2D databases are clearly shown.

    If something worked yesterday but doesn't today, the geographic info may be bad, or a new D2D netCDF template file installed which might have a different "model" variable, which is used to define the model name seen in GFE.

    Unknown PID Error.

    PID stands for Parm Identification.  It consists of:

    1. A weather element name (e.g., T).

    2. A level (e.g., SFC).

    3. A database identifier of the form BOU_GRID__Fcst_00000000_0000 which is decoded into:

    Consider the following Init log file error messages:

    22:24:53.695 Init.py 866 BUG: Caught exception
    Traceback (most recent call last):
    File "/home/gfe/release/etc/BASE/Init.py", line 875, in main
    mod.main()
    File "/home/gfe/release/etc/SITE/LOXMM8.py", line 355, in main
    LOXMM8Forecaster().run()
    File "/home/gfe/release/etc/BASE/Init.py", line 342, in run
    times = self.__sortTimes(methods)
    File "/home/gfe/release/etc/BASE/Init.py", line 604, in __sortTimes
    ttimes.append(self.__getSrcWE(p).keys())
    File "/home/gfe/release/etc/BASE/ifpc.py", line 11, in keys
    val = ifpcc.IFPWE_keys(self.this)
    Exception: Unknown PID: pc_SFC:LOX_GRID_D2D_MM5_20021006_1200

    The above error is stating the Init process is trying to access the weather element "pc" for the SFC and can't find it.

    In this case, you need to do an ncdump -h of the model netCDF file and it will show you what parameters are available.

    Consider the following scenario:

    When the ifpInit for the WsEta is run, there is the following error message in the log file:

    File "/awips/GFESuite/primary/etc/BASE/ifpc.py", line 39 in keys
    def keys(*args): return apply(_ifps.IFPWE_keys,args) "Exception: Unknown PID: QPF_SFC:ILM_GRID_D2D_WSETA_20040317_0000"

    The WsEta is in the Weather Element Browser and as an option for populating a grid, but there is the message "no grids to copy" when trying to populate with it. In the Weather Element Browser, the name is called WSETA.

    In this case, the problem lies with the ifpInit script.  The line defining the D2D database and the string for the IFP database is likely incorrect:
    The line should look like the following:
    Forecaster.__init__(self, "WSETA", "WsEta")

    Where "WSETA" is the name of the D2D database and the "WsEta" string is for the IFP database.

    You also need to make sure you have defined the 'WsEta' database in your localConfig.py.  If your ifpInit line looks like the above line, then, the problem is probably in your localConfig.py.

    Overflow Error.

    Consider this scenario:

    In attempting to provide DGEX (Downscaled GFS with Eta Extension) output in GFE, the output is viewable in AWIPS, and DGEX shows up both under the WeatherElement and Populate menus in GFE.  However, it appears SmartInit is crashing because the MyDGEXInit log file in the /awips/GFESuite/primary/data directory on px1 contains the following:

    [snip]

    "/awips/GFESuite/primary/etc/SITE/MyDGEX.py", line 790, in RHDP lvapr=log(rh*vaps/100.0) OverflowError: math range error

    In this case, numeric pythons is telling you the log() function is overflowing (or creating a number too big to store in a 32 bit float).  It is likely you are attempting to take the log of zero, which is not defined.  This could happen if either your rh or vaps is 0.  The usual fix for this is to add a very small amount to the value being logged.  The value should be small enough to make no difference to your calculation (the result is valid using the precision you are looking for), and large enough to make sure you are not taking the log of zero.  Something like this:

    lvapr=log(rh*vaps/100.0 + 0.0001)

    When a calc method fails, the grid it was supposed to create is set to the python None object.  The above error was in the calculation of T, so, any other calc method which needs T will then also fail because it is now getting a None instead of a T grid.  Once you fix your log() call the other errors should also be fixed.

    MesoEta Stopped being Generated.

    Consider the case of the MesoEta databases have stopped being generated in GFE.

    The MesoEta data are divided into two separate files. One holds the boundary layer and the other holds the upper air. In the GFE Volume Browser, these show up as MESOETAU and MESOETAS.

    Once in a while when the server first starts, one of these files may be in but the other is not.  This is because just one of the database files has been seen by the server and it tries to run the init before the other one has been seen. Why is one of these files missing?

    1. Check and see if the GFE Volume Browser has a MESOETAU and MESOETAS database (both for the same time).  Likely, one is missing. If this is the case, then can you see the MesoEta data in D2D? If so, perhaps the data is in a directory the GFE is not expecting. The server configuration parameter D2DDIRS lists out the places where the ifpServer will look for files.

    2. Check to see if there any files in these directories for the MesoEta.

    3. Check the ifpServer log file.

    Consider the following entry:

    17:55:49.064 D2DFile.C 62 DEBUG: Invalid D2D file:
    /data/fxa/Grid/SBN/netCDF/CONUS212/MesoEta/20020730_1200 :
    unknown projection: --------------------------------

    The CONUS212 directory holds the upper air portion of MesoEta. Somehow the projName attribute has been changed from LAMBERT_CONFORMAL to the long sequence of dashes. Since the IFP server has no idea what kind of projection a long series of dashes is, it can not use the model.

    In this case, the /data/fxa/Grid/SBN/netCDF/CONUS212/MesoEta/template file has become corrupted. If you run:

    ncdump -h /data/fxa/Grid/SBN/netCDF/CONUS212/MesoEta/template

    You should see something like this at the bottom of the printout:

    // global attributes:
    :cdlDate = "20020228" ;
    :depictorName = "FSL_mEtaClip.sup@17085437" ;
    :projIndex = 3 ;
    :projName = "LAMBERT_CONFORMAL" ;
    :centralLat = 25.f ;
    :centralLon = -95.f ;

    The important bit is projName LAMBERT_CONFORMAL. If this part is correct, the problem may be in the projection information on D2D. D2D doesn't use this information but GFE does. 

    Go to /data/fxa/Grid/SBN/netCDF/CONUS212/MesoEta and do the following: 

    ncdump -h yyyymmdd_hhmm

    It should look similar to the following near the end of the dump: 

    // global attributes: 
    :cdlDate = "20020228" ; 
    :depictorName = "FSL_mEtaClip.sup@17085437" ; 
    :projIndex = 3 ; 
    :projName = "LAMBERT_CONFORMAL" ; 
    :centralLat = 25.f ; 
    :centralLon = -95.f ; 
    :rotation = 25.f ; 
    :xMin = -0.1991076f ; 
    :xMax = 0.01474735f ; 
    :yMax = -0.6187723f ; 
    :yMin = -0.8301724f ; 
    :lat00 = 20.95828f ; 
    :lon00 = -126.913f ; 
    :latNxNy = 54.36781f ; 
    :lonNxNy = -91.76945f ; 
    :dxKm = 39.30751f ; 
    :dyKm = 39.30811f ; 
    :latDxDy = 39.14964f ; 
    :lonDxDy = -112.1578f ; 

    These entries are probably either missing or assigned to 0. 

    If this is the case, then you will need to do a grid localization for the D2D system.

    No MSAS in GFE.

    Consider the following scenario of having problems getting MSAS into GFE.

    1. The localConfig.py file contains the following:

    It is pointing to the right D2D directory:

    /data/fxa/Grid/FSL/netCDF/MSAS 

    and these definitions: 

    MSAS = ('MSAS', GRID, '', YES, NO, 1, 36) 
    MSAS_PARMS = [([Temp, Td, Wind], TC1)] 
    dbs = [(MSAS, MSAS_PARMS)]... among the other models located on this line 
    serverConfig.INITMODULES["MSAS"] = ["MAPSNationalSurfaceAnalysis"] 
    serverConfig.D2DDBVERSIONS["MAPSNationalSurfaceAnalysis"] = 24 

    2. The MSAS.py file looks like this: 

    from Init import * 
    class MSASForecaster(Forecaster): 
    def __init__(self): 
    Forecaster.__init__(self, "MAPSNationalSurfaceAnalysis", "MSAS") 

    ...<snip>...

    def main(): 
    MSASForecaster().run() 

    if __name__ == "__main__": 
    main() 

    3. When "ifpInit -a MSAS" is run, the error message in the MSASInit file says no databases found.

    SmartInit cannot find the D2D netCDF datasource. In the MSAS.py file, it is called "MAPSNationalSurfaceAnalysis".  On AWIPS, it is called MSAS. 

    In this case, you need to change the following two lines in localConfig.py to look like the following: 

    serverConfig.INITMODULES["MSAS"] = ["MAPS"] 
    serverConfig.D2DDBVERSIONS["MAPS"] = 24 

    and change this line in your MSAS.py file: 

    Forecaster.__init__(self, "MSAS", "MSAS")

    This is verified by looking in GFE under Weather Element in the top menu bar, selecting Weather Element Browser, select D2D, pull down listing and you should see MSAS and not MAPSNationalSurfaceAnalysis for the model name. 

    You can also go to your directory where the MAPS data files are located in D2D and use the following command to look for the "model" variable: 

    /usr/local/netcdf/bin/ncdump -v model template

     

    Question

    Which of the following statements are correct about troubleshooting problems with the SmartInit?

    A.After populating your Fcst database with model data, there are some blank grids.  The likely problem may be missing D2D model grids specified in the SmartInit.

    B.A common problem causing a SmartInit to not run is the model variable name in the netCDF file is not the same as the name in your localConfig file INITMODULES definition.

    C.A common problem is the netCDF file does not contain the right geographical information in it, so, the ifpServer won't take the database as valid (even if D2D can display it).

    D.All of the above statements are correct.

     

    Question

    An unknown PID error in a SmartInit log file means the SmartInit is trying to access an unknown weather element.

    True False

     

    9. Summary.

    The ifpServer is aware of D2D model data changes and therefore knows when IFP data grids can be created.  The IFP data are the surface derived grids according to the definition in the BASE serverConfig.py file and the unique grids you may have defined in your localConfig.py file.

    When the ifpServer first starts, or new D2D model data is detected, the ifpServer will examine the netCDF files inventory and the serverConfig/localConfig configuration files for the INITMODULES definition. If it finds a match with the newly arrived D2D model, it will spawn a process to run the appropriate model SmartInit which calculates the new IFPS grids from the model data. It then goes to sleep for 241 seconds (4 minutes and 1 second) and repeats the process.

    It is also possible there may be a time period you may see a partially populated database. Some grid element calculations require several grids. If you try to populate the Fcst database with some model data such as the AVN and there are some blank grids, the problem may be in missing D2D model grids.

    Each algorithm to derive surface sensible weather elements is coded in Python and can be modified.

    When a SmartInit process is started, it will run a particular class file, either one supplied with the release, or one you have added via a local SmartInit.  The software will examine all of the functions within that class looking for function names beginning with calc***.  These functions define the output weather element, e.g., calcT will derive T, and also the dependencies.  The dependencies are determined from the argument list for each function.

    The SmartInit software then discerns the dependencies for all of the calc*** functions and will determine what must run first, second, third, etc.  Since this is done automatically, the person making changes to the SmartInit in BASE need not be concerned about it.  The software also examines the inventory for the dependent grids and the inventory for the output grids to determine if the output grid is already present and no calculations are needed, or if the output grid needs to be created by running the algorithms.  When all the possible dependencies and algorithms have run, the process exits until the next D2D model update.

    The baseline SmartInit files are on px1 in the /awips/GFESuite/primary/etc/BASE directory.  As you should well know, no modifications should be made to any of the files in the BASE directory. User-customization to the baseline SmartInit files are placed in the /awips/GFESuite/primary/etc/SITE directory. As the case throughout IFPS, the SITE  files are not overwritten during installs and upgrades.

    The basic structure of a SmartInit file contains several parts:

    1. The header.

    It contains the following parts:

    from Init import *

    class modelForecaster(derivedFromForecaster):
        def __init__(self):
            derivedFromForecaster.__init__(self, "sourceDb", "destDb")

    This is basically defining a new class called "modelForecaster", which is derived from "derivedFromForecaster".  The __init__ function is the constructor to the class, which calls the base class (class it has been defined from) with three arguments, self, "sourceDb", and "destDb".  The source db is the name of the D2D database, such as ETA, NGM, MRF.  The destination database is the name of the output database, such as Eta, MRF, AVN.

    2. Usually a function called levels().

    This defines a set of vertical pressure levels used when accessing cube data from the models.  These levels are used to create vertical soundings.  The levels() return a list of model dependent level values.  When you try to incorporate additional model levels into your SmartInit, you need to make sure these extra levels actually contain grids.  In AWIPS and IFPS, the number of model defined grids vary from the actual number created by the model at NCEP because not all grids are not being sent over the SBN for some models. A good way to check this is to use your GFE Weather Element Browser to verify all of the levels you have defined contain grids.

    3. The function.

    A function consists of a header line, followed by a block of indented statements.  The indented statements become the function's body; the code Python executes each time the function is called.  You use the def statement to create a function.  The header specifies a function name along with a list of arguments (sometimes called parameters), which are assigned to the objects passed in parentheses at the point of call in the form:

    def <name>(arg1, arg2, ...argN):
          <statement>
         return <value>

    SmartInit commonly uses the calc function which MUST BEGIN WITH calc in it's name.  The remainder of the name of the function after "calc" is the parameter name to create.  For example, if RH is your weather element name, the name of the function to calculate RH would be calcRH().

    4. The tail end of the SmartInit file.

    It contains a definition of main() and must be similar to the following:

    def main():
        modelForecaster().run()

    if __name__ == "__main__":
        main()

    To override entries in the BASE SmartInits or add a new model SmartInit, you need to create a new file in the /awips/GFESuite/primary/etc/SITE directory on px1, named similarly, but not identical to, the class you are modifying, make your changes using the standard format structure, and modify (create first if necessary) your server localConfig.py file to override the INITMODULES section of the serverConfig.py.

    Your IFP database can be called anything.  To simplify things, it should be called by the model name such as MM5. It is the destination db from SmartInit and the one containing the parms you have defined.  The source db doesn't need to be defined, but, is what D2DDIRS points to.  The model variable in the netCDF file defines the name of the source db.  It is recommended you name your SmartInit module to be the model name such as MM5.py and name your variables to match the model name which is listed under Source in the IFPS Weather Element Browser such as the MM5 is based on what is coded in the D2D netCDF file located in the /data/fxa/Grid/SBN/netCDF/.... directory.  It is this model name ifpInit needs to know.  The constructor contains the source and destination database names. The source name is what is in the "model" netCDF variable.  The destination database anything, but, to be conventional, call it the same name.

    Most problems related to locally modified or newly created SmartInits can be traced to incorrect naming.  When a SmartInit is run, it creates a log file with the same name (For example, MyMesoEta). These log files are contained in the /awips/GFESuite/primary/data/logfiles/yyyymmdd directory on px1 and provide clues if element grids do not get created.

    Although you should never need to run the ifpInit manually, since the ifpServer monitors the received data on D2D and automatically queues SmartInit to run, you can run SmartInit from the command line, when you are developing new algorithms or modifying existing ones before you add the SmartInit to your localConfig.py file.

    There are three serverConfig variables, INITMODULES, INITSKIPS, and D2DDIRS, in your serverConfig or localConfig controlling the operation of the various SmartInit programs.  The INITMODULES defines your IFP databases and which model is used to generate them.  To get the server to run the model SmartInit automatically, add the model to INITMODULES.  If you don't want to initialize certain model runs, you can use the INITSKIPS feature.  The D2DDIRS definition in your localConfig file determines which "D2D" databases show up in your GFE.

    The program ncdump can be used to view the contents of netCDF files.  Information such as dimension names and sizes, variable names, types and shapes, attribute names and values and the values of data for all variables or selected variables in a netCDF file can be obtained.  This information can be used to diagnose SmartInit problems.

    10. Review Questions.

    Question 1

    Which of the following statements are true about the relationship between the ifpServer and SmartInit?

    A.About every four minutes, the ifpServer will look for new model data and then will spawn a process to run SmartInit for that model. 

    B.It is possible for new model data to come in, but, not get processed immediately since the ifpServer has not detected it.

    C.It is possible there may be a time period you may see a partially populated database because some elements require other elements or levels which may not have come in at the same time.

    D.All of the above statements are correct.

     

    Question 2

    All of the SmartInits are started by cron.

    True False

     

    Question 3

    The algorithms to derive surface sensible weather elements are "hard-coded" and can not be changed by the user.

    True False

     

    Question 4

    Which of the following statements are true about the header part of the SmartInit file?

    A.The source db is the name of the D2D database, such as ETA, NGM, MRF. 

    B.The destination database is the name of the output database, such as Eta, MRF, AVN.

    C.Your ifpInit script needs to be consistent with the naming convention in your Weather Element Browser.

    D.If the destDb argument is not given, then it defaults to the same name as the sourceDb

    E.All of the above statements are correct.

     

    Question 5

    Which of the following statements are true about the level() function in the SmartInit file?

    A.The levels() function defines a set of vertical pressure levels. 

    B.If a grid is missing for a level, SmartInit will not run the calculation.

    C.You can verify all of the levels you have defined contain grids via the Weather Element Browser .

    D.All of the above statements are correct.

     

    Question 6

    Which of the following statements are true about the function set in the SmartInit file?

    A.You use the def statement to create a function. 

    B.The calculation function must begin with calc in it's name, followed by parameter name to create such as Sky.

    C.The first argument for the calc*** functions is always self with the remainder of the arguments representing gridded data..

    D.All of the above statements are correct.

     

    Question 7

    When a calc method fails, the grid it was supposed to create is set to the python None object, so, any other calc method which needs T will then also fail because it is now getting a None instead of a T grid.

    True False

     

    Question 8

    The name of the class within the main() function does not need to  match the name of the class you have defined in the header.

    True False

     

    Question 9

    Which of the following statements are true about accessing scalar, vector, and weather data through numerical python?

    A.Discrete and Weather are more complicated than the simple scalar and vector cases, so, they can cause a big performance problem. 

    B.When you access a single vector grid (at a single level), you get a tuple where the first element is a grid of magnitude and the second is a grid of direction.

    C.To access either weather or discrete, you get a tuple where the first element is a grid, which contains the indexes into the key and the second element is a sequence of all of the keys.

    D.All of the above statements are correct.

     

    Question 10

    Which of the following statements are true about modifying entries in SmartInit files?

    A.The contents and format of the file should follow the standard structure containing at least a header, the replacement function, and a tail. 

    B.If you are modifying an existing function, the name of the function such as calcSnowAmt(), needs to be identical to the name in the original file.

    C.The name of the class within the main() function in the tail must match the name of the class you have defined in the header.

    D.All of the above statements are correct.

     

    Question 11

    Which of the following statements are true about the proper syntax for ifpInit?

    A.If the -t switch is not specified, then the latest model data are used. 

    B.If the -a switch is not specified, only grids that haven't been created will be calculated.

    C.Generally, the -h (host) and -p (port) switches are not needed.

    D.All of the above statements are correct.

     

    Question 12

    You only need to specify ifpInit and the algorithm file name to run ifpInit.

    True False

     

    Question 13

    Which of the following statements are correct about the INITMODULES entry in the local or serverConfig files?

    A.This entry causes the SmartInit to run automatically as new data flow into AWIPS . 

    B.INITMODULES requires two entries; the database name and the name of the ifpInit script to run.

    C.You can use INITMODULES to rename a SmartInit (del entry).

    D.All of the above statements are correct.

     

    Question 14

    Which of the following statements are correct about the logfiles?

    A.When a SmartInit runs, it creates a logfile with the same name. 

    B.You can modify the ifpServer.logPref file to obtain additional information on why the ifpServer considers a file invalid.

    C.The logfiles should provide clues as to why element grids do not get created.

    D.All of the above statements are correct.

     

    Question 15

    The most common problems causing ifpInit to not run is missing grids in the netCDF files and incorrect naming of the databases in your localConfig.py file.

    True False

     

    Question 16

    Which of the following statements are correct about your SmartInit naming conventions?

    A.Your IFP database can be called anything.  To simplify things, it should be called the model name such as MM5. 

    B.The source database doesn't need to be defined, but, is what D2DDIRS points to.

    C.The model variable in the netCDF file defines the name of the source database.

    D.All of the above statements are correct.

     

    Question 17

    The model name listed under Source in the IFPS Weather Element Browser such as the MM5 is based on what is coded in the D2D netCDF file located in the /data/fxa/Grid/SBN/netCDF/.... directory.  It is this model name ifpInit needs to know.

    True False

     

    Question 18

    You can run any SmartInit, baseline or locally defined, as any user.

    True False

     

    Question 19

    Which of the following statements are correct about the use of the ncdump program?

    A.The -c switch is useful to look at the structure and contents of a netCDF file. 

    B.The -v switch is useful to look at specific variables such as the levels for which temperature are available.

    C.The entry after the switch is the name of the netCDF file and not the model name.

    D.All of the above statements are correct.

     

    Question 20

    Which of the following statements are correct about troubleshooting problems with the SmartInit?

    A.After populating your Fcst database with model data, there are some blank grids.  The likely problem may be missing D2D model grids specified in the SmartInit.

    B.A common problem causing a SmartInit to not run is the model variable name in the netCDF file is not the same as the name in your localConfig file INITMODULES definition.

    C.A common problem is the netCDF file does not contain the right geographical information in it, so, the ifpServer won't take the database as valid (even if D2D can display it).

    D.All of the above statements are correct.

     

    Question 21

    An unknown PID error in a SmartInit log file means the SmartInit is trying to access an unknown weather element.

    True False

     

    11. Appendix.

    Here is a copy of the local MesoEta SmartInit created at the Training Center.  It creates a new sky initialization algorithm, several severe weather parameters and different thickness layers.

    The following are examples of some specific initialization of elements such as Sky, SnowAmt and QPF.

    Example Sky Inits:

    ##--------------------------------------------------------------------------
    ##  Calculates Sky condition from the height and RH cubes.
    ##  This tools adds up the amount of clouds calculated at each layer
    ##  and subtracts  the amount of clear sky still available.
    ##--------------------------------------------------------------------------
        def calcSky(self, rh_MB850, rh_MB700, rh_MB500, rh_MB1000):
           
            RH1 = rh_MB850 # 850 rh
            RH2 = rh_MB700 # 700 rh
            RH3 = rh_MB500 # 500 rh
            RH4 = rh_MB1000 # 1000 rh

            sky = ((RH1+RH2+RH3+RH4)/4)
            return sky

    ##--------------------------------------------------------------------------
    ##  Calculates Sky condition from the height and RH cubes.
    ##  This tools adds up the amount of clouds calculated at each layer
    ##  and subtracts  the amount of clear sky still available.
    ##--------------------------------------------------------------------------

        def calcSky(self, gh_c, t_c,dpt_c,topo):
            t_c   = t_c-273.15
            dpt_c = dpt_c-273.15

            Vt = 6.11 * pow(10,(t_c * 7.5 / (t_c + 237.3)))
            Vd = 6.11 * pow(10,(dpt_c * 7.5 / (dpt_c + 237.3)))

            rh_c = (Vd / Vt) * 100.0
           
            # only use the first levels (up to MB650)
            gh_c = gh_c[3:14,:,:]
            rh_c = rh_c[3:14,:,:]
            rh_c = where(less(gh_c, topo), 0, rh_c)
            rh_c = where(less(rh_c, 50), 0, rh_c)
            # calculate the areas where rh is 50-70%, 70-85% and 85-100%
            m1 = logical_and(greater(rh_c, 50), less(rh_c, 70))
            m2 = logical_and(greater(rh_c, 70), less(rh_c, 85))
            m3 = greater(rh_c, 85)
            ## calculate the amount of cloud contribution of each layer
            skylayer = where(m1, self.linear(50, 70, 0, 25, rh_c),
                             where(m2, self.linear(70, 100, 25, 100, rh_c),
                                   where(m3, 100, rh_c)))

            skylayer = skylayer / 100
            sky = skylayer[0]
            ## for each layer, subtract the amount of clear sky it obscures
            for i in xrange(1, skylayer.shape[0]):
                sky = sky + skylayer[i] - sky * skylayer[i]

            sky = clip(sky, 0, 1)
            sky = sky * 100
            return sky

     
    Example of SnowAmt Inits:

    ##--------------------------------------------------------------------------
    ##  Calculates Snow amount based on the Temp, Freezing level, QPF,
    ##  topo and Weather grid
    ##--------------------------------------------------------------------------


        def calcSnowAmt(self, T, FzLevel, QPF, topo, Wx):
            # figure out the snow to liquid ratio
            m1 = less(T, 9)
            m2 = greater_equal(T, 30)
            snowr = T * -0.5 + 22.5
            snowr = where(m1, 20, snowr)
            snowr = where(m2, 0, snowr)
            # calc. snow amount based on the QPF and the ratio
            snowamt = where(less_equal(FzLevel - 1000, topo * 3.28),
                            snowr * QPF, 0)
            # Only make snow at points where the weather is snow
            snowmask = logical_or(equal(Wx[0], 1), equal(Wx[0], 3))
            snowamt = where(snowmask, snowamt, 0)
            return snowamt

    Example of QPF Inits:

    ##--------------------------------------------------------------------------
    ## Calculates QPF from the total precip field out of the model
    ##--------------------------------------------------------------------------

        def calcQPF(self, tp_SFC):
            qpf = tp_SFC / 25.4   # convert from millimeters to inches
            return qpf

    Example of Wind, Swell, and Period Inits for the ENP


##--------------------------------------------------------------------------
#
#  Wind direction and speed
#
##--------------------------------------------------------------------------

    def calcWind(self, wd_SFC, ws_SFC, CoastalWaters):

	mag = self.convertMsecToKts(ws_SFC)
	dir = wd_SFC

        mag = where(CoastalWaters, mag, 0)	        # Apply only the CoastalWaters edit area
	dir = where(CoastalWaters, dir, 0)

	return (mag, dir) # assemble speed and dir into a tuple

##--------------------------------------------------------------------------
#
#  Significant Wave Height
#
##--------------------------------------------------------------------------

    def calcWaveHeight(self, htsgw_SFC, CoastalWaters):

	# CoastalWaters - edit area covering the coastal waters

	waveheight = htsgw_SFC
        waveheight = clip (waveheight, 0 , 100)

	waveheight = where(CoastalWaters, waveheight, 0)

	return waveheight * 3.28 #convert to feet


##--------------------------------------------------------------------------
#
#  Swell height - primary swell (assume significant wave height is primary swell height)
#  Set swell direction to primary wave direction
#
##--------------------------------------------------------------------------

    def calcSwell(self, WaveHeight, Wind, dirpw_SFC, CoastalWaters):

	# Assume the significant wave height is composed only of the primary swell and wind waves
	#
	# Compute the wind wave, then remove from significant wave height to get primary swell magnitude


        wvhgtsq = WaveHeight * WaveHeight           # Square of the signigicant wave height
      
        wndwvhgt = greater((Wind[0] - 5),0) / 5     # Compute wind wave height

	wndwvhgtsq = wndwvhgt * wndwvhgt            # Square of the wind wave height

        swellhgtsq = wvhgtsq - wndwvhgtsq

        mag = sqrt(swellhgtsq)						# Swell magnitude

	dir = dirpw_SFC                             # Set swell direction to primary wave direction

	##dir = self._empty + 280                   # Apply initial swell direction - previous version
                                                    # of SmartInit for the ENP

        mag = where(CoastalWaters, mag, 0)          # Apply only to CoastalWaters edit area
	dir = where(CoastalWaters, dir, 0)

	return (mag, dir) # assemble height and dir into a tuple


##--------------------------------------------------------------------------
#
#  Primary swell period
#
##--------------------------------------------------------------------------

    def calcPeriod(self, perpw_SFC, CoastalWaters):

	period = perpw_SFC

	period = where(CoastalWaters, period, 0)   # Apply only to CoastalWaters edit area

  	return period