wiki:Rose/Rose Stem FAQ

Rose Stem FAQ


Frequently Asked Questions about Rose Stem

Contents

  1. General
    1. Q: What is Rose Stem?
    2. Q: Where is the platform-specific information for the rose-stem suite?
    3. Q: How do I compile my code with rose-stem?
    4. Q: How do I run a rose-stem test?
    5. Q: How do I run stem tests to test code changes for a component like OPS?
    6. Q: How do I make a new stem test to go with my source code changes?
    7. Q: What if my code changes cause stem tests to fail rose-ana?
    8. Q: The stem test controls are failing at the head of the trunk
    9. Q: How do I have a look at a processed suite definition which was produced using 'meto' as the value of SITE
  2. Issues
    1. Q: What does it mean if I run a task and it can't find the host?

General

Q: What is Rose Stem?

A: A means to test code changes against standard tests. Each source code also contains a rose-stem directory which contains the tests. These are run as cylc suites and tested against known good output (KGO) or control files.

Rose Stem Documentation

The UM rose-stem Suite for External Users

VAR & OPS Rose Stem Documentation

Q: Where is the platform-specific information for the rose-stem suite?

A: Under rose-stem/site/<SITE>/ are a series of directories containing suite*.rc files which contain site-specific configurations, environment variables, suite families and groups of tasks to run.

Q: How do I compile my code with rose-stem?

A: The rose-stem/app directory contains fcm_make_extract and fcm_make tasks that extract code from the various repositories, and compile the code. Note that the extract and build config is part of the source code under src/config.

To run, the command is:

rose stem -vv --task=fcm_make:ops_x86_64_ifort_opt_mirror

This specifies the configuration to use (the platform, compiler and optimization level) and in this case extract from a mirror. fcm_make is dependent on fcm_make_extract, so fcm_make_extract will run first, if it hasn't been run yet. Executing this command will result in a gcylc window opening and the fcm_make family of tasks running.

Q: How do I run a rose-stem test?

A: Like the rose-stem command used to do a build the command option, '--task' is used to specify the ID of the application to run and the FCM make config file to be used in the build.

To run, the command is:

rose stem -vv --task=<task_name>:<config_version> --define-suite=DEP_TASKS_COMPILE=false
e.g.
rose stem -vv --task=process_atms_glb:ops_x86_64_ifort_opt_mirror --define-suite=DEP_TASKS_COMPILE=false

To avoid running rose-ana tasks, you can use the following

rose stem --task=... -S DEP_TASKS_COMPARE_EXTERNAL=false

Q: How do I run stem tests to test code changes for a component like OPS?

A: Stem testing is intended to make sure you haven't broken anything when you modify the source code. You should always run all the relevant stem tests after any code changes, and check that the outputs haven't changed (unless they are supposed to). The outputs are tested against KGOs using rose-ana tasks, which exist along side the stem tests.

For UM testing, see The UM rose-stem Suite for External Users

For OPS stem testing, there are a range of apps to test ODB creation, and OPS processing of all the various observation types that are in use. These stem tests are run by the Met Office as well as by NCI, and include almost all stem tests with _bom_ in the name, as well as tasks with access_c in the name. You can run the full array of OPS stem tests by typing rose stem --task=bomall This will compile the code using the default config ops_x86_64_ifort_opt_mirror and run all the stem tests, comparing against KGOs. N.B. the KGOs are not always updated in sync with the trunk, so it's a good idea to check if the tests fail before making any code changes.

For SURF and VAR, stem tests are not set up and controlled. However, it is probably possible to run some of the existing stem tests as the input files usually exist on jasmin (if not in the jasmin mirror at NCI). Some modification of the source code might be required to run the stem tests.

Q: How do I make a new stem test to go with my source code changes?

A: When you add to the source code, sometimes you need to create a stem test to exercise that new part of the code. In that case, there should be a stem test to go with your code changes, when you submit them for review. Here's how to create a stem test.

  1. You should already have a ticket for your change, and a branch with the change.
  2. Take a look at the stem tests in the <your_branch>/rose-stem/app directory and see if there are any similar tasks (e.g. a createodb task or a process task)
  3. Create your test (maybe by copying from another test)
    • You should aim to have the input files in the common location under $DIR/Acceptance_Tests where $DIR may be $OPSIR or $VARDIR or $SURF_HOME. Check in rose-stem/site/nci/<component>/suite-env.rc
    • (optional) For portability, in the suite-app.rc, set the config_default with jinja thus {% set config_default = variables['default_<component>_config'] %} and make sure the default config is set in the suite-variables.rc file in rose-stem/site/nci/<common|component>/suite-variables.rc. This is standard in OPS but not SURF.
    • Additionally, you can add your test to suite-apps.rc (to give it an abbreviation), suite-groups.rc (to include it in a batch of tests)
  4. Run your test
  5. Copy results to a control dir and set up suite-controls.rc
  6. Set up a rose_ana task for your test (easiest is to copy another test)
  7. Retest your branch and make sure your new test succeeds and the rose-ana test succeeds (can find the controls)
  8. If necessary, get an ITO to copy the start files and controls to the appropriate directories.
  9. Put your start files on jasmin in the TIDS workspace, for the code owners to access
  10. Update your ticket with the test info and inputs location, changesets
  11. Submit your branch to review, assigning your ticket to a code owner

Q: What if my code changes cause stem tests to fail rose-ana?

A: Sometimes code changes will cause changes to the output. This may cause rose-ana to fail. If this happens, make sure your results are correct (i.e. the differences are intended.) Once you understand why the outputs have changed, update your ticket with this information.
You may need to update the controls for the stem tests, along with your code changes. In this case, get your new KGOs (normally the log and work dirs of your stem test) and copy them to the KGO location, into a new directory named appropriately (e.g. after your ticket number). Update the suite-controls.rc file to point to your new control.
When your code is reviewed and approved, the other partner(s) will also update their controls for that test.

Q: The stem test controls are failing at the head of the trunk

A: Often, code changes are made without information us that they will alter the KGOs for some stem tests. If it is known that they are tests run by specific partners, ideally the person making the changes will notify the partners, and a new control can be organized to be added at the same time as the changes are committed to the trunk. However for OPS the stem tests were only recently introduced and VAR and SURF are not set up, so this hasn't been happening. The end result is that our KGOs become out of date, and need to be updated. This is a description of how to update the controls once they start to fail.

  1. Create a ticket and cc at least one code owner, plus appropriate ITO (if you aren't one).
  2. Create a branch off the trunk.
  3. Compile and run the stem tests to see which ones fail rose-ana.
  4. At this point you might want to check why the tests are failing.
    • If you go to <your_branch>/rose-stem/site/meto/<component>/suite-controls.rc you can see which controls have been updated since the last release.
    • You may need to flick back through previous revisions to find all relevant changes
    • The Met Office controls include the ticket number for which the controls were updated, and you can check the ticket to see why the controls changed.
    • Alternatively, if you click on 'blame' at the top of the page, you can see the revision when the control changed. If you click on the revision number (right columm) you will get the change set, which usually includes the ticket number as well.
  5. Once you're satisfied which controls need to be updated, you need to copy the control files to the shared location. For OPS this is "/g/data/access/OPS/KGO/"
    • Create a new directory for the controls, which for example is named "control-t<ticket_number>", either the ticket number which altered the code, or your ticket to update the controls.
    • Into this directory copy the log and work directories, i.e. cp $CYLC_SUITE_DIR/log <your_control_dir>; cp $CYLC_SUITE_DIR/work <your_control_dir>
  6. Modify the <your_branch>/rose-stem/site/nci/<component>/suite-controls.rc to point to your control dir for the relevant tasks.
  7. Rerun the stem tests and check that no rose-anas fail.
  8. If necessary, get an ITO to copy the new controls to the correct directory, then repeat the above two steps, with the correct path.
  9. Commit your changes to the repository
  10. Update your ticket with changeset info and anything else
  11. Submit your ticket for review and assign to a code owner.

Q: How do I have a look at a processed suite definition which was produced using 'meto' as the value of SITE

A: Sometimes during the development of rose-stem suites and tasks for our site it's useful to examine processed rose-stem suite definitions ('suite.rc.processed') which are produced as if 'meto' was used for SITE.

Use following rose-stem arguments:

rose stem --group=ops_createbufrdirodb_kma_iasi_global: -S SITE='"meto"' -- --hold

This runs the group/task, "ops_createbufrdirodb_kma_iasi_global" using a default configuration. SITE is 'meto'.

Issues

Q: What does it mean if I run a task and it can't find the host?

A: This usually means that the configuration was not defined correctly.

Last modified 2 years ago Last modified on Nov 7, 2016 4:22:01 PM