wiki:ticket/280/TicketSummary

Importing VARstats into the access-svn utils repository

Peter Steinle received a tarball copy of VARstats and a copy of VARstats User Guide from Keir Bovis on 24 May 2016 (see attachment below).

The VARstats source was imported into the following location of access-svn utils repository,

https://access-svn.nci.org.au/svn/utils/trunk/varstats

The source with local modifications, explained below, can be found in

https://access-svn.nci.org.au/svn/utils/branches/varstats

Installing VARstats on Raijin

Location of installation,

raijin6:/projects/access/da/utilities/var/varstats/varstats

Note that the reason for repeated 'varstats' in the directory name is because we may need to upgrade and install newer version of VARstats, in which case we may need a backup copy. This could be handled by having a backup directory,

raijin6:/projects/access/da/utilities/var/varstats/varstats.upto20161010

Then the updated installation may go into,

raijin6:/projects/access/da/utilities/var/varstats/varstats

Note 1. VARstats was not installed on Accessdev as Python module, 'matplotlib' is not installed on Accessdev

Modifications to the source code

The following modifications have been introduced to the source code

  1. In run_varstats.sh

PYTHONPATH variable was renamed to become VARSTATSPATH_SRC In fact, PYTHONPATH it an environment variable, set by the account configuration. Using it as a variable overwrites the account settings and creates confusion. Basically, it prevents code from running and must be corrected

  1. In run_varstats.sh

Hardcoded paths to python2.7 (in two places), which are clearly UK MO's paths, were replaced with the 'python' command. This command points to the default python's installation on everybody's account. Lack of this change prevents code from running and must be corrected

  1. In run_varstats.sh

Python netCDF4 module needs to be loaded. A relevant line was introduced right under the header of the form, "module load pythonlib/netCDF4/1.0.4". Lack of this change produces an informative error message, which allows to identify the problem easily. It must be corrected for the code to run

Few remarks on how to run it

The way to run it is to execute the run_varstats.sh script with an argument

run_varstats.sh setupfiles/my_setup_dat.py in .../varstats/

On its first run the software will create .../varstats/plotsdir, and put the outputs there, under a sub-directory with a meaningful name. The name of the sub-directory will refer to the names of the trials that were analysed, i.e. the experiment and the control simulation.

I think my_setup_dat.py can be located pretty anywhere but I did not check it.

In my_setup_dat.py you need to define

a) trial_dictionary This requires trialid and 3 descriptive variables, as named in my_setup_dat.py Trialid can be suiteid. The important variable is the 4th one, i.e. directory_containing_stats Software understands the structure of rose-cylc-created outputs. The path needs to point up from the root down to log sub-directory created by rose.

b) trials_to_process_dictionary This requires three variables, as named in my_setup_dat First and second variable are important. 'experid' refers to the experimental simulation 'cntrlid' refers to the control simulation Both ids refer to trialids defined in trial_dictionary for the two simulations in question

In my_setup_dat.py you need to specify which experiment-control couple you will run. This is set via a number in the processtriallist. By default, the script sets processtriallist = [2] so that it can run, without modifications, on trial datasets provided with the software distribution.

c) satinstidlist satinstidlist accepts dictionary entries, which are either pre-defined lists of satellite instruments ids or satellite instrumnets ids. By default it is set up to 'example_baseline'. All the pre-defined lists of satellite instruments are words/dictionary entries. Each dictionary entry is defined in ...src/varstats/info/satinstrlists.py under PS_satinst_dictionary. In fact they correspond to baseline suites. The entries currently coded are: 'example_baseline' - two satellite instruments: amsua_ma and amsubhr_n18 'PS34_baseline' 'PS37_baseline' 'PS37_UKV_baseline'

Furthermore, in .../src/varstats/info/satinstrdict.py under satinst_dictionary we find information associated with each satellite instrument id (which comprise the abovementioned lists of instruments for baseline suites) expanded into a platform and instrument.

The following dictionaries need not be specified as defaults are set up for them in the code: a) my_channel_selection_dictionary for which a default can be found in ...src/varstats/runplotting/readandplotvarstats.py b) my_statistics_to_plot_list for which a defaults can be found in ...src/varstats/info/plottingoptions.py c) my_types_of_plots_list for which the defaults can be found in ...src/varstats/info/plottingdict.py d) my_plotting_options_dict for which the defaults can be found in ...src/varstats/info/plottingoptions.py

run_varstats.sh can run with both VAR and OPS statistics. The log files (job.stats) is needed and the best is using the log files located at Accessdev-->cylc-run-->suite-job-id (i.g.: aa670)-->log. One need to copy this directory to RAIJIN since one cannot run under Accessdev. In "my_setup_dat.py", One only link is pointing this directory, no unzip or ungzip is necessary. That the handy characters of VARSTATS.

Also the data read from log file stored in *.pkl file. It’s a binary file.

Note 1. When Varstats runs it creates the 'plotsdir/' subdirectory in the installation directory. Since this package will be phased out and won't be developed the script was not modified to change this behaviour. The most convenient way to run Varstats is to take a branch of the source in the branch (https://access-svn.nci.org.au/svn/utils/branches/varstats) which has local modifications. Then check out your own working copy and modify it to suit your needs.

Documentation

As VARstats is not available yet on the MOSRS repository its documentation is not browseable. As a temporary solution I have put a copy of Anna Booton's User Guide on my accessdev 'public_html' location. So it's URL is,

https://accessdev.nci.org.au/~jtl548/varstats/VARstats.pdf

VARSTATS Testing

Xudong Sun currently tests VARSTATS at /g/data/dp9/xds548/source/VARSTATS.

Relevant plots at above directory.

sample display and its explanation at Wiki:

http://ngamai04:8011/APS1/wiki/SWGNotes_20160603

Some examples of our APS3 run: i.e:, ad269 (hybrid var) and ad040 for (Sat) Obs impacts on VAR and OPS [ ​https://accessdev.nci.org.au/~xds548/varstatplots_ad040_wrt_ad269.html ] [ ​https://accessdev.nci.org.au/~xds548/opsstatplots_ad040_wrt_ad269.html ]

Last modified 2 years ago Last modified on Jun 27, 2018 2:37:20 PM

Attachments (2)