Changes between Version 16 and Version 17 of access/SMOCS

Timestamp:

Jul 18, 2018 9:16:12 AM (5 years ago)

Author:

Lawrence Rikus

Comment:

Revision for July 2018

access/SMOCS

@@ -10 +10 @@
   * Structure software so specific directory/file structure is at upper levels so that the underlying processes are as general as possible.
   * Develop tools to interrogate the collected data for various purposes
-      * Quantify model health (e.g. do the statistics stray from expected ranges)
+      * Quantify model health (e.g. do the statistics stray from expected ranges?)
       * Evaluate the differences between models
           - for model upgrades
@@ -24 +24 @@
 == Collection stage == 
 
-   The collection scripts cycle over the output files from each model run. They can either be run at the end of each model output step or over a directory containing a set of output files. This means they can be used to collect statistics for an operational model by being run over the operational output directory or on the model output archive at the NCI (rr4). Currently the input format has to be netcdf or UM pp-fieldsfiles so grib archives need translation before they can be 'collected'. The scripts read each file and try to read all the fields on each file. The mean, standard deviation and maximum and minimum values are calculated over the entire domain as well as over the sea and land separately and saved for each model output time step. The position in the field of the maximum and minimum are also saved. The results are saved as a json file for each model run and these in turn are saved in a directory structure which separates each month.As well as the general collection scripts there are separate 'utility collection' scripts. These calculate fields of interest which are not output by the model (such as the hourly rainfall accumulations or overall budget terms) and add them to the forecast summary json file. For 3D fields the results are saved for the individual levels as well as the full 3D domain.
+   The collection scripts cycle over the output files from each model run. They can either be run at the end of each model output step or over a directory containing a set of output files. This means they can be used to collect statistics for an operational model by being run over the operational output directory or on the model output archive at the NCI (lb4). Currently the input format has to be netcdf or UM fieldsfiles so grib archives need translation before they can be 'collected'. The scripts read each file and try to read all the fields on each file. The mean, standard deviation and maximum and minimum values are calculated over the entire domain as well as over the sea and land separately and saved for each model output time step. The position in the field of the maximum and minimum are also saved. For 3D fields the results are saved for the individual levels as well as the full 3D domain.
+Certain fields have the option for collecting either a percentile or histogram (bin) table which effectively bins values and their coordinates. 
+
+
+The results are saved as a json file for each model run and these in turn are saved in a directory structure which separates each month. As well as the general collection scripts there are separate 'utility collection' scripts. These calculate fields of interest which are not output by the model (such as the hourly rainfall accumulations or overall budget terms) and add them to the forecast summary json file.
+
+Currently the I/O engine is based on cdms which is a python library which reads in netcdf or UM fieldsfiles and uses a netcdf variable API.
 
 == Harvesting Programs ==
@@ -34 +40 @@
 '''bodoView.py''' - a simple json tree viewer available to look at individual SMOCS json files. Available in the Tools svn subdirectory or from my python directory on all machines.
 
+'''smocsView.py''' - an updated version of bodoView.
+
 === Known Bugs ===
   * Name translation errors
-     * accumulated precip is identified as accum_prcp_rate_pa which propagates through the rate calculations
-     * maximum screen temperature is identified as temp_scrn
+     * New fields not in the translation table are identified by their STASH id.
 
 
@@ -43 +50 @@
 
   * Energy integral calculation for model monitoring purposes.
-  * Add in percentile binning to augment/replace the binning process.
   * Submodel domain generation (e.g. r,c in g; c in r; tc in g?)
   * Sub-region (e.g. NH, SH, tropics etc) for global domains.
   * Efficiency improvements:
-       * Speed up binning routines
-       * parallelize in python
+       * Speed up binning/percentile routines
+  * Investigate a move to iris and/or mule
  
 = File Locations =