wiki:ticket/193/TicketSummary

Updating agrepsGP to use OPS30.0.0

Preliminary

  • GE1 used OPS27.2 and the obsgroups used were SURFACE, SCATWIND, AIRCRAFT/SONDE, SATWIND, ATOVS, AIRS, IASI (separate smsfile), GPSRO.
  • GE2 uses OPS30.0.0 and additional obsgroups are introduced: SSMIS, CrIS, ATMS, MTSATCSR
  • In this work following OPSUI jobs are used to update agrepsGP suite tasks:
ID Owner Obsgroup Remark
oamkn vinodk SURFACE
oajsm pgregory SCATWIND
oamko vinodk AIRCRAFT/SONDE (???? separate jobs ????)
oajsl pgregory SATWIND
oamkb vinodk ATOVS
oameb fbaordo AIRS
oamec fbaordo IASI
oamkl vinodk GPSRO
oamka vinodk SSMIS new in APS2
oamge jinlee CrIS new in APS2
oameh fbaordo ATMS new in APS2
oameg fbaordo MTSATCSR new in APS2


Modifying suite tasks

4 scripts need modifying to allow OPS30.0.0 to run within the SMS suite. This is the procedure that we will follow:

Following changes were made to "OpsList_Main" to run OPS30.0.0

Parts of script needing modification Actual modification
OPS_DOMAINFROMHEADER changed from false to blank (i.e. no value)
OPS_MAKE_NEW_ODB changed from true to false
Fortran namelist groups, &GeneralControlNL remove all namelist variables except "CapSupersat=F" as default values given to these variables are correct
???? VarobsVersionNumber may need to be set depending on how old LETKF is - it may not be able to read varobs which has VarobsVersionNumber=4 ????
Fortran namelist groups, &QcNL remove this namelist group entirely
Fortran namelist groups, &ExtractControlNL &ExtractControlNL for IASI, SEVIRIClear, GOESClear are not needed
Fortran namelist groups, &FieldsControlNL retain just 'ForecastPeriods=9,12,15'
Fortran namelist groups, &ProcessControlNL &ProcessControlNL for IASI, SEVIRIClear, GOESClear are not needed
Fortran namelist groups, &VarobsControlNL not needed as default set of VarFields should be O.K.
Fortran namelist groups, &CxControlNL ???? should some be removed ????


Following changes were made to "OpsList_IASI" to run OPS30.0.0

Parts of script needing modification Actual modification
OPS env vars, export OPS_GPSROCOEFFS_DIR removed
OPS_DOMAINFROMHEADER changed from false to blank (i.e. no value)
OPS_SSMIS changed from true to false
OPS_SEVIRICLR changed from true to false
OPS_GOESCLR changed from true to false
OPS_MAKE_NEW_ODB changed from true to false
Fortran namelist groups, &GeneralControlNL remove namelist variables except "CapSupersat=F" as default values given to these variables are correct
???? VarobsVersionNumber may need to be set depending on how old LETKF is - it may not be able to read varobs which has VarobsVersionNumber=4 ????
Fortran namelist groups, &QcNL remove this namelist group entirely
Fortran namelist groups, &ExtractControlNL &ExtractControlNL for all obsgroups removed except for IASI
Fortran namelist groups, IASI.nl -> &ExtractControlNL 1. namelist variable, MetdbSubtype changed from "IASI" to "IASIG"
2. Added BufferSize=2500 and MaxBatchesSubType=40 (are these needed for extraction from obstore????)
Fortran namelist groups, &FieldsControlNL retain just 'ForecastPeriods=9,12,15'
Fortran namelist groups, &ProcessControlNL &ProcessControlNL for for all obsgroups removed except for IASI
Fortran namelist groups, &VarobsControlNL not needed as default set of VarFields should be O.K.
Fortran namelist groups, &CxControlNL not needed


Following changes were made to "EGP_ProduceModelObs_Main.sms" to run OPS30.0.0

Parts of script needing modification Actual modification
OPS_BACKERRUM removed as this is used only in BackErrCreate - looks like a confusion
OPS_UMBACK_LIST This env var should point to CX background so it now directly assigned to $PP_012_DIR/ens$ENS_MEMBER.pp_012.$yyyymmddhh_m12h
Stationlist ???? Need to use same stationlist as deterministic ????
Bias regression coefficients ???? What coefficients need to be used - for each obsgroup OPS runs N times using at each time CX background from one ensemble member; so in theory the bias coefficient should be unique to each ensemble member ????


Following changes were made to "EGP_ProduceModelObs_IASI.sms" to run OPS30.0.0

Parts of script needing modification Actual modification
OPS_BACKERRUM removed as this is used only in BackErrCreate - looks like a confusion
OPS_UMBACK_LIST This env var should point to CX background so it now directly assigned to $PP_012_DIR/ens$ENS_MEMBER.pp_012.$yyyymmddhh_m12h
Stationlist ???? Need to use same stationlist as deterministic ????
Bias regression coefficients ???? What coefficients need to be used - for each obsgroup OPS runs N times using at each time CX background from one ensemble member; so in theory the bias coefficient should be unique to each ensemble member ????


Comparing ensemble OPS tasks

For each obtype one varobs file from the old ensemble OPS task for one member is compared with corresponding varobs file from the newer ensemble OPS task. The varobs files compared are:

ngamai03:/g/sc/home/jinlee/da/ens/data/test_ens_ops/old/Process04.varobs

  • Copy of ngamai03:/g/sc/fs4/flush/dhsmith/GP1/2015051800/etkf/EnsembleOPS/Process04.varobs

ngamai03:/g/sc/home/jinlee/da/ens/data/test_ens_ops/new/Process04.varobs

  • copy of ngamai03:/g/sc/fs4/flush/dhsmith/agrepsGP2/2015051800/etkf/EnsembleOPS/Process04.varobs
obtype total # obs in old varobs total # obs in new varobs remark
Aircraft 25795 24844 as far as matching obs are concerned they are identical
Sonde 976 973 For matching obs obs values are slightly different; PGE's are different too
Surface 38019 35594 For matching obs obs values are; PGE's are different
AIRS 2315 7182 print-varobs produces large files


Notes

  1. "OpsList_Main" processes all obsgroups except IASI. This is different to the way OPS tasks run in an assimilation cycle where one OPS task processes one obsgroup


ToDo

Task Description Status
All control, coeffs, bias files need to point to BNOC directory. Also the version numbers should be checked against those in UKMO collab wiki page open
varobsversion (????) should be checked so that subsequent jobs - LETKF (?) - can read varobs open
Is the format of CX background right as an input to OPS30.0.0? open
Is the format of bgerr right as input/output to/from OPS30.0.0? open
Check TrimObstore accepts new obsgroups The inputs to TrimObstore are obstore and varobs files for different obtypes. The obstore files contain exactly the observations contained in ODB's which were used by the deterministic OPS. The varobs files were also produced by the deterministic OPS but they contain subsets of observations after a series of quality control steps and thinning. TrimObstore produces output obstore files which contain the same observations as corresponding varobs files. These trimmed obstore files are used to update inflation factor (how?) open
On p3 "ETKF Documentation" there is a remark about $ENS_PERTVAROBS_DIR and $ENS_CURVAROBS_DIR. According to this document deterministic varobs should no longer be used by TrimObstore and instead control varobs be used. Also see p6 of the document ???? ????
The ETKF transform matrix calculation is done at N216 resolution - same resolution as the forecast part. In this upgrade need to check any parts which may depend on resolution The parts which may be resolution-dependent are:
1. CX background
Open
An inflation factor is multiplied to the raw transform matrix for each localisation centre. Each inflation factor is made equal to the ratio of the rms error of the ensemble mean (i.e. innovation vector = y_o - H(x_bar), where x_bar is the mean of forecast ensemble from previous cycle and y_o is the vector of current observations) and the rms spread of forecast ensemble (in observation space). This ensemble spread calibration also requires accurate observation error variance (R). For this reason only certain observation error variance and hence certain observations are used. Observation groups which are supported by ETKF can be found in EnsMod_Varobs/EnsMod_VarobsNumber.f90. It looks like current ENS source code doesn't support SSMIS, CrIS, ATMS and MTSATCSR open
Compare varobs/modelobs files from OPS tasks of the control SMS suite with those from OPS30.0.0 Use print-varobs and other OPS utilities ????
Need to decide on setting OPS stationlist, bias coefficients files, SatRad coefficients files, MetDB elements files ???? ????
???? ???? ????
Last modified 5 years ago Last modified on Jul 15, 2015 12:49:18 PM