Opened 4 years ago

Last modified 3 years ago

#161 assigned

au-aa068: Develop APS2 ACCESS-G Rose suite and associated IT infrastructure at NCI for NWP research

Reported by: Jin Lee Owned by: Jin Lee
Priority: major Component: ACCESS model
Keywords: APS2 N512 ACCESS-G Rose suite Cc:

Description

Since early 2014 work has been going on to build an IT intrastructure at NCI to develop and support APS2 N512 ACCESS-G Rose suite. This is a collaborative effort by Fabrizio Baordo, Jin Lee, Vinod Kumar, Paul Gregory and other ESM scientists and IT staff.

The workpackage this ticket describes is very large encompassing every decision, change, bugfix, result, etc. associated with the development of the suite and attendant IT infrastructure. Unfortunately this ticket was created well after the commencement of work so a lot of relevant documentation is recorded elsewhere:

http://ngamai04.bom.gov.au:8011/APS1/wiki/APS2components

Attachments (14)

zonal_mean.diff.anal_inc.a207.N108_12x32.N216_8x48-uk.2014020206.qT.png (34.4 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for qT
zonal_mean.diff.anal_inc.a207.N108_12x32.N216_8x48-uk.2014020206.u.png (37.6 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for u
zonal_mean.diff.anal_inc.a213.18x20-12x32.2014020206.qT.png (42.2 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for qT
zonal_mean.diff.anal_inc.a213.18x20-12x32.2014020206.theta.png (34.6 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for theta
zonal_mean.diff.anal_inc.a213.18x20-12x32.2014020206.u.png (39.8 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for u
zonal_mean.diff.anal_inc.a207-uk.2014020206.qT.png (41.9 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for qT - ours vs UKMO
zonal_mean.diff.anal_inc.a207-uk.2014020206.theta.png (30.9 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for theta - ours vs UKMO
zonal_mean.diff.anal_inc.a207-uk.2014020206.u.png (38.7 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for u - ours vs UKMO
zonal_mean.diff.anal_inc.a220-a207.2014020206.qT.png (45.5 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for qT - ours vs UKMO
zonal_mean.diff.anal_inc.a220-a207.2014020206.theta.png (34.6 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for theta - ours vs UKMO
zonal_mean.diff.anal_inc.a220-a207.2014020206.u.png (40.4 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in analysis increments for u - ours vs UKMO
zonal_mean.diff.fc.a198-a171.2014120500.+24.T.png (21.9 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in +24 forecast for temp
zonal_mean.diff.fc.a196-a171.2014120500.+24.u.png (32.6 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in +24 forecast for u
zonal_mean.diff.fc.a196-a171.2014120500.+24.u.2.png (32.6 KB) - added by Jin Lee 4 years ago.
zonally averaged difference in +24 forecast for u

Download all attachments as: .zip

Change History (30)

comment:1 Changed 4 years ago by Jin Lee

Owner: set to Jin Lee
Status: newassigned

comment:2 Changed 4 years ago by Jin Lee

A methodology to track down problems with OPS and VAR

Motivation

During the development of the APS2 ACCESS-G suite iver diagnostics suggested persistent problems in its forecasts compared to operational APS1. The following summarises a systematic approach used in finding the source of the forecast problem.


comment:3 Changed 4 years ago by Jin Lee

Testing VAR

All the tests described here rely on the (obvious) fact that given a set of inputs to a component - e.g. inputs to OPS, VAR, SURF or UM - which is identical to a reference (in our case UKMO) the output from the component test should be identical to that from the reference within a random computational noise. The random computational noise is not zero because our test of a component may use different PE domain decomposition to UKMO reference; it can also arise from round-off errors which may or may not come from the use of different compiler settings.

The approach I've taken to deal with this random computational noise when comparing our test result and UKMO reference is to apply zonal averaging. Zonally averaged difference between our result and UKMO reference highlights major systematic differences and offers us a clue as to the cause of the difference.

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for qT

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for u

comment:4 Changed 4 years ago by Jin Lee

UKMO runs hybrid En-Var and so I had to derived a hybrid set-up (au-aa207) from our APS2 suite. The inputs to the hybrid En-Var are:

  • varobs and varcx
  • linearisation states (LS states) for N108 and N216
  • error modes (EM) from 44-member MOGREPS-G for N108 and N216
  • CovStats

First I had to test the new hybrid En-Var in au-aa207 was correct by comparing the analysis increment from the suite with that from the VAR tasks (glu_var_anal_n108 and glu_var_anal_n216) in the UKMO suite (au-aa213 which is a copy of mi-aa408). This was done.

Next, I had to have an estimate of the size of the computational noise. This is done by running VAR using different PE domain decompositions. A typical differences in the analysis increments from 2 different PE domain decompositions are shown below:

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for qT

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for theta

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for u

comment:5 Changed 4 years ago by Jin Lee

So an approximate estimate of the computational noise is:

  • for qT, ~10-3 g/kg
  • for theta, 0.1 K (mainly near the top of model)
  • for u, 10-2 m/s

For all the variables the noise is confined spatially.

So during our test if we get an analysis increment which differs from that from UKMO, and the size of the difference is larger than the computational noise then we can conclude that our set-up and that of UKMO are different. On the other hand, if our analysis increment is within the computational noise of the UKMO increment then we can conclude our settings are identical to that of UKMO.

comment:6 Changed 4 years ago by Jin Lee

Testing OPS

Next step is to test OPS tasks. As an intial test I used the suite's OPS tasks to produce varobs and varcx with all the other inputs to OPS-VAR coming from UKMO - i.e. LS states, EM and bgerr.

The OPS-VAR outputs are under:

raijin5:/home/548/jtl548/da/access_verif/aps2_da_problem/data/au-aa207/from_ukmo.LS.EM.own_varobs_varcx

Plots are attached:

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for qT - ours vs UKMO

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for theta - ours vs UKMO

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for u - ours vs UKMO

comment:7 Changed 4 years ago by Jin Lee

Compared to the computational noise the analysis increment difference between APS2 suite and UKMO is:

  • for qT, 1-2 orders of magnitude larger
  • for theta, 3 orders of magnitude larger
  • for u, 2-3 orders of magnitude larger

This tells us that there is a problem with one or more of our APS2 OPS task(s).

comment:8 Changed 4 years ago by Jin Lee

Testing OPS - individual obsgroups - first set of trials

There are 2 APS2 ACCESS-G suites curretly available: one running on Ngamai but is controlled by SCS; another running at NCI and uses !Rose/cylc . For some reason the !Rose/cylc version running at NCI verified worse than the one running on Ngamai. So I decided to test !Rose/cylc OPS tasks using the warm-running files of Ngamai suite as a reference instead of UKMO files. This test was done before I decided to use UKMO files so the chronology of these entries in this ticket is a bit out of whack.

However the methdology is the same. In each of the test suite same OPS tasks as our APS2 Roce/Cylc suite processes obsgroups except for the obsgroup which is under the test. For each obsgroup under investigation varobs/cx comes from samtm. The analysis increments from these suites are then compared with that from au-aa171 (copy of Rose/Cylc APS2 suite). If the difference in the analysis increment is large then we can conclude that that particular obsgroup is to blame for difference.

Following suites were set up:

suite ID obtypes whose varobs/cx come from samtm
au-aa194 AIRS, ATOVS, IASI
au-aa195 GPSRO
au-aa196 Satwind
au-aa197 Scatwind
au-aa198 Sonde, Aircraft

It turned out that the difference in the analysis increment between au-aa198 and au-aa171 was large. See plot below:

Last edited 4 years ago by Jin Lee (previous) (diff)

comment:9 Changed 4 years ago by Jin Lee

Last edited 4 years ago by Jin Lee (previous) (diff)

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for qT - ours vs UKMO

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for theta - ours vs UKMO

Changed 4 years ago by Jin Lee

zonally averaged difference in analysis increments for u - ours vs UKMO

Changed 4 years ago by Jin Lee

zonally averaged difference in +24 forecast for temp

comment:10 in reply to:  8 Changed 4 years ago by Jin Lee

Replying to jtl548:

Another attempt at attaching plot at the right place. See "zonal_mean.diff.fc.a198-a171.2014120500.+24.T.png"

comment:11 Changed 4 years ago by Jin Lee

The plot "zonal_mean.diff.fc.a198-a171.2014120500.+24.T.png" indicates that somehow Ops_as was causing forecast to be different. It turned out that Ops_as was using wrong MetDB elements file for Aircraft and Sonde: the number of vertical levels expected in the MetDB elements file did not match the actual number of vertical levels in Bufr or ODB files.

Changed 4 years ago by Jin Lee

zonally averaged difference in +24 forecast for u

comment:12 Changed 4 years ago by Jin Lee

It also turned out that the difference in the 24-forecasts between au-aa196 - varobs/cx for AMV's come from samtm but for all other obsgroups varobs/cx are produced by various tasks of the suite - and au-aa171 were very large, particular for the u-component winds. See attached plot.

Changed 4 years ago by Jin Lee

zonally averaged difference in +24 forecast for u

comment:13 Changed 4 years ago by Jin Lee

The plot above suggests that APS2's OPS task that processes AMV observations has a problem. It turned out that there was a minor bug with the app of Ops_amv, which resulted in varcx files being written to a wrong place. Without varcx for AMV's 4DVAR did not make use of AMV observations at all.

comment:14 Changed 4 years ago by Jin Lee

Testing OPS - individual obsgroups - second set of trials

In this second set of tests for each obsgroup the observations are processed by a suite while for other obsgroups varobs/varcx files from UKMO are used. Analysis increments from APS2 OPS/Hybrid En-Var are compared with the equivalent from UKMO. If OPS settings are identical to that of UKMO then the difference in analysis increments should be within computational noise; otherwise the difference would be larger.

In this test all the settings of APS2 OPS tasks are made as close to the UKMO settings as possible. This means all OPS control files (SatRad coefficents, radiance bias coefficients, stationlists, etc.) are identical to UKMO.

Following suites were set up:

suite ID obtypes whose varobs/cx are processed by suite
au-aa215 AIRS, ATOVS, IASI
au-aa216 CrIS, ATMS
au-aa217 GPSRO
au-aa218 Satwind
au-aa219 Scatwind
au-aa220 Sonde, Aircraft
au-aa221 Surface

comment:15 Changed 4 years ago by Jin Lee

Component: Accessdev ServerACCESS model

comment:16 Changed 3 years ago by Jin Lee

Problem with 'ReconSST' and 'InitialFC' tasks

The keywords 'manual completion' were discontinued in later versions of Cylc so the following generates an error,

     [[ReconSST]]
         inherit = UM
         description = "update sst/seaice"
         manual completion = True

The suite definition needs to be changed.

However, I'm not able to locate start dumps for the period, Nov-Dec 2014 so I'm not able to test this. Start dumps for the period are not available from,

flurry2:/g/ns/lustre/users/bnoc/rto/access_g/saneg

Note: See TracTickets for help on using tickets.