wiki:ticket/370/TicketDetails

Version 92 (modified by Jin Lee, 6 weeks ago) (diff)

--

Assess possible degradation of TC forecasts by ACCESS-G caused by not assimilating some observation types

Bureau currently does not receive and so its models do not assimilate a large number of observation types that UKMO doesn. This is thought to degrade Bureau's global model forecasts.

There is a view that the observation coverage is sufficient (Kelly et al. 2007).

However Fiorino (2009) showed that changes (in this case IFS physics) may not show up in gross verification statistics but can show up in tropics: e.g. tropical winds, tropical cyclone tracks.

Also the impact of one observation type on its own may not be clear. However the synergistic effect of a number of observation types can be magnified.

Additional observation types

Following two tables show a comparison of observations which are currently used operationally in ACCESS-G3 and those used in UKMO OS42,

In-situ observations

Obstype Usage in ACCESS-G3 Usage in UKMO OS42 Comment
METAR, SYNOP used used
TEMP, PILOT, SONDE used used
AIREPS, AMDARS used used

Satellite observations

Obstype Usage in ACCESS-G3 Usage in UKMO OS39 Comment
ATOVS MetOp-1, MetOp-2, NOAA-15, NOAA-18, NOAA-19 MetOp-1, MetOp-2, NOAA-15, NOAA-18, NOAA-19
AIRS Aqua Aqua
IASI MetOp-1, MetOp-2 MetOp-1, MetOp-2
CrIS S-NPP S-NPP
ATMS S-NPP S-NPP
AMSR GCOM-W1 GCOM-W1
SSMIS DMSP17, DMSP18 DMSP17, DMSP18
GMI not used GPM UKMO OS40 started assimilating GMI from 20180213T06
MWTS-1, MWHS-1 not used FY-3B, FY-3C In OPS app config MetDB subtype is ATOVS for FY-3B and MWTS for FY-3C; Q. Does this mean MWHS is not used?
Satwind METEOSAT-8, METEOSAT-11, H-8, GOES-15, GOES-16, Aqua METEOSAT-8, H-8, GOES-15, MetOp-1, MetOp-2, NOAA-15, NOAA-18, NOAA-19, S-NPP, Aqua, Terra, mixture of GEO/LEO ACCESS-G3 uses METEOSAT-11 and GOES-16 which are not used by UKMO; the control and the experiment do not use these 2 satellites
Scatwind MetOp-1, MetOp-2, CORIOLIS MetOp-1, MetOp-2, CORIOLIS
GPSRO MetOp-1, MetOp-2, TerraSAR-X, FY-3C MetOp-1, MetOp-2, TerraSAR-X, COSMIC-1, COSMIC-6, FY-3C N.B. Less satellites are used by ACCESS-G3
Ground GPS used used
MT/SAPHIR not used MT From some time in Dec 2017 until 20180101T00 no data in obstore; from 20180101T06 obstores contain data
SEVIRI CSR not used METEOSAT-8, METEOSAT-10
AHI CSR H-8 H-8
GOES CSR not used GOES-13, GOES-15

Note 1. For satwind and surface obstypes I could not come up with apps that would allow reading of all obs. For this I compared the numbers given by print-obstore with numbers in job.stats

Experimental set-up

For each TC case we will use a control and a test. The control will assimilate all observation types including those that we currently do not assimilation but are used by UKMO; the test will use only the observations used operationally in ACCESS-G3.

We will mainly assess the difference in track forecasts between control and test. However a question arises as to how robust the difference in track forecasts is. It's possible that the difference arose because of the inherent chaotic nature of the data assimilation system and the forecast model. To allow us to approximately the effect of this randomness we will also create an initial condition by adding global, random perturbations to the control.

For the trial period, 20180224T06-20180501T12 2 suites were set up:

  • u-bl688 - control
  • u-bm460 - experiment that excludes all observations which are not used by ACCESS-G3

For the trial period, 20190809T06-20190909T12 2 suites were set up:

  • u-bm976 - control
  • u-bn066 - experiment that excludes all observations which are not used by ACCESS-G3

Trial periods

The following is a list of Western Pacific TC's used in this experiment and the trial periods,

TC First cycle of trial last cycle of trial lon/lat when first declared Category Comments
KAI-TAK 2017120406 2017122312
TEMBIN 2017121106 2017122512 straight track
HILDA 2017121706 2017122812
BOLAVEN 2017122406 2018010312
IRVING 2017122706 2018010612 Cat 1
JOYCE 2018010106 2018011312 Cat 1
CEBILE 2018011806 2018020812
FEHI 2018011906 2018013012
GITA 2018013106 2018022212 Cat 4
SANBA 2018020106 2018021612
KELVIN 2018020706 2018021912
HOLA 2018022406 2018031112 Cat 4 named on 20180306 and at its naming Cat 1
LINDA 2018030306 2018032512
MARCUS 2018030606 2018032512
NORA 2018030906 2018032712
IRIS 2018031406 2018040512
JELAWAT 2018031406 2018040112
JOSIE 2018032106 2018040212
KENI 2018032906 2018041112
FLAMBOYAN 2018041706 2018050112
EWINIAR 2018052306 2018060912

Starting cycletimes are chosen 10 days prior to the dates when TC's were declared ("Start date"). ???? might need to go a little further to allow analysis of pre-storm environment ?????

Tropical storms in the Atlantic Ocean over a similar period,

Tropical storm First cycle of trial last cycle of trial lon/lat when first declared Category Comments
Tropical Storm Alberto On 25 May, 2018 organisation sufficient to classify it as sub-tropical storm; 28 May it became a tropical storm and reached its peak intensity

Tropical cyclones for case study,

Tropical storm First cycle of trial last cycle of trial lon/lat when first declared Category Comments
Dorian 20190809 20190909 19 Aug NHC identified a tropical wave in a monsoon trough over western Africa; Cat 5 on 1 Sep

Configuration

As one of the verification measures to be used in this experiment is the ability of the trials to predict TC tracks it is important to resolve the inner core structure of a TC; even better, to have a few gridpoints within the eye of a storm (Q. In order to model the interaction between a TC and its environment in related to TC movement do we need to resolve the eye?). To achieve this I would like to run UM at N640 resolution (see here for some issues encountered when changing the resolution of the ported PS41 suite from N320 to N640). However, running at this resolution for a trial lasting for more than 3 months is unaffordable at NCI. So as a compromise I decided to run at N320 UM resolution.

OPS

  • Obstore files were retrieved from the update runs of UKMO operational OS,
    • all obstypes are tar'ed in a single tarball
    • raijin4:/g/data/dp9/as2291/obstores
    • 1.1 GB of data per cycle
    • from 20171204 00Z till 20180222 18Z (inclusive)
  • Ops_CreateODB and Ops_ExtractAndProcess read observations from obstore files
  • OPS control/data files
    • stationlist
      • thinning distance:
Obstype Thinning distance - extratropics Thinning distance - tropics
AHIClear no thinning no thinning
AIRS 125 154
AMSR 80 80
ATOVS 100 100
ATMS 125 154
CrIS 125 154
FY3B 125 154
GMIhigh 80 80
GMIlow 80 80
GOESClear no thinning no thinning
GPSRO no thinning no thinning
IASI 80 154
MTSAPHIR no thinning no thinning
MWSFY3C 80 80
Satwind 200 200
Scatwind 80 80
SEVIRIClear no thinning no thinning
SSMIS 100 154
  1. How does the thinning distance interact with PFM resolution? For each obs Cx is a column interpolated horizontally and in time to the obs location. It is fixed in the inner loop. Cx+ = Cx + S^-I*(Cw) where Cw is dw interpolated horizontally and in time to the obs location in PFM grid. The observation operator then maps Cx+ to the observation space, y=V(Cx+) to give an estimate of the observed value.

VAR

  • non-hybrid 4DVar
    • alternatively hybrid uncouple 4DVar with error modes from an archive of a single run
  • PFM resolutions are N108 and N216
  • used a new VAR build to get around a bug that incorrectly opens RTTOV coefficients - details here

UM

  • Resolution: N320L70 (0.5625 deg lon and 0.375 deg lat; 63 km x 42 km at equator)
  • PC2 cloud scheme
  • ????

Results

Overall verification

While trial still running: GES output

  • Score cards show the experiment run has degraded forecasts - blue dominates over green
  • Reduced first-guess departures for the control,
    • IASI - more pronounced improvement around MetDB ch 120 which are mid- to lower tropospheric channels (weak, ozone channels), some window channels, around ch 270 (low-peaking, water vapour channels)
    • similar trend for other IR and MW instruments
  • unexplained result is the obs count - in some cases the obs count for the control dropped (Q. Is this what I think it is?)

Note. Control (u-bl688) uses all the observations that UKMO use and the experiment (u-bm460) uses those observations that ACCESS-G3 uses

Verificaition of TC tracks

  • TC tracker needs following fields,
    • essential: MSLP; 10-m winds; 850, 700, 500 hPa u/v winds and geopotential heights (available in the archived files, "*_glm_t???.ff")
    • additional: 800, 500, 300 hPa heights
    • fields need to be in Grib1 format
    • TC bogus files are needed to be used as first guess for the tracker
    • Jim to do an initial test of TC tracker once analysis and forecast files are available from a single basetime
      • test whether all STASH fields are output for TC tracker by cold-starting the suite from 20180306T06; long forecast from 20180306T12 should produce forecast files which can be used by the tracker

Resources

  • Jim Fraser's List of TCs in West Pacific & Eastern Indian Ocean from G3 trial periods
    • Note "First Lat" and "First Lon" columns show lat/lon when TC's were first named; they are the TC locations on the dates under the column, "Start date"
  • JTWC bogus central pressure observations are available from Dec, 2017
    • the central pressure data are converted to text files and then these text files are read in by OPS

Useful information

  • Following table has different gridspacing used for various resolutions of UM and PFM
Resolution dx x dy at the equator (deg) dx x dy at the equator (km)
N108 1.6667 x 1.1111 185 x 124
N216 0.8333 x 0.5556 93 x 62
N320 0.5625 x 0.375 63 x 42
N640 0.3515625 x 0.234375 39 x 26
  • The diammeter of the inner core of a typical tropical cyclone is ~300 km (Emanuel, 2018, Meteorological Monographs). So what is resolved for different resolutions is,
    • N320 - 5 gridpoints within the inner core, which means inner core is just resolved (using the "4 delta x" criterion)
    • N216 - 3 gridpoints within the inner core, which means inner core is not quite resolved
    • N108 - 2 gridpoints within the inner core, which means inner core is definitely not resolved

Diary of runs

u-bl688

Cycle time Failed task Reason for failure Action taken
20180224T06 - 20180302T12 Not all batches in MTSAPHIR obstore files were read in; consequently ~1 in 10 observations was written out to varobs In OPS namelist group, extractcontrolnl the variable, maxbatchessubtype was set too low maxbatchessubtype changed to main app config value of 40
20180224T06 - 20180304T18 ODB2 files are not archived for this earlier time period I didn't switch on archiving of ODB ARCH_ODB2=True is used now
20180501T0600Z gl[mu]_ops_process_background_seviriclear SEVIRIClear obstore file contains METEOSAT-11 (SatId 70); as VarBC file does not contain parameters for this satellite the task fails None since it is very nearly the end of the trial period, 20180501T12

u-bm460

Cycle time Failed task Reason for failure Action taken
20180224T06 - 20180304T18 ODB2 files are not archived for this earlier time period I didn't switch on archiving of ODB ARCH_ODB2=True is used now
20180501T0600Z gl[mu]_ops_process_background_seviriclear SEVIRIClear obstore file contains METEOSAT-11 (SatId 70); as VarBC file does not contain parameters for this satellite the task fails None since it is very nearly the end of the trial period, 20180501T12

u-bm976

Cycle time Failed task Reason for failure Action taken
20190819T1200Z glu_surf_ascat_ekf Reading in Bufr file (?) skipped all SURF tasks; ran glu_um_fcst task without updating soil moisture with glu_smc
20190902T1800Z gl_ver_hk_ard ARD_056_008_+7.00000E+02_T00000_G03_u-bm976-GM.con corrupted Replaced the corrupted ARDfile with an archived copy; note the archive of ARD was done after all tasks of 20190902T00 so forecasts from 06Z, 12Z and 18Z are not in the ARDFile as well as missing verification stats for 06Z, 12Z and 18Z
20190904T1200Z gl[mu]_ops_process_background_cris the job failed when updating ODB1 reset to succeeded; not assimilate CrIS for this cycle
20190904T1200Z glu_ops_process_analysis_cris Unknown reset to succeeded

u-bn066

Cycle time Failed task Reason for failure Action taken
20190809T06 glu_um_fcst NaN or Infinities developed Did a FASTRUN
20190826T0000Z glu_ops_process_analysis_satwind Unknown Set to succeeded
20190826T0600Z glu_ops_process_background_cris Unknown; varobs and varcx were created; exactly same error occurred to u-bm976 but it ran after retry reset to succeeded; ToDo? Check if CrIS were used by VAR
20190826T0600Z glu_ops_process_analysis_cris Unknown reset to succeeded
20190827T0600Z glu_ops_process_analysis_ssmis Same as what happen to glu_ops_process_analysis_cris.20190826T0600Z; TicketDetails/job.out reset to succeeded
20190828T0000Z gl[mu]_ops_process_background_cris failure during updating of ODB1 reset to succeeded
20190828T0000Z glu_ops_process_analysis_cris All job.out has /home/548/jtl548/cylc-run/u-bn066/share/cycle/20190828T0000Z/glu_odb/cris is locked; because of the failure of glu_ops_process_background_cris while writing back to ODB1 .lock file might have been created reset to succeeded
20190828T1200Z gl_ops_ver_odb_obstore_satwind Looks like error during opening of ODB1 reset to succeeded
20190828T1200Z glu_ops_process_analysis_satwind Looks like error during opening of ODB1 reset to succeeded
20190904T1200Z gl[mu]_ops_process_background_cris the job failed when updating ODB1 reset to succeeded; not assimilate CrIS for this cycle
20190904T1200Z glu_ops_process_analysis_cris Unknown reset to succeeded
20190905T1200Z glu_ops_process_analysis_atms Unknown reset to succeeded

Things to do

  • What do gl?_ops_process_analysis_* tasks do - generate feedback ODB1? Should these tasks be removed from the suite?
  • TCBOGUS data seem to be available in UKMO obstores. Will they be used by VAR?
  • What to do about locally received data?
  • for gl_ops_process_satwind select based on satellite
  • for gl_ops_process_gpsro select based on satellite
  • set up 2 identical twins for a case study: e.g. Sandy, Irma or Dorian
  • try using the following to fix the problem of job log output files not being retrieved,
    [hosts] -> [[HOST]] -> retrieve job logs max size
    [hosts] -> [[HOST]] -> retrieve job logs retry delays