Opened 12 months ago

Last modified 12 months ago

#364 new

Assess OPS and VAR of G3 N1024 Terra trial using Varstats

Reported by: Jin Lee Owned by:
Priority: major Component: ACCESS model
Keywords: Varstats OPS VAR G3 Cc: Susan Rennie, Peter Steinle, Zhihong Li, ying.zhao@…, Yi Xiao, Gary Dietachmayer, Robin Bowen, Michael Naughton, fs5892, Chris Tingwell

Description

It appears the APS3 ACCESS-G N1024 trial that ran on Terra did not perform as expected. In this ticket Varstats will be used to the assess the OPS and VAR aspects of the trial.

Attachments (1)

ps39_m400_trial_ps39_m400_trial_copy_var_iasi_ma_countmean_AveVals_i0_l1_s3_ch164--263.png (73.2 KB) - added by Jin Lee 12 months ago.
ps39_m400_trial_ps39_m400_trial_copy_var_iasi_ma_mean_AveVals_i0_l1_s3_ch164--263.png

Download all attachments as: .zip

Change History (10)

comment:1 Changed 12 months ago by Yi Xiao

Cc: Yi Xiao added

comment:2 Changed 12 months ago by Peter Steinle

Cc: Gary Dietachmayer Robin Bowen added

comment:3 Changed 12 months ago by Peter Steinle

Cc: Zhihong Li added; zhk548 removed

comment:4 Changed 12 months ago by Michael Naughton

Cc: zhl5148 Michael Naughton added; Zhihong Li removed

comment:5 Changed 12 months ago by Michael Naughton

Cc: Zhihong Li added; zhl5148 removed

Changed 12 months ago by Jin Lee

ps39_m400_trial_ps39_m400_trial_copy_var_iasi_ma_mean_AveVals_i0_l1_s3_ch164--263.png

comment:6 Changed 12 months ago by Jin Lee

Cc: fs5892 added

VAR stats files

2 trials are compared. The source of their data are,

  • G3 N1024 Terra trial: /samaccess_dev/ACCESS_dev/ps39_m400_trial/2016/0[56]
  • UKMO PS39 N1280 global suite: raijin5:/home/548/jtl548/da/access_verif/g3trial_terra/data/ps39_n1280_ukmo/ps39_varstats (Fiona retrived files from UKMO MASS)

Varstats

Varstats expects 2 trials - experiment and control - to be run over the same period. For the current comparison this is not the case. To get around this atypical usage dummy runs were created - copies of experiment and control.

Results

  1. For many of the observation types more observations are used in G3 than in UKMO global: see "GPSRO_countmean_AveVals", "iasi_ma_countmean_AveVals", for example. Exceptions are Satwind, Scatwind, Surface,
  1. For most observation types the mean of the first-guess departure (o-b) is similar in both trials (perhaps G3 is slightly worse). However G3 has higher standard deviation of o-b for most observation types. Presumably this is because the forecasts from the UKMO global is better than those from G3.
  1. Comparing the values of observation penalty functions for a number of observation types the numbers for G3 are larger than for those of UKMO. Most likely because more observations are used in G3 than in UKMO: e.g. see "GPSRO_penalty_ts_penalties", "iasi_mb_penalty_ts_penalties"
  1. We know there were problems with the assimilation of AMV's in G3. Looking at the plots for the Satwind observation type,
    1. Contrary to other observation types less Satwind observations are used in G3 than in UKMO: ~24000 for G3 compared to ~40000 on average (see "satwind_U_countmean_AveVals")
    2. Comparison of the mean values of of o-b shows mixed result

comment:7 Changed 12 months ago by Jin Lee

Cc: Chris Tingwell added

comment:8 Changed 12 months ago by fs5892

I am not convinced the biases are consistent enough. The biases in the tropospheric temperature sounding channels for the hyperspectral sounders, and also AMSU-4 is quite a lot bigger in G3. Also the bias against Sonde for temperature is larger throughout the troposphere. The lower level humidity also looks more biased in G3 - some AMSU-B/MHS show this, but also the sonde bias looks quite different.

Sonde wind biases are also different, but this could be explained by the lack of AMV data.

Also, we shouldn't be assimilating SSMIS until May 2018. This needs to be switched off. I did mention this previously. http://logan.bom.gov.au:8011/nwp-wg/wiki/SSMISAvgDataCheck.

We should bear in mind this trial is not running with recommended SatRad? coefficients (they were only fixed yesterday, so clearly it wasn't) - you can see this because SSMIS 2&5 are still on and have been switched off in the new version. I don't think this is a major issue - the only changes are switching off channels that might cause problems under certain circumstances which are unlikely to be met in a given short period.

comment:9 Changed 12 months ago by fs5892

I've been looking at this again in a new light having just checked the *new* G3 setup. I think I was a bit over-zealous in my interpretation of the differences in the means - they are bigger for G3, but very small overall, so probably not too big a concern for now.

A comment on the observation counts, following conversation with Jin: Although there are a few mysteries, actually the observation counts are pretty similar if you compare G3-glm vs MO-glu.

Note: See TracTickets for help on using tickets.