Opened 2 years ago

Last modified 2 years ago

#311 accepted

Investigate reproducibility issue with arbitrary restarts

Reported by: Peter Dobrohotoff Owned by: Peter Dobrohotoff
Priority: major Component: ACCESS-CM2
Keywords: Cc:

Description


Attachments (2)

ocn_sst_diff.png (19.9 KB) - added by Martin Dix 2 years ago.
sst.png (9.8 KB) - added by Martin Dix 2 years ago.

Download all attachments as: .zip

Change History (9)

comment:1 Changed 2 years ago by Peter Dobrohotoff

Owner: set to Peter Dobrohotoff
Status: assignedaccepted

comment:2 Changed 2 years ago by Peter Dobrohotoff

  1. ACCESS-CM2 1-month and 3-month resubmissions give different results
  2. ACCESS-OM similarly does not reproduce
  3. ACCESS GA7 (JULES) AMIP is ok
  4. ACCESS GA7 (CABLE) AMIP to be tested

comment:3 Changed 2 years ago by Martin Dix

Set up a coupled model suite that compares 1+1 and 2 day runs.

Changed 2 years ago by Martin Dix

Attachment: ocn_sst_diff.png added

comment:4 Changed 2 years ago by Martin Dix

The GA6 and 7 AMIP suites include tests for reproducibility on different decompositions, with and w/o OpenMP and also that a sequence of shorter runs matches a single longer run.

The normal test with the 360 day calendar is that 3 10 day runs matches a standard 30 day run.

I've now also checked this with the Gregorian calendar with a variant of the vn10.6 GA7.1 suite u-aj458. A sequence of 2 one month runs matches a single two month run.

Note that with UM reproducibility requires that each run write dump files at the same frequency.

The suite u-an301 compares two coupled model runs in the same way by comparing the final UM restart files. 2x1 month is different to a 2 month run.

Comparing the UM solver diagnostics from the start of the second month shows that the first difference appears at 1-02-01 03:20:00, i.e. just after the coupling step.

Ocean model from 2 month run

 From ocean_tracer_mod: tracers after explicit tendency update (taup1)
  
 === Prognostic tracer checksum follows ===
yyyy/mm/dd hh:mm:ss =    1/ 2/ 1  0:30: 0
[chksum] temp                                      7854997267852092459

From start of second month of 2x1 month runs

 From ocean_tracer_mod: tracers after explicit tendency update (taup1)
  
 === Prognostic tracer checksum follows ===
yyyy/mm/dd hh:mm:ss =    1/ 2/ 1  0:30: 0
[chksum] temp                                     -1392703324634173096

Comparing OCN_SST from the coupling diagnostic files at 0300 shows a coherent large scale pattern which suggests some difference in the physics rather than just random noise.


I couldn't find any per step ice diagnostics to compare. The ocean difference at 0030 could be due to ice-ocean coupling but the pattern of SST differences makes me suspect MOM is to blame.

Last edited 2 years ago by Martin Dix (previous) (diff)

comment:5 Changed 2 years ago by Martin Dix

If the UM is set to save daily dump files then in AMIP mode 2x1day compares to a 2 day run. However this doesn't compare with the coupled model. The symptoms are the same as for the monthly run with the differences in the MOM checksum and a similar pattern of SST differences from the coupler diagnostics.

comment:6 Changed 2 years ago by Martin Dix

u-an301 suite details. From rose-suite.conf

RESUB='P2D'
RESUB_CRUN='P1D'
RUNLEN='P2D'
TEST_CRUN=true

This does a normal run of length 2 days and another run of 2x1 day. Task names are coupled and coupled_crun. UM output goes to History_Data and History_Data_CRUN. Data is archived to share/archive/an301 and share/data/an301_CRUN rather than to the normal archive subdirectory.

Changed 2 years ago by Martin Dix

Attachment: sst.png added

comment:7 Changed 2 years ago by Martin Dix

Change diag_table to have

"HISTORY/ocean_daily",                30,  "minutes",  1, "days", "time",

gives SST per time step. Plotting this shows a clear jump in the SST at the start of the second day in the 2x1 run (red line).


This looks like the difference one might see using a forward time step at the start of a run rather than the normal centered time step (c.f. the flat section at the very start of the run).

Last edited 2 years ago by Martin Dix (previous) (diff)
Note: See TracTickets for help on using tickets.