Building and running the ACCESS 1.0 coupled model on raijin

Build

The build details are based on Hailin Yan's notes, adapted for raijin. Note that the MOM and CICE builds are the same for the CMIP5 versions of ACCESS1.0 and ACCESS1.3. However the current version of ACCESS1.3 (basis of ACCESS 1.4 development) uses OASIS-MCT.

The instructions here use OASIS3 via the module ~access/modules/oasis3 whereas we originally built it as a separate step in the coupled model build. Scripts all include

module use ~access/modules
module load oasis3/3

The module sets the environment variables for the compiler search paths so that explicit include and library paths are not necessary (similarly for netCDF).

Preparation

Create the directory ACCESSHOME (you can use whatever name you prefer), then check out the /bin directory from cmip5 trunk:

mkdir ACCESSHOME
cd ACCESSHOME
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/bin@r80

Create “submodels” directory under ACCESSHOME

mkdir ACCESSHOME/submodels

CICE4.1

cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/cice/branches/auscom/release-1.0-raijin cice4.1_cmip5
cd cice4.1_cmip5
./comp_access_cice4_RAIJIN 6

Here 6 is the number of cpus allocated for CICE model in the standard ACCESS configuration. Unfortunately CICE must be built for a fixed number.

MOM4p1:

cd ACCESSHOME/submodels
svn co https://access-svn.nci.org.au/svn/mom4/branches/mom4p1_cmip5_raijin
cd mom4p1_cmip5_raijin/compile
./comp_auscom_mom4p1.RAIJIN

UM

cd ACCESSHOME/submodels
mkdir UM
cd UM

svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/compile
svn co https://access-svn.nci.org.au/svn/um/branches/dev/han32j/VN7.3-HadGEM2-1.1 umbase_hg2-r1.1
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/submodels/UM/ummodel_hg2-r1.1

cd compile

qsub compile_hadgem2-r1.1

After compilation all the executables are in ACCESSHOME/bin. Names have a date suffix, e.g.

% ls -lt /short/p66/mrd599/ACCESS_BUILD/bin   
-rwxr-x--- 1 mrd599 p66 28191506 Oct 17 15:36 um_hg2-r1.1.exe-20131017
-rwxr-x--- 1 mrd599 p66 34463096 Oct 17 15:06 mom4_MPI1.exe.20131017.RAIJIN
-rwxr-x--- 1 mrd599 p66    98971 Oct 17 14:58 mppnccombine.RAIJIN
-rwxr-x--- 1 mrd599 p66  8993556 Oct 17 14:55 cice_MPI1.RAIJIN_6p_20131017

Running the model

All necessary input files for the historical simulation are in directories ~access/data/ACCESS_CMIP5 (model control files), ~access/data/ancil/CMIP5 (CMIP5 emissions) and ~access/data/ancil/access_v2 (land sea mask, vegetation properties etc).

The coupled model control scripts for the original ACCESS 1.0 CMIP5 run are saved at ​https://access-svn.nci.org.au/trac/cmip5/browser/trunk/exp/hg2-r11Mhd?rev=75. These have been modified to use the files under ~access. The init script has also been modified to allow starting a new run from any year of an existing run (provided the appropriate restart files are in ~access/data/ACCESS_CMIP5/restart/ (this restriction can likely be removed when the /g/data filesystem is available on raijin). At the moment the files for starting the historical run at 1951 are there.

Experiment IDs for ACCESS-CM 1.0 are as follows:

Experiment ID	Description
hg2-r11M1p	1% per year CO2
hg2-r11M4c	4x CO2
hg2-r11Mhd	Historical to 2006, then RCP45
hg2-r11Mpd	Pre-Industrial Control
hg2M-RCP85	RCP85

Historical

To set up the model run,

mkdir -p $HOME/ACCESS/exp
cd $HOME/ACCESS/exp
svn co https://access-svn.nci.org.au/svn/cmip5/trunk/exp/a10_hist
cd a10_hist

In script a10_hist, change ACCESS_BINDIR to ACCESSHOME/bin where ACCESSHOME is the directory you used for the build. In a10_hist.init, set the run period (iniyear and finalyear). Also, change the executable specification

  cp -f $ACCESS_BINDIR/mom4_MPI1.exe.20131017.RAIJIN 	$ocn_exe
  cp -f $ACCESS_BINDIR/cice_MPI1.RAIJIN_6p_201310128   $ice_exe
  cp -f $ACCESS_BINDIR/um_hg2-r1.1.exe-20131017        $atm_exe

to match the build dates of your versions. There are two instances of these copy commands in a10_hist.init.

Then to start the run

qsub a10_hist

The model runs in 3 month chunks, each taking about 70 minutes.

Run log files will appear in this directory. The model run directory is

/short/$PROJECT/$USER/work/um_coupled/a10_hist

and results are archived to

/short/$PROJECT/$USER/archive/a10_hist

Note that to change the experiment name, the directory and the 3 files must have names changed consistently.

The experiment setup will have to be modified further to go past 2005.

Pre-Industrial Control

The pre-industrial control experiment works much the same as the Historical version

mkdir -p $HOME/ACCESS/exp
cd $HOME/ACCESS/exp
svn co https://access-svn.nci.org.au/svn/cmip5/branches/dev/saw562/a10_piControl
cd a10_piControl

In a10_piControl set ACCESS_BINDIR, and in a10_piControl.init set the executable build dates as above.

The model starts from archived CSIRO restart files stored on Raijin at /g/data1/p66/ACCESSDIR/har599/ACCESS/output/hg2-r11Mpd/restart. You can set the file to start from by changing the iniyear variable in the file a10_piControl.init.

Submit the run with

qsub a10_piControl

By default the run will progress in three-month segments, automatically resubmitting itself after a segment has completed, until it reaches the year given in the finalyear variable.

RCP 45/85

The RCP scenarios can be enabled by swapping out some of the namelist files from the historical experiment.

The CNTLATM and INITHIS UM namelist files need to be replaced with the corresponding RCP versions. The different versions of these files can be found in the directory /projects/access/data/ACCESS_CMIP5/um_control/hg2-r1.1-M2_Hist.

The RCP scenarios should use the files (replace RCP45 with RCP85 if required):

INITHIS.RCP45_ocff_hg2r11_dust
CNTLATM.template-RCP45_so2_ocff_hg2r11

These two files set the emssions forcings for the model.

To swap them out, change the lines in the *.init file:

cp INITHIS.Hist_ocff_hg2r11_dust   INITHIS

and

cp CNTLATM.template-Hist_so2_ocff_hg2r11 CNTLATM

to copy the template files appropriate for your experiment