wiki:access/CentralAncillaryProgram

Central Ancillary Program

The Met Office Central Ancillary Program (CAP) creates external data files for the UM, e.g. land sea mask, orography, vegetation, soil. The CAP reads high resolution master data files, interpolates to the model grid and writes files in UM format.

We have a repository for the CAP at https://access-svn.nci.org.au/trac/cap/. Branches currently in use are 7.7 for generating New Dynamics files and 8.3 for generating ENDGAME files. There was also a significant change in the way the program is built, controlled and run between these versions.

CAP 8.3 documentation at https://accessdev.nci.org.au/umdocs/CAP_8.3/

CAP 7.7

https://access-svn.nci.org.au/trac/cap/browser/branches/dev/Share/VN7.7_local_changes

fcm co https://access-svn.nci.org.au/svn/cap/branches/dev/Share/VN7.7_local_changes ANCIL_VN7.7_local_changes

CAP requires the UM GCOM library but can work with most versions. Build configurations for vayu (still valid for raijin) and ngamai are in src/config/Ancil_Dev. The vayu one finds GCOM with a module while the ngamai version is hardcoded to use /g/sc/ophome/access/apps/gcom/GCOM3.5/ngamai/default_for_vn7.5/lib/.

The dev-ext.MACHINE.cfg files basically just specifies the build location ($WORKDIR/WORKING/linux here) while linux-ifort-MACHINE.cfg has the compiler options.

Building on raijin

module load intel-fc
module load intel-cc
module load openmpi/1.6.5
module use ~access/modules
module load gcom/4.5
module load fcm

fcm co https://access-svn.nci.org.au/svn/cap/branches/dev/Share/VN7.7_local_changes ANCIL_VN7.7_local_changes
cd ANCIL_VN7.7_local_changes/src/config/Ancil_Dev
fcm extract dev-ext.vayu.cfg

cd $WORKDIR/WORKING/linux
fcm build

% ls bin
ancil_aerosol  ancil_orog   ancil_smow    ancil_vegfracsoil
ancil_general  ancil_ozone  ancil_sstice  central_ancillary.exe
ancil_mask     ancil_slt    ancil_top     fcm_env.ksh

Scripts in bin (ancil_*) have been modified to set paths etc appropriately for raijin and ngamai. All copied to ~access/apps/cap/7.7/bin/.

Building on ngamai

module load intel
module load openmpi/1.6.5
module use ~access/modules
module load fcm

fcm co https://access-svn.nci.org.au/svn/cap/branches/dev/Share/VN7.7_local_changes ANCIL_VN7.7_local_changes
cd ANCIL_VN7.7_local_changes/src/config/Ancil_Dev
fcm extract dev-ext.ngamai.cfg

cd $WORKDIR/WORKING/linux
fcm build

% ls bin
ancil_aerosol  ancil_orog   ancil_smow    ancil_vegfracsoil
ancil_general  ancil_ozone  ancil_sstice  central_ancillary.exe
ancil_mask     ancil_slt    ancil_top     fcm_env.ksh

Configuration files

The CAP comes with a set of grid and option namelists for all the Met Office domains, e.g. ~access/apps/cap/7.7/control/namelist/grids/

Master data files

raijin:~access/data/master

The master dataset includes the GLOBE orography and 1 degree and 30 minute resolution. The representation of Australia in this has many errors and should not be used (even for global models). See HighResolutionAustralianOrography.

Instead the scripts have been altered to add an option GLOBE30_AUS and the original data file made inaccessible.

Testing and validation

A simple test is to generate the N48 mask and orography using the Met Office options (note that this includes some explicit changes to the mask from the file ~access/apps/cap/7.7/control/changes/mask/mask_changes.n48). Repository directory src/scripts/create_files contains the Met Office jobs for various grids, including the options used, e.g. https://access-svn.nci.org.au/trac/cap/browser/branches/dev/Share/VN7.7_local_changes/src/scripts/create_files/n48/mask.ll has

export MASKOPTS="-m4 -k1 -l -3"
module load cap
export DATASETS="mask"
export MASKOPTS="-m4 -k1 -l -3"
ancil_top -model n48 -queue express

This creates a batch job, $WORKDIR/ancil.vn7.7/n48/make_ancil_file and submits it. The script default job time and memory limits are suitable for creating the mask.

Compare with standard Met Office file distributed with the external release test job.

$UMDIR/vn8.5/normal/utils/cumf  ~access/data/UM_Input/vn8.2/ancil/atmos/land_sea_mask/igbp/v1/qrparm.mask qrparm.mask 

cumf_full output file should have

Field     1 : Stash Code    30 : LAND MASK (No halo) (LAND=TRUE)      : Level 88
88

Field     1 has     0 LOGICAL Differences
 OK

Orography (pack 2 option gives 32 bit fields to match the Met Office version)

export DATASETS="orog"
export OROGOPTS="-d GLOBE1"
ancil_top -model n48 -queue express -memory 4 -pack 2
$UMDIR/vn8.5/normal/utils/cumf  ~access/data/UM_Input/vn8.2/ancil/atmos/orography/globe1/v1/qrparm.orog qrparm.orog

This time the comparison is not quite exact.

Field     1 : Stash Code    33 : OROGRAPHY (/STRAT LOWER BC)          : Level 8888

ITEM=   251    0.128055529785156E+04 (40940238A0000000)    0.128055541992188E+04 (40940238C0000000)
ITEM=   468   -0.337153363227844E+01 (C00AF8E6A0000000)   -0.337153387069702E+01 (C00AF8E6C0000000)
 NUMBER OF DIFFERENT VALUES =                      2
 MAXIMUM DIFFERENCE=   1.220703125000000E-004  AT PT.                    251

CAP 8.3

This uses fcm make to build, so requires a more recent fcm (default module on raijin is ok). It uses some code from the UM and OPS repositories. In the Met Office set up it also builds GCOM but here we use our standard library.

Building on raijin

module load intel-fc
module load intel-cc
module load openmpi/1.6.5
module use ~access/modules
module load gcom/4.5
module load fcm

To extract the appropriate branch directly from the repository

fcm make -f https://access-svn.nci.org.au/svn/cap/branches/dev/Share/ANCIL8.3_nci/fcm-make.cfg

Alternatively to use a locally modified copy of the code or of the build configuration files

fcm co https://access-svn.nci.org.au/svn/cap/branches/dev/Share/ANCIL8.3_nci
cd ANCIL8.3_nci/src/config/Ancil_Dev

To use the local configuration files rather than the repository versions

export ANCIL_CFG_ARCH=$PWD/x86_64-ifort.cfg
export ANCIL_CFG_TYPE=$PWD/opt.cfg 
export ANCIL_CFG_BUILD=$PWD/build.cfg 
export ANCIL_CFG_CORE=$PWD/core.cfg 
export ANCIL_CFG_DEP=$PWD/dep.cfg 

To use the local extracted code (working copy) rather than the repository version

export ANCIL_REPOSITORY=$WORKDIR/ANCIL8.3_nci

Then run

fcm make

Either method extracts a local copy of the code in a subdirectory of the current directory called "extract" and mirrors it to $WORKDIR/ancil/build/Ancil_Dev/. To build

cd $WORKDIR/ancil/build/Ancil_Dev/
fcm make

This creates build/bin/central_ancillary.exe.

Running

The options for creating ancillary files are now in XML files. This is read by a wrapper script and used to set up the job that creates the file. See CAP_raijin

CAP 9.1

CAP is now built as part of a rose stem test.

Last modified 4 years ago Last modified on Oct 25, 2016 2:17:53 PM