Status of local implementation of UM versions
Testing UKMO PS34 Global N768L70 ENDGame Build and Run jobs
UNDER CONSTRUCTION - - Work in trying out ps34 and the writing of this documentation currently on-going
UKMO Documentation on Parallel Suite 34 (PS34)
- Documentation on PS34 is published on collab wiki at this url: http://collab.metoffice.gov.uk/twiki/bin/view/Support/ParallelSuite34
- Note: Login into collab wiki ( http://collab.metoffice.gov.uk ) is required to access PS34 page.
- Files accociated with "Global N768L70 (ENDGame)" have been downloaded and available on raijin and ngamai in:
- ~access/downloads/
- Note: Login into collab wiki ( http://collab.metoffice.gov.uk ) is required to access PS34 page.
Building PS34 Global executable
Requires patching standard UM VN8.5 with UM PS34 Patch and JULES PS34 Patch
azs's local um ps34 branch
- Apply patches from vn8.5_PS34_Global_and_EG_Configuration_patch.tgz
- Create branch from trunk at vn8.5 and apply patch.
- Branch URL=https://access-svn.nci.org.au/svn/um/branches/dev/axs599/um8.5_ps34
- Create branch from trunk at vn8.5 and apply patch.
- In finalising my local ps34 branch, the following also requires consideration
- https://access-svn.nci.org.au/svn/um/branches/dev/vn8.5/local_changes
- https://access-svn.nci.org.au/svn/um/branches/dev/vn8.5/metoffice_patches
- List of changes in ...vn8.5/local_changes compared to trunk@vn8.5
Modified files (14): fcm-make/meto-x86-ifort/inc/um-atmos.cfg fcm-make/meto-x86-ifort/inc/x86-ifort-mpich.cfg src/script/control/qsatmos src/script/control/make_parexe.pl src/script/control/qsresubmit src/script/control/qsoasissetup src/atmosphere/dynamics_advection/set_halos.F90 src/atmosphere/convection/shallow_conv-shconv5a.F90 src/atmosphere/convection/deep_conv-dpconv5a.F90 src/configs/machines/linux-ifort-nci/ext_libs/gcom_mpp.cfg src/configs/machines/linux-ifort-nci/ext_libs/netcdf.cfg src/configs/machines/linux-ifort-nci/ext_libs/gcom_serial.cfg src/configs/machines/linux-ifort-nci/ext_libs/drhook.cfg src/configs/machines/linux-ifort-nci/machine.cfg New files: 11 fcm-make/linux-ifort-nci/inc/um-scm.cfg fcm-make/linux-ifort-nci/inc/um-atmos.cfg fcm-make/linux-ifort-nci/inc/ifort-nci.cfg fcm-make/linux-ifort-nci/inc/um-utils.cfg fcm-make/linux-ifort-nci/um-scm-debug.cfg fcm-make/linux-ifort-nci/um-atmos-debug.cfg fcm-make/linux-ifort-nci/um-utils-safe.cfg fcm-make/linux-ifort-nci/um-scm-safe.cfg fcm-make/linux-ifort-nci/um-atmos-safe.cfg fcm-make/linux-ifort-nci/um-scm-high.cfg fcm-make/linux-ifort-nci/um-atmos-high.cfg- Merge local changes to my working copy of https://access-svn.nci.org.au/svn/um/branches/dev/axs599/um8.5_ps34 and commit it.
cd /g/sc/data/azs/ps34/um8.5_ps34 svn merge https://access-svn.nci.org.au/svn/um/branches/dev/vn8.5/local_changes svn commit
- Inspect https://access-svn.nci.org.au/svn/um/branches/dev/vn8.5/metoffice_patches
- This branch contains 2 modified sources
- src/script/control/qsoasissetup
- src/atmosphere/dynamics_advection/set_halos.F90
- Both already in "local_changes"
- This branch contains 2 modified sources
- Merge local changes to my working copy of https://access-svn.nci.org.au/svn/um/branches/dev/axs599/um8.5_ps34 and commit it.
azs's local jules ps34 branch
- Apply patch from JULES_um8.5_PS34_Global_Configuration_patch.tgz
- Create branch from trunk (tagged at vn8.5) and apply patch. URL=https://access-svn.nci.org.au/svn/jules/branches/dev/axs599/jules8.5b_ps34
Build job vajda
- Upload UKMO basis_dljub into vajda in accessdev's UMUI
- Apply local customisations:
---------------------------------------------------------------------------------------------------------------- Job 1: Accessdev-vajd.a "ps34_Build_and_forecast_job (from basis_dljub)" Job 2: Accessdev-vajd.x "ps34_Build_and_forecast_job (from basis_dljub) Orig" Date: 20150216 LONG COMPARISON ---------------------------------------------------------------------------------------------------------------- 00007: Entry box: Mail-id for notification of end-of-run 00008: Job vajd.a: Entry is set to 'axs599@accessdev.nci.org.au' 00009: Job vajd.x: Entry is set to 'nomail' 00010: 00012: Entry box: Specify alternative name 00013: Job vajd.a: Entry is set to 'vajd' 00014: Job vajd.x: Entry is set to 'umgl' 00015: 00017: Entry box: Target Machine user-id: 00018: Job vajd.a: Entry is set to '$USER' 00019: Job vajd.x: Entry is set to 'frpe' 00026: 00027: Check box: Change machine config file ($UM_MACHINE) 00028: Job vajd.a: Entry is set to 'ON' 00029: Job vajd.x: Entry is set to 'OFF' 00030: 00031: 00032: Check box: Change target machine name ($TARGET_MC) 00033: Job vajd.a: Entry is set to 'ON' 00034: Job vajd.x: Entry is set to 'OFF' 00035: 00036: 00037: Entry box: Repository directory containing FCM machine.cfg file 00038: Job vajd.a: Entry is set to 'linux-ifort-nci' 00039: Job vajd.x: Entry is inactive 00040: 00041: 00042: Entry box: Host name 00043: Job vajd.a: Entry is set to 'raijin.nci.org.au' 00044: Job vajd.x: Entry is set to 'hpc2e' 00045: 00046: 00047: Radio button: Define submission method 00048: Job vajd.a: Entry is set to 'PBS Pro (Raijin)' 00049: Job vajd.x: Entry is set to 'LoadLeveler' 00050: 00051: 00052: Entry box: Target machine name 00053: Job vajd.a: Entry is set to 'linux' 00054: Job vajd.x: Entry is inactive 00062: Entry box: DATAM : Define the directory for written output with time-stamped names 00063: Job vajd.a: Entry is set to '/short/$PROJECT/$USER/85/$RUNID' 00064: Job vajd.x: Entry is set to '$DATADIR/$RUNID' 00065: 00066: 00067: Entry box: DATAW : Define the directory for other output file 00068: Job vajd.a: Entry is set to '/short/$PROJECT/$USER/85/$RUNID' 00069: Job vajd.x: Entry is set to '$DATADIR/$RUNID' 00077: Differences in Table Hand edits 00078: 1,10c1,10 00079: < /g/data1/dp9/axs599/ps34/hand_edits/GL_HANDEDITS_8.5_stashc_DUSTPS32 Y 00080: < /g/data1/dp9/axs599/ps34/hand_edits/GL_HANDEDITS_8.5_foamblk Y 00081: < /g/data1/dp9/axs599/ps34/hand_edits/GL_HANDEDITS_8.5_SMNSout_7p5minTS Y 00082: < /g/data1/dp9/axs599/ps34/hand_edits/vn8.5_p2t_weight_fix.pl Y 00083: < /g/data1/dp9/axs599/ps34/hand_edits/vn8.5_eta_s_0.5.pl Y 00084: < /g/data1/dp9/axs599/ps34/hand_edits/vn8.5_sc_1361.pl Y 00085: < /g/data1/dp9/axs599/ps34/hand_edits/vn8.5_filter_cloud_tau0.01 Y 00086: < /g/data1/dp9/axs599/ps34/hand_edits/vn8.5_srf_agg.ed Y 00087: < /g/data1/dp9/axs599/ps34/hand_edits/vn8.5_emis_ssi_full.pl Y 00088: < /g/data1/dp9/axs599/ps34/hand_edits/vn8.5_EG_package_hack.ed Y 00089: --- 00090: > ~gmdd/um/handedits/vn8.5/GL_HANDEDITS_8.5_stashc_DUSTPS32 Y 00091: > ~gmdd/um/handedits/vn8.5/GL_HANDEDITS_8.5_foamblk Y 00092: > ~gmdd/um/handedits/vn8.5/GL_HANDEDITS_8.5_SMNSout_7p5minTS Y 00093: > ~gmdd/um/handedits/vn8.5/vn8.5_p2t_weight_fix.pl Y 00094: > ~gmdd/um/handedits/vn8.5/vn8.5_eta_s_0.5.pl Y 00095: > ~gmdd/um/handedits/vn8.5/vn8.5_sc_1361.pl Y 00096: > ~gmdd/um/handedits/vn8.5/vn8.5_filter_cloud_tau0.01 Y 00097: > ~gmdd/um/handedits/vn8.5/vn8.5_srf_agg.ed Y 00098: > ~gmdd/um/handedits/vn8.5/vn8.5_emis_ssi_full.pl Y 00099: > ~gmdd/um/handedits/vn8.5/vn8.5_EG_package_hack.ed Y 00108: Entry box: Local machine root extract directory (UM_OUTDIR) 00109: Job vajd.a: Entry is set to '$HOME/UM_OUTDIR' 00110: Job vajd.x: Entry is set to '$HOME/um_extracts' 00111: 00112: 00113: Entry box: Target machine root extract directory (UM_ROUTDIR) 00114: Job vajd.a: Entry is set to '/short/$PROJECT/$USER/UM_ROUTDIR' 00115: Job vajd.x: Entry is set to '/data/nwp/nm' 00123: Entry box: Specify revision number or keyword of code base to use 00124: Job vajd.a: Entry is set to 'HEAD' 00125: Job vajd.x: Entry is inactive 00126: 00127: 00128: Check box: Use precompiled build 00129: Job vajd.a: Entry is set to 'OFF' 00130: Job vajd.x: Entry is set to 'ON' 00131: 00132: 00133: Check box: Include modifications from branches 00134: Job vajd.a: Entry is set to 'OFF' 00135: Job vajd.x: Entry is set to 'ON' 00136: 00137: 00138: Check box: Use different version of the UM code base from the default for this UMUI version 00139: Job vajd.a: Entry is set to 'ON' 00140: Job vajd.x: Entry is set to 'OFF' 00141: 00142: 00143: Entry box: The Subversion URL (UM_SVN_URL) 00144: Job vajd.a: Entry is set to 'https://access-svn.nci.org.au/svn/um/branches/dev/axs599/um8.5_ps34' 00145: Job vajd.x: Entry is set to 'fcm:um-tr' 00153: Entry box: Specify revision number or keyword of JULES code base 00154: Job vajd.a: Entry is set to 'HEAD' 00155: Job vajd.x: Entry is set to 'um8.5' 00156: 00157: 00158: Entry box: The Subversion URL (JULES_SVN_URL) 00159: Job vajd.a: Entry is set to 'https://access-svn.nci.org.au/svn/jules/branches/dev/axs599/jules8.5b_ps34' 00160: Job vajd.x: Entry is set to 'fcm:jules-tr' 00161: 00162: 00163: Check box: Include modifications from branches 00164: Job vajd.a: Entry is set to 'OFF' 00165: Job vajd.x: Entry is set to 'ON' 00173: Entry box: Filename for the Model executable 00174: Job vajd.a: Entry is set to '${RUNID}_um-atmos.exe' 00175: Job vajd.x: Entry is set to 'um-atmos.exe' 00176: 00177: 00178: Entry box: Filename for the Reconfiguration executable 00179: Job vajd.a: Entry is set to '${RUNID}_um-recon.exe' 00180: Job vajd.x: Entry is set to 'um-recon.exe' 00188: Check box: Including the following list of user file overrides 00189: Job vajd.a: Entry is set to 'OFF' 00190: Job vajd.x: Entry is set to 'ON' 00199: Differences in Table Specify the STASHmaster files 00200: 1,4c1,4 00201: < /g/data1/dp9/axs599/ps34/user_stashmaster/st_0_246 00202: < /g/data1/dp9/axs599/ps34/user_stashmaster/tca_up_to_6km 00203: < /g/data1/dp9/axs599/ps34/user_stashmaster/STASHmaster_thermal 00204: < /g/data1/dp9/axs599/ps34/user_stashmaster/eg_test_stmaster 00205: --- 00206: > ~gmdd/um/userstash/vn8.5/st_0_246 00207: > ~gmdd/um/userstash/vn8.5/tca_up_to_6km 00208: > ~gmdd/um/userstash/vn8.5/STASHmaster_thermal 00209: > ~gmdd/um/userstash/vn8.5/eg_test_stmaster 00210:
- On "Submit" qsub command not found
Submitting umui_runs/vajda-047163645/stage_1_submit via 'qsub' on raijin.nci.org.au /bin/bash: qsub: command not found MAIN_SCR: Submit failed
- Try adding "module load pbs" in .profile
- For now work-around by manually qsubbing on raijin
- Investigate if UMUIX setup on accessdev can be updated
- With manual qsub, job failed exceeding walltime
axs599@raijin4 5056> tail -18 /home/599/axs599/output/vajda000.vajda.d15047.t163647.comp.leave mpif90 -o ni_conv_ctl.o -I/short/dp9/axs599/UM_ROUTDIR/axs599/vajda/umatmos/inc -I/short/dp9/axs599/UM_ROUTDIR/axs599/vajda/baserepos/JULES/inc -I/short/dp9/axs599/UM_ROUTDIR/axs599/vajda/baserepos/JULES/inc -I/short/dp9/axs599/UM_ROUTDIR/axs599/vajda/baserepos/UMATMOS/inc -O3 -xHost -fp-model precise -g -traceback -mcmodel=medium -g -i8 -r8 -openmp -c /short/dp9/axs599/UM_ROUTDIR/axs599/vajda/umatmos/ppsrc/UM/atmosphere/convection/ni_conv_ctl.f90 ifort: command line warning #10212: -fp-model precise evaluates in source precision with Fortran. ifort: command line remark #10010: option '-pthread' is deprecated and will be removed in a future release. See '-help deprecated' =>> PBS: job killed: walltime 3647 exceeded limit 3600 make: *** [ni_conv_ctl.o] Terminated ====================================================================================== Resource Usage on 2015-02-17 15:00:51.891711: JobId: 9268436.r-man2 Project: dp9 Exit Status: 271 (Linux Signal 15) Service Units: 6.08 NCPUs Requested: 6 NCPUs Used: 6 CPU Time Used: 01:00:14 Memory Requested: 9000mb Memory Used: 664mb Vmem Used: 818mb Walltime requested: 01:00:00 Walltime Used: 01:00:49 jobfs request: 100mb jobfs used: 1mb ====================================================================================== axs599@raijin4 5057>
- With wall time increased significantly, build job finally succeeded in building um-atmos executable, but fail to build qxreconf executable.
/short/dp9/axs599/UM_ROUTDIR/axs599/vajda/umrecon/ppsrc/UM/control/misc/ukmo_grib_mod.f90(108): error #6404: This name does not have a type, and must have an explicit type. [ZHOOK_OUT] IF (lhook) CALL dr_hook('DECODE',zhook_out,zhook_handle) ---------------------------------^
- Seek advice from Scott Wales and Martin Dix
- Try out standard um8.5 build job
- This job (vajdy) built and ran successfully.
- Use vajdy to build qxreconf executable using ps34 source (from my branch).
- This also built successfully.
Reconfiguration job to re-instate ancil fields
Reconfiguration job to add ancil fields stripped from daily-downloaded UKMO initial conditions files qwqg00.reduced.YYYYMMDD400.T+3.gz
- Set up job vajdf from UKMO's dljtc
- from http://collab.metoffice.gov.uk/twiki/pub/Support/ParallelSuite34/basis_dljtc.rcf.um8.5.gz
- Ran into problem due to unrecognised namelists:
???????????????????????????????????????????????????????????????????????????????? ???!!!???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!???!!!? ? Error in routine: check_iostat ? Error Code: 19 ? Error Message: Error reading namelist temp_fixes. Please check input list against code. ? Error generated from processor: 0 ? This run generated 0 warnings ????????????????????????????????????????????????????????????????????????????????
- The above problem and similar namelist issues was solved by turning off all hand-edits in vajdf
- The job then complained about vertlev file
Vertical Levels file: /projects/access/umdir/vn8.5/ctldata/vert/vertlevs_L70_50t_20s_80km ???????????????????????????????????????????????????????????????????????????????? ???!!!???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!???!!!? ? Error in routine: Rcf_Read_Namelists ? Error Code: 80 ? Error Message: Vertical Levels Namelist file does not exist! ? Error generated from processor: 0 ? This run generated 1 warnings ????????????????????????????????????????????????????????????????????????????????
- Replace reference to vertlevs_L70_50t_20s_80km with vertlevs_L70_80km
- After that the reconf job went on to produce an astart files with 142 field types
- but alas eventually aborted complaining the absence of Field 418 Sec 0:
???????????????????????????????????????????????????????????????????????????????? ???!!!???!!!???!!!???!!!???!!!???!!! ERROR ???!!!???!!!???!!!???!!!???!!!???!!!? ? Error in routine: Rcf_Set_Data_Source ? Error Code: 30 ? Error Message: Section 0 Item 418 : Required field is not in input dump! ? Error generated from processor: 0 ? This run generated 1 warnings ????????????????????????????????????????????????????????????????????????????????
- According to STASHmaster, the field is "Dust parent soil clay fraction"
> grep 418 STASHmaster_A 1| 1 | 0 | 418 |Dust parent soil clay fraction (anc)|
- Study UMUI job vajdf again and found that the ancil settings to add "SOILDUST" is through Scientific section.
- Model Selection
- Atmosphere
- Scientific Parameters and Sections
- Section by section choices
- -- Section 17: Aerosols
- Follow-up panel "DUST"
- -- Section 17: Aerosols
- Section by section choices
- Scientific Parameters and Sections
- Atmosphere
- Model Selection
- Turn "dust" on. Enter $UM_ANCIL_SOILDUST_DIR & $UM_ANCIL_SOILDUST_FILE in relevant boxes
- Job ran much further but failed due to memory limitation.
/projects/access/umdir/vn8.5/linux/scripts/qsrecon: Executing dump reconfiguration program ********************************************************* RCF Executable : /short/dp9/axs599/UKD/ps34/bin/vajdy_qxreconf ********************************************************* =>> PBS: job killed: mem 22012688kb exceeded limit 8192000kb mpiexec: killing job... ====================================================================================== Resource Usage on 2015-02-25 15:40:07.653319: JobId: 9408219.r-man2 Project: dp9 Exit Status: 271 (Linux Signal 15) Service Units: 0.03 NCPUs Requested: 4 NCPUs Used: 4 CPU Time Used: 00:00:57 Memory Requested: 8000mb Memory Used: 21497mb Vmem Used: 30892mb Walltime requested: 00:10:00 Walltime Used: 00:00:28 jobfs request: 100mb jobfs used: 1mb ======================================================================================
- Even after significant increase in memory allocation, memory problem persist
- ... to be continued
UM model run job
- TO--BE--ADDED
UKD Suite
- TO--BE--ADDED
======================================================================================
Last modified 8 years ago
Last modified on Feb 25, 2015 8:40:01 PM
