Opened 4 years ago

Closed 4 years ago

#189 closed (fixed)

Rationalise setting of Cylc initial environment for jobs

Reported by: Jin Lee Owned by: Martin Dix
Priority: major Component: ACCESS model
Keywords: TIWG Rose Cylc FCM job environment Cc:

Description

Whether a job is submitted to Raijin login nodes or to the PBS queues Cylc executes a script before starting a job. The script provides runtime environments for tasks. The relevant lines are in job scripts (e.g. accessdev:.../cylc-run/[ID]/log/job/[yyyymmddhh]/[task]/job):

# DIRECTIVES:
#PBS -N Ops_cris.201503
...
#PBS -l ncpus=32,mem=30G,walltime=00:15:00

...

prelude() {
# N.B. CYLC_DIR has been updated on the remote host
    export CYLC_DIR='/projects/access/apps/cylc/6.3.0'
    export CYLC_VERSION='6.3.0'
    for FILE_NAME in \
        "${HOME}/.cylc/job-init-env.sh" \
        "${CYLC_DIR}/conf/job-init-env.sh" \
        "${CYLC_DIR}/conf/job-init-env-default.sh"
    do
        if [[ -f "${FILE_NAME}" ]]; then
            . "${FILE_NAME}" 1>/dev/null 2>&1
            break
        fi
    done
}
prelude

prelude() {
# N.B. CYLC_DIR has been updated on the remote host
    export CYLC_DIR='/projects/access/apps/cylc/6.3.0'
    export CYLC_VERSION='6.3.0'
    for FILE_NAME in \
        "${HOME}/.cylc/job-init-env.sh" \
        "${CYLC_DIR}/conf/job-init-env.sh" \
        "${CYLC_DIR}/conf/job-init-env-default.sh"
    do
        if [[ -f "${FILE_NAME}" ]]; then
            . "${FILE_NAME}" 1>/dev/null 2>&1
            break
        fi
    done
}
prelude

...
...

# COMMAND SCRIPTING:
rose task-run --verbose

So before the command script runs Cylc executes one of the following scripts:

"${HOME}/.cylc/job-init-env.sh"
"${CYLC_DIR}/conf/job-init-env.sh"
"${CYLC_DIR}/conf/job-init-env-default.sh"

The precedence dictates that "${HOME}/.cylc/job-init-env.sh" is executed if it exists. If not then "${CYLC_DIR}/conf/job-init-env.sh" is executed and so on.

In our current Cylc set-up "${HOME}/.cylc/job-init-env.sh" and "${CYLC_DIR}/conf/job-init-env.sh" don't exist so "${CYLC_DIR}/conf/job-init-env-default.sh" is executed.

UKMO uses those scripts for setting initial environment for jobs: e.g. environment for running FCM. We should decide on what environment should go into which script.

Change History (4)

comment:1 Changed 4 years ago by Jin Lee

Here's a copy of an email from Matt Shin (IT Infrastructure/UKMO) on how to use those scripts for setting Cylc runtime environment:

At Met Office, the default "${CYLC_DIR}/conf/job-init-env-default.sh" is sufficient, because we have arrangements with our system admin teams to ensure that (wrappers of) FCM, Rose and cylc are all automatically in PATH.

You should copy "${CYLC_DIR}/conf/job-init-env-default.sh" to "${CYLC_DIR}/conf/job-init-env.sh", and add any site specific settings to the latter.

Matt

comment:2 Changed 4 years ago by Scott Wales

Keywords: TIWG added

comment:3 Changed 4 years ago by Martin Dix

Owner: set to Martin Dix
Status: newassigned

For 6.4.1 created job-init-env.sh with

. /etc/profile
. ~/.profile
module use ~access/modules
module load cylc/wrapper
# Load matching rose module for this version of cylc
module load rose/2015.04.1
module load fcm

For 6.3.0, load rose/2015.02.0 instead.

Cylc6 suites now run ok w/o requiring ~access/bin in the raijin path.

comment:4 Changed 4 years ago by Scott Wales

Resolution: fixed
Status: assignedclosed
Note: See TracTickets for help on using tickets.