wiki:access/BomAccessDocumentation/bom-conda

Version 8 (modified by Vinod Kumar, 6 weeks ago) (diff)

--

BoM Conda Package and Environment Management

Motivation

An effort to set up a centralised, consultative, consolidated python-conda environment has been undertaken in the BoM/NWP (Atmospheric Modelling and Data Assimilation) section, which encompass all the packages we use for our work. Some of the potential advantages of having such a shared environment is:

  1. Writing codes which are transferrable between team members.
  2. Eliminate potential for duplicate personal python environments.
  3. Implicit sharing of information. Allows you to dodge issues that has been already tackled by someone else earlier (i.e., compatible packages already installed and tested).
  4. Save you from loading multitudes of python modules and figuring out compatibilities.
  5. Prevent further proliferation of python modules available under ~access/modules on Raijin (as it stands there are about 141 python packages under ~access/modules).
  6. Possibly easy for ITOs to manage.
  7. Allows you to set up, test and use your own new/additional packages, on top of this environment. This allow you to still get on with your work, until an ITO find some time to install the requested package.

Vision

Our Python environment will encompass all the packages that can be used by researchers for their individual need as well as to run NWP related suites/tools (e.g., GES, RES, obsmon) at NCI.

A possible solution

There's a discussion going on within UKMO and across the partnership about a portable Python environment (see the section under the heading, "Useful information" below). One of the ideas proposed is some kind of containerised environment, but whatever solution UKMO comes up it will take some time before we can take advantage. As an interim solution we propose that,

At each release of a suite the relevant UKMO suite developer publishes following information (preferrably in a yml file),

  • Python version used
  • a list of Python packages and their versions
  • a list of Python tools and their versions

The agreement to do this and a new working practice can be discussed through TISD workpackage.

Current problems encountered when using Python, Python packages, Python tools

PS global suites

  • PS global suites use scitools for some tasks. Some examples of how scitools versions are used,
PS scitools version used
PS41 global standard suite production-os41-1
PS42 global standard suite scitools/production_legacy-os42-1

We have no way of knowing what Python version, Python packages and their versions and other Python tools and their versions were used.

ACCESS-C (UKV) suites

PS scitools version used
PS43 MOGREPS-UK component trial suite scitools/production_legacy-os42-1

Global Evaluation Suite (GES)

  • GES uses Python and its standard packages in addition to some UKMO-developed Python tools such as VerPy and BoiteNoire (a part of VarPy)
  • The GES suite sets up Python environment using either,
module load scitools

or

module load scitools/default-previous

But this gives no indication what packages are in the UKMO mofulefile, scitools. It's often an educated guess or trial-and-error when we port GES trying to figure out which versions of packages are used

  • UKMO GES suite developers use these Python packages and tools quite indepedently of RES suite developers and so the Python environment in GES is different to the Python environment in RES

Regional Evaluation Suite (RES)

The current version of RES in MetOffice uses the following module

scitools/production-os41-1

After lengthy testing in relation to RES porting, Vinod/Wenming? has set up the following module consisting of various python packages to run RES at NCI

rmedtools/1.0.0

obsmon

  • Apart from standard library packages, direct use is made of numpy, matplotlib, netCDF4, cartopy and odbserver. odbserver is not available through conda download, it is installed separately.
  • odbserver is sensitive to the compiler environment. If the version of gcc (for the underlying C++ code) is not compatible with the installation version then a Exception: Can't find libOdb.so error is raised.
  • Cartopy is usually made available through iris and that has some compatibility issues as noted elsewhere. Some versions of cartopy have issues with some versions of matplotlib. See varpy ticket 67 and varpy ticket 185 for examples.
  • Cartopy requires data files that are automatically downloaded into a user's .local directory if the are not found in a pre-installed location. This must be avoided as it will break any suites that do not run cartopy on a data-mover node (i.e., all of them) as shared and compute nodes do not have external access. A shared location for these files (that is defined in the installed package) is essential.
  • The sphinx package, pep8, and associated modules are used for automatic documentation generation and code tidying respectively. These are not required in the operation of the code but they are for admin purposes in varpy. Sphinx and pep8 are part of the Met Office scitools environment.

Installations and Related Documentations

Base installations

Various family of environments

Gadi transition

Gadi is the new NCI super computer going to be operational on 11 November 2018. It has been decided that the python environment in Gadi will be managed using conda, following up on the nwp_pytools27 development on Raijin (see above). All the legacy python libraries on Raijin under /projects/access/apps/pythonlib will not be carried across.

The python environments in Gadi will also be called nwp_pytools. After discussions between Milton, Jin and Vinod, it has been agreed that the naming scheme could be of the format nwp_pytools2/YYYY.MM and nwp_pytools3/YYYY.MM. That is, two different "package names" for python 2 and 3 variants, with each environment identified by installation date (similar to Intel compilers). Users will get the latest environment by loading the package without specifying a version (e.g. module load nwp_pytools2).

It was found that for the same yml file, two versions of miniconda produce two different environments. This highlights the risk of depending on third party miniconda installations. Hence it is agreed that Bureau manages its own miniconda versions on Gadi. Wenming has already done this for the conda/nwp_pytools27 environment on Raijin, where it used conda 4.4 (see section Base installation).

The environment related locations proposed by Wenming is:

conda software: ~access/bom/conda/software
conda channel: ~access/bom/conda/channel
conda envs: ~access/bom/conda/env
conda installations: ~access/apps/miniconda, 
modules files: ~access/modules/bom_conda_env

Useful information

Things to do

  • Ask Mike Thurlow and Adam Martins what version of scitools were used in global and GES suites