Changes between Initial Version and Version 1 of access/BomAccessDocumentation/bom-conda/nwp-pytools

Jul 26, 2019 10:49:41 AM (12 months ago)
Vinod Kumar



  • access/BomAccessDocumentation/bom-conda/nwp-pytools

    v1 v1  
     1== Motivation ==
     2An effort to set up a centralised, consultative, consolidated python-conda environment has been undertaken in the BoM/NWP (Atmospheric Modelling and Data Assimilation) section, which encompass all the packages we use for our work. This shared conda environment can be an entirely new one or an environment layered upon an existing one (conda/analysis27-18.07 or conda/analysis3-18.04). Some of the potential advantages of having such a shared environment is:
     41. Writing codes which  are transferrable between team members.
     52. Eliminate potential for duplicate personal python environments
     62. Implicit sharing of information. Allows you to dodge issues that has been already tackled by someone else earlier (i.e., compatible packages already installed and tested).
     73. Save you from loading multitudes of python modules and figuring out compatibilities.
     85. Prevent further proliferation of python modules available under ~access/modules (as it stands there are about 141 python packages under ~access/modules).
     96. Possibly easy for ITOs to manage.
     107. Allows you to set up, test and use your own new/additional packages, on top of this environment. This allow you to still get on with your work, until an ITO find some time to install the requested package.
     12== Current problems encountered when using Python, Python packages, Python tools ==
     14=== PS global suites ===
     16* PS global suites use scitools for some tasks. Some examples of how scitools versions are used,
     18|| PS || scitools version used ||
     19|| || ||
     20|| PS41 global standard suite || production-os41-1 ||
     21|| PS42 global standard suite || scitools/production_legacy-os42-1 ||
     23  We have no way of knowing what Python version, Python packages and their versions and other Python tools and their versions were used.
     25=== ACCESS-C (UKV) suites ===
     27|| PS || scitools version used ||
     28|| || ||
     29|| PS43 MOGREPS-UK component trial suite || scitools/production_legacy-os42-1 ||
     31=== Global Evaluation Suite (GES) ===
     33* GES uses Python and its standard packages in addition to some UKMO-developed Python tools such as !VerPy and !BoiteNoire (a part of !VarPy)
     35* The GES suite sets up Python environment using either,
     38module load scitools
     41   or
     44module load scitools/default-previous
     47  But this gives no indication what packages are in the UKMO mofulefile, `scitools`. It's often an educated guess or trial-and-error when we port GES trying to figure out which versions of packages are used
     49* UKMO GES suite developers use these Python packages and tools quite indepedently of RES suite developers and so the Python environment in GES is different to the Python environment in RES
     51=== Regional Evaluation Suite (RES) ===
     53 The current version of RES in !MetOffice uses the following module
     57 After lengthy testing in relation to RES porting, Vinod/Wenming has set up the following module consisting of various python packages to run RES at NCI
     62=== obsmon ===
     64* Apart from standard library packages, direct use is made of {{{numpy}}}, {{{matplotlib}}}, {{{netCDF4}}}, {{{cartopy}}} and {{{odbserver}}}. odbserver is not available through conda download, it is installed separately.
     66* odbserver is sensitive to the compiler environment. If the version of gcc (for the underlying C++ code) is not compatible with the installation version then a {{{Exception: Can't find}}} error is raised.
     68* Cartopy is usually made available through iris and that has some compatibility issues as noted elsewhere. Some versions of cartopy have issues with some versions of matplotlib. See [ varpy ticket 67] and [ varpy ticket 185] for examples.
     70* Cartopy requires data files that are automatically downloaded into a user's {{{.local}}} directory if the are not found in a pre-installed location. This must be avoided as it will break any suites that do not run cartopy on a data-mover node (i.e., all of them) as shared and compute nodes do not have external access. A shared location for these files (that is defined in the installed package) is essential.
     72* The sphinx package, pep8, and associated modules are used for automatic documentation generation and code tidying respectively. These are not required in the operation of the code but they are for admin purposes in varpy. Sphinx and pep8 are part of the Met Office scitools environment.
     75== Vision ==
     77Our Python environment will be identical to UKMO `scitools` and so Python, Python packages and Python tools within GES, RES and obsmon will work at NCI  without any modifications and tweaking.
     79== A possible solution ==               
     81There's a discussion going on within UKMO and across the partnership about a portable Python environment (see the section under the heading, "Useful information" below). One of the ideas proposed is some kind of containerised environment, but whatever solution UKMO comes up it will take some time before we can take advantage. As an interim solution we propose that, 
     83At each release of a suite the relevant UKMO suite developer publishes following information (preferrably in a yml file),
     85* Python version used           
     86* a list of Python packages and their versions         
     87* a list of Python tools and their versions             
     89The agreement to do this and a new working practice can be discussed through TISD workpackage.
     91== Name of the environment ==
     93We propose "nwp-pytools" as the package name. There is a possibility that we could eventually end up with a total of 4 versions. There is a requirement to support both python2 and python3. Also, as a release strategy, it would be easier to manage if there is an unstable version for each of the above packages. The module could be potentially named as:
     97conda/nwp-pytools27-unstable or conda/nwp-pytools27-dev
     99conda/nwp-pytools3-unstable or conda/nwp-pytools3-dev
     102Note 1. It would be worth having a tag for a particular nwp-pytools version, indicating the Met Office "scitools" version it corresponds to. For example, the initial version of "nwp-pytools" seems to be working for suites which use "scitools/production-os-41-1" at the Met Office. Hence, a suggested name would be nwp-pytools27-production-os41-1 or nwp-pytools27-os41-1 (if collective preference is for a shorter name).
     104Note 2. The naming convention`unstable` is used by CoE but we may use `dev` instead since each `dev` release would be reasonably stable and would become the basis for the next 'os' release (?)
     106=== Documentation on the installation of each individual environment ===
     107[wiki:access/BomAccessDocumentation/bom-conda/nwp-pytools/nwp-pytools27 nwp-pytools27]