Changes between Version 7 and Version 8 of access/BomAccessDocumentation/bom-conda


Ignore:
Timestamp:
Nov 7, 2019 2:58:51 PM (8 days ago)
Author:
Vinod Kumar
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • access/BomAccessDocumentation/bom-conda

    v7 v8  
    22
    33== Motivation ==
    4 An effort to set up a centralised, consultative, consolidated python-conda environment has been undertaken in the BoM/NWP (Atmospheric Modelling and Data Assimilation) section, which encompass all the packages we use for our work. This shared conda environment can be an entirely new one or an environment layered upon an existing one (conda/analysis27-18.07 or conda/analysis3-18.04). Some of the potential advantages of having such a shared environment is:
     4An effort to set up a centralised, consultative, consolidated python-conda environment has been undertaken in the BoM/NWP (Atmospheric Modelling and Data Assimilation) section, which encompass all the packages we use for our work. Some of the potential advantages of having such a shared environment is:
    55
    661. Writing codes which  are transferrable between team members.
    7 2. Eliminate potential for duplicate personal python environments
     72. Eliminate potential for duplicate personal python environments.
    882. Implicit sharing of information. Allows you to dodge issues that has been already tackled by someone else earlier (i.e., compatible packages already installed and tested).
    993. Save you from loading multitudes of python modules and figuring out compatibilities.
    10 5. Prevent further proliferation of python modules available under ~access/modules (as it stands there are about 141 python packages under ~access/modules).
     105. Prevent further proliferation of python modules available under ~access/modules on Raijin (as it stands there are about 141 python packages under ~access/modules).
    11116. Possibly easy for ITOs to manage.
    12127. Allows you to set up, test and use your own new/additional packages, on top of this environment. This allow you to still get on with your work, until an ITO find some time to install the requested package.
     13
     14== Vision ==
     15
     16Our Python environment will encompass all the packages that can be used by researchers for their individual need as well as to run NWP related suites/tools (e.g., GES, RES, obsmon) at NCI.
     17
     18== A possible solution ==               
     19               
     20There's a discussion going on within UKMO and across the partnership about a portable Python environment (see the section under the heading, "Useful information" below). One of the ideas proposed is some kind of containerised environment, but whatever solution UKMO comes up it will take some time before we can take advantage. As an interim solution we propose that, 
     21
     22At each release of a suite the relevant UKMO suite developer publishes following information (preferrably in a yml file),
     23
     24* Python version used           
     25* a list of Python packages and their versions         
     26* a list of Python tools and their versions             
     27                       
     28The agreement to do this and a new working practice can be discussed through TISD workpackage.
    1329
    1430== Current problems encountered when using Python, Python packages, Python tools ==
     
    7591
    7692
    77 == Vision ==
    78 
    79 Our Python environment will be identical to UKMO `scitools` and so Python, Python packages and Python tools within GES, RES and obsmon will work at NCI  without any modifications and tweaking.
    80 
    81 == A possible solution ==               
    82                
    83 There's a discussion going on within UKMO and across the partnership about a portable Python environment (see the section under the heading, "Useful information" below). One of the ideas proposed is some kind of containerised environment, but whatever solution UKMO comes up it will take some time before we can take advantage. As an interim solution we propose that, 
    84 
    85 At each release of a suite the relevant UKMO suite developer publishes following information (preferrably in a yml file),
    86 
    87 * Python version used           
    88 * a list of Python packages and their versions         
    89 * a list of Python tools and their versions             
    90                        
    91 The agreement to do this and a new working practice can be discussed through TISD workpackage.
    92 
    93 
    9493= Installations and Related Documentations =
    9594
     
    9998== Various family of environments ==
    10099* [wiki:access/BomAccessDocumentation/bom-conda/nwp-pytools nwp-pytools]
     100
     101== Gadi transition ==
     102
     103Gadi is the new NCI super computer going to be operational on 11 November 2018. It has been decided that the python environment in Gadi will be managed using conda, following up on the {{{nwp_pytools27}}} development on Raijin (see above). All the legacy python libraries on Raijin under {{{/projects/access/apps/pythonlib}}} will not be carried across.
     104
     105The python environments in Gadi will also be called nwp_pytools. After discussions between Milton, Jin and Vinod, it has been agreed that the naming scheme could be of the format nwp_pytools2/YYYY.MM and nwp_pytools3/YYYY.MM. That is, two different "package names" for python 2 and 3 variants, with each environment identified by installation date (similar to Intel compilers). Users will get the latest environment by loading the package without specifying a version (e.g. module load nwp_pytools2).
     106
     107It was found that for the same yml file, two versions of miniconda produce two different environments. This highlights the risk of depending on third party miniconda installations. Hence it is agreed that Bureau manages its own miniconda versions on Gadi. Wenming has already done this for the {{{conda/nwp_pytools27}}} environment on Raijin, where it used conda 4.4 (see section `Base installation`).
     108
     109The environment related locations proposed by Wenming is:
     110{{{
     111conda software: ~access/bom/conda/software
     112conda channel: ~access/bom/conda/channel
     113conda envs: ~access/bom/conda/env
     114conda installations: ~access/apps/miniconda,
     115modules files: ~access/modules/bom_conda_env
     116}}}
     117 
    101118
    102119