wiki:access/BomAccessDocumentation/bom-nwptools

Version 5 (modified by Jin Lee, 4 months ago) (diff)

--

BOM Conda Environment

Motivation

An effort to set up a centralised, consultative, consolidated python-conda environment has been undertaken in the BoM/NWP (Atmospheric Modelling and Data Assimilation) section, which encompass all the packages we use for our work. This shared conda environment can be an entirely new one or an environment layered upon an existing one (conda/analysis27-18.07 or conda/analysis3-18.04). Some of the potential advantages of having such a shared environment is:

  1. Writing codes which are transferrable between team members.
  2. Eliminate potential for duplicate personal python environments
  3. Implicit sharing of information. Allows you to dodge issues that has been already tackled by someone else earlier (i.e., compatible packages already installed and tested).
  4. Save you from loading multitudes of python modules and figuring out compatibilities.
  5. Prevent further proliferation of python modules available under ~access/modules (as it stands there are about 141 python packages under ~access/modules).
  6. Possibly easy for ITOs to manage.
  7. Allows you to set up, test and use your own new/additional packages, on top of this environment. This allow you to still get on with your work, until an ITO find some time to install the requested package.

Current problems encountered when using Python, Python packages, Python tools

Global Evaluation Suite (GES)

  • GES uses Python and its standard packages in addition to some UKMO-developed Python tools such as VerPy and BoiteNoire (a part of VarPy)
  • The GES suite sets up Python environment using,
module load scitools

But this gives no indication what packages are in the UKMO mofulefile, scitools. It's often an educated guess or trial-and-error when we port GES trying to figure out which versions of packages are used

  • UKMO GES suite developers use these Python packages and tools quite indepedently of RES suite developers and so the Python environment in GES is different to the Python environment in RES

Regional Evaluation Suite (RES)

  • ????

obsmon

  • ???

Vision

Our Python environment will be identical to UKMO scitools and so Python/Python? packages/Python tools within GES, RES and obsmon will work at NCI without any modifications and tweaking.

Name of the environment

We propose "bom-nwptools" as the package name. There is a possibility that we could eventually end up with a total of 4 versions. There is a requirement to support both python2 and python3. Also, as a release strategy, it would be easier to manage if there is an unstable version for each of the above packages. The module could be potentially named as:

conda/bom-nwptools27
conda/bom-nwptools27-unstable
conda/bom-nwptools3
conda/bom-nwptools3-unstable

Note: It would be worth finding out more about the CoE-CMS naming convention used for their conda packages.

Python 2.x.x environment

The following is an initial list of packages for conda/bom-nwptools27 environment, given in the format of yml file. There are obviously a few packages that are not in the channels. Claire recommend we create our own channel. Otherwise, this has to be installed out of the source code directly.

name: bom-nwptools27
channels:
  - anaconda
  - auto
  - coecms
  - conda-forge
  - defaults
  - jochym
  - scitools/label/archive
  - terradue
  - 
dependencies:
  - affine
  - afterburner
  - agg-regrid
  - alembic
  - appdirs
  - arccssive
  - arm_pyart
  - asciitree
  - asn1crypto
  - atomicwrites
  - attrs
  - backports
  - backports.functools_lru_cache
  - backports.lzma
  - backports.os
  - backports.shutil_get_terminal_size
  - backports.ssl_match_hostname
  - backports.tempfile
  - backports.weakref
  - backports_abc
  - basemap
  - basemap-data-hires
  - beautifulsoup4
  - blas
  - bleach
  - blosc
  - bokeh
  - boost-cpp
  - boto3
  - botocore
  - bottleneck
  - brewer2mpl
  - bsddb
  - bufr-metadb
  - bufr-transform
  - bzip2
  - ca-certificates
  - cairo
  - cartopy
  - cdat-lite
  - cdms2
  - cdo
  - certifi
  - cf_units
  - cfchecker
  - cffi
  - cftime
  - cfunits
  - cf-units
  - chardet
  - cis
  - click
  - click-plugins
  - cligj
  - cloudpickle
  - cmocean
  - cmor
  - colorcet
  - compliance-checker
  - configobj
  - configparser
  - cryptography
  - cryptography-vectors
  - curl
  - cycler
  - cyordereddict
  - cython
  - cytoolz
  - dask
  - dask-core
  - dataset
  - datashader
  - datashape
  - db
  - dbus
  - decorator
  - descartes
  - distributed
  - docopt
  - drslib
  - eccodes
  - ecmwfapi
  - ecmwf-api-client
  - entrypoints
  - enum34
  - esgf-pyclient
  - esmf
  - expat
  - f90nml
  - fastcache
  - fasteners
  - ffmpeg
  - fftw
  - fiona
  - fontconfig
  - freetype
  - freexl
  - fs
  - funcsigs
  - functools32
  - future
  - futures
  - g2clib
  - gdal
  - geopandas
  - geos
  - geotiff
  - gettext
  - giflib
  - glib
  - gmp
  - gmpy2
  - gnutls
  - gprof2dot
  - graphite2
  - graphviz
  - gsl
  - gst-plugins-base
  - gstreamer
  - gsw
  - gunicorn
  - h5netcdf
  - h5py
  - harfbuzz
  - hdf4
  - hdf5
  - hdfeos2
  - hdfeos5
  - heapdict
  - html5lib
  - icu
  - idna
  - imagehash
  - imageio
  - info.gianlucacosta.iris
  - ipaddress
  - ipykernel
  - ipython
  - ipython_genutils
  - ipywidgets
  - iris
  - iris_grib
  - iris-grib
  - isodate
  - jasper
  - jbig
  - jinja2
  - jmespath
  - joblib
  - jpeg
  - json-c
  - jsonschema
  - jupyter
  - jupyter_client
  - jupyter_console
  - jupyter_core
  - kealib
  - kiwisolver
  - krb5
  - lcms
  - libcurl
  - libdap4
  - libedit
  - libffi
  - libgcc
  - libgcc-ng
  - libgdal
  - libgfortran
  - libgfortran-ng
  - libiconv
  - libkml
  - libmagic
  - libmo_unpack
  - libnetcdf
  - libpng
  - libpq
  - libsodium
  - libspatialindex
  - libspatialite
  - libssh2
  - libstdcxx-ng
  - libtiff
  - libtool
  - libuuid
  - libxcb
  - libxml2
  - libxslt
  - linecache2
  - llvmlite
  - locket
  - lxml
  - lzo
  - mako
  - markupsafe
  - mask_tools
  - matplotlib
  - maybe
  - mdssdiff
  - mechanicalsoup
  - memory_profiler
  - metpy
  - mistune
  - mo_pack
  - monotonic
  - monty
  - more-itertools
  - mpc
  - mpfr
  - mpi
  - mpich
  - mpmath
  - msgpack-python
  - mule
  - multipledispatch
  - multiprocessing
  - munch
  - mysql-python
  - natgrid
  - nb_conda
  - nb_conda_kernels
  - nbconvert
  - nbformat
  - nccompress
  - ncdu
  - nchash
  - ncimonitor
  - ncl
  - nc-time-axis
  - ncurses
  - netcdf4
  - netcdf-fortran
  - nettle
  - networkx
  - normality
  - nose
  - notebook
  - numba
  - numcodecs
  - numexpr
  - numpy
  - olefile
  - openblas
  - openh264
  - openjpeg
  - openssl
  - owslib
  - packaging
  - pandas
  - pandoc
  - pandocfilters
  - pango
  - param
  - partd
  - pathlib
  - pathlib2
  - patsy
  - payu
  - pbr
  - pcre
  - pendulum
  - pep8
  - pexpect
  - pickleshare
  - pillow
  - pint
  - pip
  - pixman
  - pkg-config
  - pkginfo
  - plotly
  - pluggy
  - pooch
  - poppler
  - poppler-data
  - previous
  - proj4
  - prometheus_client
  - prompt_toolkit
  - psutil
  - psycopg2
  - pthread-stubs
  - ptyprocess
  - py
  - pybufr-ecmwf
  - pycairo
  - pycodestyle
  - pycparser
  - pyct
  - pydap
  - pyepsg
  - pyferret
  - pygeoif
  - pygments
  - pygrib
  - pyhdf
  - pykdtree
  - pyke
  - pyngl
  - pynio
  - pyopenssl
  - pyparsing
  - pyproj
  - pyqt
  - pyresample
  - pysal
  - pyshp
  - pysocks
  - pyspharm
  - pytables
  - pytest
  - python
  - python-bufr
  - python-cdo
  - python-dateutil
  - python-eccodes
  - python-editor
  - python-graphviz
  - python-magic
  - pytz
  - pytzdata
  - pywavelets
  - pyyaml
  - pyzmq
  - qt
  - qtconsole
  - readline
  - regex
  - regionmask
  - requests
  - requests-toolbelt
  - retrying
  - rpy2
  - rtree
  - s3transfer
  - scandir
  - scikit-image
  - scikit-learn
  - scipy
  - seaborn
  - send2trash
  - setuptools
  - sh
  - shapely
  - simplegeneric
  - singledispatch
  - sip
  - six
  - snuggs
  - sortedcontainers
  - sortedcontainers
  - sqlalchemy
  - sqlite
  - statsmodels
  - statsmodels
  - subprocess32
  - sympy
  - tabulate
  - tblib
  - tblib
  - terminado
  - testpath
  - tk
  - toolz
  - tornado
  - tqdm
  - traceback2
  - traitlets
  - trmm_rsl
  - twine
  - typing
  - tzlocal
  - udunits2
  - um_utils
  - umfile_utils
  - umtools
  - unittest2
  - urllib3
  - util-linux
  - virtualtime
  - vtk
  - wcwidth
  - webencodings
  - webob
  - wheel
  - widgetsnbextension
  - windspharm
  - wrapt
  - wrf-python
  - x264
  - xarray
  - xerces-c
  - xerces-c
  - xlrd
  - xorg-imake
  - xorg-kbproto
  - xorg-libice
  - xorg-libsm
  - xorg-libx11
  - xorg-libxau
  - xorg-libxaw
  - xorg-libxdmcp
  - xorg-libxext
  - xorg-libxmu
  - xorg-libxpm
  - xorg-libxrender
  - xorg-libxt
  - xorg-makedepend
  - xorg-renderproto
  - xorg-xextproto
  - xorg-xproto
  - xz
  - yamanifest
  - yaml
  - yaml
  - zarr
  - zeromq
  - zict
  - zlib

  - pip:
    - ants
    - fobis.py
    - gcircle==7.43
    - msgpack==0.5.6
    - pipedviewer==7.43
    - scientificpython
    - scitools-iris==2.0.0
    - tables==3.4.3
    - um-utils==2017.8.1

Useful information