wiki:access/UserGuides/conda

Miniconda

Accessdev provides Python libraries to users using Miniconda. Conda has the advantage that it provides a sophisticated dependency resolution mechanism for libraries, avoiding the need to create modules and track dependencies for individual libraries.

The environments used by ACCESS are available on the Anaconda cloud.

Using Conda environments

The ACCESS Anaconda environments are available on Raijin and the NCI virtual desktops as well as Accessdev.

Note: Please use the virtual desktops or Raijin instead of Accessdev for data analysis

module use /g/data3/hh5/public/modules
module load conda/analysis3

ACCESS environments

The current list of environments can be found at https://anaconda.org/accessdev

For the full list of packages available in an environment run 'conda list'

analysis3 (default)

Python 3 libraries for analysing climate and weather data, including

  • numpy
  • scipy
  • xarray
  • iris
  • matplotlib
  • cartopy
  • seaborn
  • netcdf4

analysis27

As 'analysis3', but for Python 2.7

python3

A basic environment with Python 3, including

  • numpy
  • scipy

python27

As 'python3', but for Python 2.7

Jupyter/Ipython notebooks

The NCI virtual desktops are the best place to run Jupyter Notebooks from - they have access to the /g/data filesystem and are not as heavily used as the Raijin login nodes.

To run Jupyter notebooks on VDI:

module use /g/data3/hh5/public/modules
module load conda
jupyter notebook

Creating personal environments

Note: Conda environments can become really big. Consider asking access_help to install packages centrally

You can create your own Conda environments by using 'conda create'

module load conda

conda create -c conda-forge -n my-env python netcdf4
source activate my-env

It's good practice to download from the 'conda-forge' channel, since that is what the central environments uses. Packages will get downloaded into your own space in /short/$PROJECT/$USER/conda (/local/$PROJECT/$USER/conda on Accessdev). If that directory does not exist it must be created before this step, otherwise any additional packages will be installed under $HOME, which has a small disk quota (2GB per user).

Requesting libraries

Libraries can be requested to be added to the ACCESS environment by emailing access_help@….

Managing Environments

The environments are defined by environment files, which are a list of packages to install in YAML format.

The environment files are stored in the git repository https://github.com/ScottWales/conda-envs, let Scott know if you'd like commit access. Changes to the environment files in the repository are automatically uploaded to Anaconda by travis-ci.org.

The environment can then be updated manually by a user with write access with e.g.

conda env update accessdev/python2

Known Issues

The analysis conda environments are tested by importing every installed package.

There are some issues with this approach. In some cases code in a package is for testing only and will not import cleanly. These imports are flagged as exceptions.

In other cases there are known issues with some code from a package. If there is still valuable and usable functionality in the rest of the package we will ignore the erroneous imports.

Here are the list of exceptions:

Currently there is no testing of executables installed by Conda.

Last modified 21 months ago Last modified on Sep 14, 2017 1:47:31 PM