wiki:access/AccessFS

The Access File System

The ACCESS system installed at NCI makes use of a common directory structure to hold scripts, programs, libraries and ancillary data related to the ACCESS models. This shared system reduces duplication of stored files & allows for collaboration between the different organisations comprising ACCESS.

File system quotas

To ensure limited storage space is used effectively NCI enforces file system quotas for all users & projects. These quotas are generally flexible, if you ask access_help@… for more space with a decent reason for it you should be able to get it.

The shared file system

The shared file system is administered by members of BoM, CSIRO & the CoE. To request programs or libraries be installed please email access_help@….

File Systems

/projects/access (~access)

The access administration home directory, used to hold scripts and data for running the UM & other models. It is backed up.

/g/data/access

This is also mounted on accessdev so is used for files that are useful on both systems like model prebuilds. Not backed up so only files that can be regenerated or re-downloaded should be stored here.

Useful paths

~access/modules

~access/data

Used to store larger files, e.g. ancillary data sets. Historically this was on a different filesystem but there's no no real distinction between it and the main ~access directory.

~access/bin

Helpful scripts for working with the UM

~access/umdir

Holds settings files and small executables for different UM versions.

~access/data/ancil

Ancillary data for different UM configurations (mainly older versions)

~access/umdir/ancil/atmos

New location for ancillary files. Many Met Office suites expect these to be in $UMDIR/ancil/atmos.

File access control lists

File access control lists allow finer grained control than the normal unix user, group and other permissions. They're used here to ensure that all access users (members of the access group at NCI) can use files in ~access and that the several members of the access.admin group can maintain the directories.

As an example for a particular file

% ls -l ~access/data/ancil/access_v2/qrparm.mask 
-rw-rw-r--+ 1 saw562 access.admin 245760 Apr 20  2011 /projects/access/data/ancil/access_v2/qrparm.mask

% getfacl ~access/data/ancil/access_v2/qrparm.mask getfacl: Removing leading '/' from absolute path names
# file: projects/access/data/ancil/access_v2/qrparm.mask
# owner: saw562
# group: access.admin
user::rw-
group::rwx			#effective:rw-
group:access:r-x		#effective:r--
group:access.admin:rwx		#effective:rw-
mask::rw-
other::r--

Default FACL settings for directories should mean that all files created in ~access have read/write permission for the access.admin group and read permission for the access group.

If you have problems with file permissions send a message to access_help.

FACLs and /short/PROJECT and /home

The /short/$PROJECT directories normally have read permission only for project members which can make wider collaboration difficult. The CSIRO p66 and BOM dp9 projects have used FACLs so that all access members can see the top level directories. Individual users then have the option of making their /short/PROJECT/USER directories more open.

Note that this doesn't affect permissions of directories that have more restrictive file permissions like $HOME/.ssh. CHECK HOW THIS WORKS.

User file system

To ensure that simulations can be easily run by different users we encourage the use of a few standard directories for model runs and outputs. Users have access to two main file systems, /home & /short.

File Systems

/home/xxx/$USER

This is the home directory of a user. It is backed up, & by default has a file system quota of 2 GB.

/short/$PROJECT/$USER

This is the user's scratch space. It is not backed up, and files left untouched for some time may be deleted, with the assumption that they can be recreated by the user. Each project has a quota for /short space, shared by all the users of that project. For example, if you log in using project w35 then any files you create under /short will use w35's quota.

Useful paths

/home/xxx/$USER/um_outdir

Output from UM run scripts, error messages & resource usage statistics will appear in these files

/home/xxx/$USER/umui_runs

UM run scripts, the umui copies scripts here before submitting them to the queue

/short/$PROJECT/$USER/UM_ROUTDIR/$USER

UM build & run directories, output from UM runs is located here


Porting Considerations

It may be valuable to have a more structured approach to installing shared programs and libraries on the new machine - e.g. CAP is currently at ~access/ancil.vn7.9, ~access/cap, ~access/umdir/vn7.7/ancil.

To date logins to ~access have been somewhat ad-hoc, with little accountability

The difference between ~access and /data/projects/access is historical - both are on the same filesystem & backed up now

Little use of the module facility for switching between versions

Will the directory structure be compatible with vayu - keeping a fair bit of cruft - or will it be restructured - requiring (possibly automated) porting for each model.

Possibilities

  • Use sudo to get to the access account, rather than ssh
  • Use access control lists to limit the need for ~access
  • Only have a single base access directory rather than two
  • Mirror NCI's /apps, /apps/Modules structure under access
  • Have a specific location for ancillary data for each model

One option

/projects/access
                     apps
                            gcom
                                4.0
                                   lib
                                   include
                                4.2
                                   lib
                                   include
                            cylc
                                2.4.3
                                   bin
                                git
                                   bin
                     modules
                            gcom
                                4.0
                                4.2
                            cylc
                                2.4
                                git
                     ancil
                          um
                            access-1.3-cmip
                            access-1.3-amip
                            access-1.3-amip-N48
                          cable
                               access-1.3-cmip
                               some-cable-configuration

All directories viewable by users, each directory under apps has a documented owner(s) with write permissions who maintain that program/library. A limited number (~4?) of super users can sudo into the access account and edit anything in case of emergencies.

Data under ancil is separated by model (um,cable,mom), each with a set of administrators. Further separation should be by model configuration, each folder should correspond to a standard run, with the run owners also having write access. There may be benefit to having different subdirectories for different resolutions - note it will be hard to change once directories are set up and in use by simulations.

It would be nice if we could keep track of metadata for ancil files somehow - who created it, how, what jobs use it.

Should the global module system be used? Would need to consult with NCI to see if this is possible

Use Cases

Three types of users - ACCESS admin, Module admin & general users

Install a new module

A user wants a new piece of software/library installed for use by a subset of ACCESS users

  • Nominate a person/people to be the Module admin & handle installation, testing and maintenance
  • Send a request via helpdesk for a new module, providing description, contact, original source (provide a template to fill out?)
  • ACCESS admin creates an apps subdirectory & sets permissions & creates module subdirectory & module file
  • Module admin populates apps subdirectory & tests
  • Successful installation announced to target users

Upgrade an existing module

Use a program module

Use a library module

Last modified 4 years ago Last modified on Oct 26, 2016 5:04:48 PM