wiki:access/BomAccessDocumentation/bom-dp9-management/user_guide

User Guide

This instruction guides users on how to use the module of nci_admin to check information for a project, such as dp9 on Gadi, including,

  • general information on disk and computing resource
  • settings for disk space and inodes
  • top users in various categories
  • overview usage of a project
  • history of computing usage
  • information for individual user


Load Module nci_admin

You must load the correct module to be able to run any of the following commands to get corresponding information on Gadi,

module purge
module use /projects/access/modules/
module load bom_envs/python_3.7.5
module load nci_admin/3.0.0

Note: The version jumps to 3.0.0 from 1.0.0 to specify that this version of nci_admin is Python3 ready.


Check Individual User

This is to check the detailed information about a core user,

nci_admin.py -p dp9 -u $USER

Here is the output,

Wenming_Lu
wenming.lu@bom.gov.au
wml548

dp9
/g/data/ usage  =    2605G  share = 1.16%
/g/data/ iUsage =       0M  share = 0.80%
mdss     usage  =    3855G  share = 0.22%
/scratch usage  =  123929G  share = 15.13%
Service Unit usage  = 2680.00KSU

By default, the script uses the current date to produce the latest information for the user. If aiming for a specific date, an extra argument is needed,

nci_admin.py -p dp9 -u $USER -d 20200401

If you are a guest user, those commands are,

nci_admin.py -p dp9 -u $USER -c guest
nci_admin.py -p dp9 -u $USER -c guest -d 20200401

Note:

  • NCI only updates the usage report once a day. Therefore, changes cannot be reflected in real time. Users have to come back the second day to verify the details of changes.
  • Uers may get an error if the report(s) for the date input does not exist. This applies to all commands in the later part of the guide.
  • NCI only updates mdss report for dp9 monthly.
  • Users will get an error message if not a member of dp9 or specifying wrong category of the user, i.e., using -c guest when being a core member and otherwise.


General Project Information

To check the general information of a project,

nci_admin.py -p dp9 -r 1  

The output is,

=== Project: dp9 ===
massdata        Grant=1700T Usage=1154T(67%) Avail= 545T iGrant=  82M iUsage=   0M( 0%) iAvail=  82M
gdata1a         Grant= 220T Usage= 167T(76%) Avail=  52T iGrant=  45M iUsage=  28M(62%) iAvail=  16M
scratch         Grant= 800T Usage= 335T(41%) Avail= 464T
SU              Grant= 9MSU Usage=8.4MSU(93%) Avail=0.6MSU

The total granted quota on disk and inode of mdss, /g/data, /scratch and computing resources in SU are displayed along with the usages. Please note that due to technical reasons, it is not possible to present the information for mdss inode usage.

If you are aiming for a specific date, then likewise,

nci_admin.py -p dp9 -r 1 -d 20200401

However, please be noted that the report may not be available for the date and an error message will be printed out.


List Top Users

To check top 10 users on mdss tape drive of dp9,

nci_admin.py -p dp9 -r 2a 

with output as,

USER     USAGE
==================
ttl548    162T
hhz548    152T
ycx548    132T
cit548    115T
zhl548    111T
cnf599     79T
sjr548     72T
mxk548     61T
jrf548     43T
zxs548     37T
==================
These are Top Users in MDSS of dp9 for Core Members.

To check top 10 users on /g/data disk of dp9 (output omitted),

nci_admin.py -p dp9 -r 2b

To check top 10 users on /g/data inode of dp9 (output omitted),

nci_admin.py -p dp9 -r 2c

To check top 10 users on /scratch disk of dp9 (output omitted),

nci_admin.py -p dp9 -r 2d

To check top 10 users on SU inode of dp9 (output omitted),

nci_admin.py -p dp9 -r 2e

Similarly, if you are interested in top users for any particular date, an extra argument is needed,

nci_admin.py -p dp9 -r 2a -d 20200401 
nci_admin.py -p dp9 -r 2b -d 20200401
nci_admin.py -p dp9 -r 2c -d 20200401
nci_admin.py -p dp9 -r 2d -d 20200401
nci_admin.py -p dp9 -r 2e -d 20200401

For guest users,

nci_admin.py -p dp9 -r 2a -d 20200401 -c guest
nci_admin.py -p dp9 -r 2b -d 20200401 -c guest
nci_admin.py -p dp9 -r 2c -d 20200401 -c guest
nci_admin.py -p dp9 -r 2d -d 20200401 -c guest
nci_admin.py -p dp9 -r 2e -d 20200401 -c guest


Check Current Project Setting

This is to check the current settings for the project,

nci_admin.py -p dp9 -r 3   

The settings are,

=== General Setting for dp9 ===
Max Individual Share of mdss Tape = 10% (170T)
Max Individual Share of /g/data Disk = 15% (33T)
Max Individual Share of /g/data inode = 15% (6.75M)
Max Individual Share of /scratch Disk = 30% (240T)
Emergency Level of mdss Tape Usage = 90%
Emergency Level of /g/data Disk Usage = 85%
Emergency Level of /g/data inode Usage = 85%
Emergency Level of /scratch Disk Usage = 90%
Emergency mdss Big User Threshold = 50T
Emergency /g/data Disk Big User Threshold = 5T
Emergency /g/data inode Big User Threshold = 2M
Emergency /scratch Disk Big User Threshold = 200T
Guest /g/data/dp9 Usage <= 500G
Guest /g/data/dp9 inode <= 1M
Guest /mdss dp9 Usage = 0T
Guest /scratch/dp9 Usage <= 5000G

You may go to Policy: dp9 Resource Management for detailed meaning of those settings.


Produce Usage Overview

To get an overview of usage on /g/data/ and mdss,

nci_admin.py -p dp9 -r 4

will print out the usages of all users in all categories,

  • 1st: User NCI ID
  • 2nd: User's /g/data disk usage (this column is used to order users in the table)
  • 3rd: User's /g/data inode usage
  • 4th: User's mdss tape usage
  • 5rd: User's /scratch disk usage
  • 6th: User's SU usage

Here is a sample output,

<<<<<< Core User Info for dp9 >>>>>>
Core /g/data/dp9 Usage  = 163T
Core /g/data/dp9 inode  = 27M
Core /mdss dp9 Usage    = 1096T
Core /scratch/dp9 Usage = 333T
==================================================================================
USER     GDATA_DISK      GDATA_INODE     MDSS_TAPE       SCRATCH              SU
==================================================================================
jtl548    51314G           5475K          35839G           2938G              7KSU
cnf599    31641G            256K          81622G          11366G             12KSU
hhz548    18534G           1633K         156605G          43724G            841KSU
fs5892    12108G            264K           8271G              2G              0KSU
...
iib548     4068G           2803K           3599G           2188G           1240KSU
cit548     3904G            220K         118411G              0G              1KSU
...
...

To get an overview for core users for a specific date,

nci_admin.py -p dp9 -r 4 -d 20200401

Similarly, to get an overview for guest users for the current date and a specific date,

nci_admin.py -p dp9 -r 4 -c guest
nci_admin.py -p dp9 -r 4 -c guest -d 20200401


Produce SU Quarterly History

To get the history of SU usage from the beginning of the quarter to the current date,

nci_admin.py -p dp9 -r 5
YYYYYMMDD       USAGE           DAILY_AVG       QUOTA   PROJECTION
------------------------------------------------------------------
20200401        0.00MSU         2.29KSU         6.0MSU  0.2MSU
20200402        0.02MSU         11.54KSU        6.0MSU  1.1MSU
20200403        0.02MSU         7.20KSU         6.0MSU  0.7MSU
...
...
20200621        8.09MSU         98.66KSU        9.0MSU  9.0MSU
20200622        8.24MSU         99.28KSU        9.0MSU  9.0MSU
20200623        8.37MSU         99.64KSU        9.0MSU  9.1MSU

The last four columns denote the total SU used, average daily usage, SU quota and projected usage at the end of the quarter.



END OF THE GUIDE
Last modified 7 weeks ago Last modified on Jun 23, 2020 3:55:39 PM