Changes between Version 1 and Version 2 of access/BomAccessDocumentation/bom-dp9-management/user_guide


Ignore:
Timestamp:
Jun 23, 2020 3:55:39 PM (5 months ago)
Author:
Wenming Lu
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • access/BomAccessDocumentation/bom-dp9-management/user_guide

    v1 v2  
    11== User Guide ==
    22
    3 This instruction guides users through the journey how to use on ''Gadi'' the module of ''**nci_admin**'' to check general information and settings for a project, top users and overview of a project, and the information about an individual user.
     3This instruction guides users on how to use  the module of ''**nci_admin**'' to check information for  a project, such as ''dp9'' on ''Gadi'', including,
     4 * general information on disk and computing resource
     5 * settings for disk space and inodes
     6 * top users in various categories
     7 * overview usage of a project 
     8 * history of computing usage
     9 * information for individual user
    410
    511[[BR]]
    612==== Load Module nci_admin
    7 You must load the correct module to be able to run any of the following commands to get corresponding information,
     13You must load the correct module to be able to run any of the following commands to get corresponding information on ''Gadi'',
    814{{{
    915module purge
     
    1218module load nci_admin/3.0.0
    1319}}}
    14 Note: The version jumps to 3.0.0 from 1.0.0  to denote that this version of ''**nci_admin**'' is Python3 ready.
     20Note: The version jumps to 3.0.0 from 1.0.0  to specify that this version of ''**nci_admin**'' is Python3 ready.
    1521
    1622[[BR]]
    1723==== Check Individual User
    18 This is to check the detailed information about a particular user,
    19 {{{
    20 nci_admin.py -p dp9 -u wml548
     24This is to check the detailed information about a core user,
     25{{{
     26nci_admin.py -p dp9 -u $USER
    2127}}}
    2228
     
    2834
    2935dp9
    30 /g/data/ usage  =    2424G  share = 1.08%
    31 /g/data/ iUsage =       0M  share = 0.77%
    32 mdss     usage  =    3856G  share = 0.22%
    33 }}}
    34 
    35 By default, the script uses the current date to produce the latest information for the user. If aiming for a particular date, an extra argument is needed,
    36 {{{
    37 nci_admin.py -p dp9 -u wml548 -d 20200401
     36/g/data/ usage  =    2605G  share = 1.16%
     37/g/data/ iUsage =       0M  share = 0.80%
     38mdss     usage  =    3855G  share = 0.22%
     39/scratch usage  =  123929G  share = 15.13%
     40Service Unit usage  = 2680.00KSU
     41}}}
     42
     43By default, the script uses the current date to produce the latest information for the user. If aiming for a specific date, an extra argument is needed,
     44{{{
     45nci_admin.py -p dp9 -u $USER -d 20200401
    3846}}}
    3947
    4048If you are a guest user, those commands are,
    4149{{{
    42 nci_admin.py -p dp9 -u wml548 -c guest
    43 nci_admin.py -p dp9 -u wml548 -c guest -d 20200401
     50nci_admin.py -p dp9 -u $USER -c guest
     51nci_admin.py -p dp9 -u $USER -c guest -d 20200401
    4452}}}
    4553
    4654Note:
    47  * NCI only updates the usage once a day. Therefore, any changes will not be reflected in real time. Users have to come back the second day to get the details after changes.
    48  * Uers may get an error if the report for the date input does not exist. This applies to all commands in the later part of the guide.
    49  * NCI only updates ''**mdss**'' report for ''dp9'' quarterly (It is in talk whether a monthly report could be issued).
    50  * Users will get an error message if not a member of ''dp9'' or specifying wrong category of the user, i.e., using ''-c guest'' when being a core member and not using ''-c guest'' when actually a guest member.
     55 * NCI only updates the usage report once a day. Therefore, changes cannot be reflected in real time. Users have to come back the second day to verify the details of changes.
     56 * Uers may get an error if the report(s) for the date input does not exist. This applies to all commands in the later part of the guide.
     57 * NCI only updates ''**mdss**'' report for ''dp9'' monthly.
     58 * Users will get an error message if not a member of ''dp9'' or specifying wrong category of the user, i.e., using ''-c guest'' when being a core member and otherwise.
    5159
    5260[[BR]]
     
    5967{{{       
    6068=== Project: dp9 ===
    61 massdata Grant=1700T Usage=1189T(69%) Avail= 510T iGrant=  82M iUsage=   0M( 0%) iAvail=  82M
    62 gdata1a  Grant= 220T Usage= 175T(79%) Avail=  44T iGrant=  45M iUsage=  26M(58%) iAvail=  18M
    63 }}}
    64 The total granted quota on disk and inode of ''**mdss**'' and ''**/g/data/**'' are displayed along with the usages. Please note that due to technical reasons, it is not possible to present the information for ''**mdss**'' inode usage.
    65 
    66 If you are aiming for a particular date, then likewise,
     69massdata        Grant=1700T Usage=1154T(67%) Avail= 545T iGrant=  82M iUsage=   0M( 0%) iAvail=  82M
     70gdata1a         Grant= 220T Usage= 167T(76%) Avail=  52T iGrant=  45M iUsage=  28M(62%) iAvail=  16M
     71scratch         Grant= 800T Usage= 335T(41%) Avail= 464T
     72SU              Grant= 9MSU Usage=8.4MSU(93%) Avail=0.6MSU
     73}}}
     74The total granted quota on disk and inode of ''**mdss**'', ''**/g/data**'', ''**/scratch**'' and computing resources in ''**SU**''  are displayed along with the usages. Please note that due to technical reasons, it is not possible to present the information for ''**mdss**'' inode usage.
     75
     76If you are aiming for a specific date, then likewise,
    6777{{{
    6878nci_admin.py -p dp9 -r 1 -d 20200401
    6979}}}
    70 However, please be noted that the report may not be available for the date and an error message will pop up.
     80However, please be noted that the report may not be available for the date and an error message will be printed out.
    7181
    7282[[BR]]
     
    7888with output as,
    7989{{{
    80 hhz548    198T
     90USER     USAGE
     91==================
    8192ttl548    162T
     93hhz548    152T
    8294ycx548    132T
    8395cit548    115T
    8496zhl548    111T
    85 cnf599     73T
     97cnf599     79T
    8698sjr548     72T
    8799mxk548     61T
    88100jrf548     43T
    89101zxs548     37T
    90 }}}
    91 
    92 To check top 10 users on ''**/g/data/**'' disk of ''dp9'' (output omitted),
     102==================
     103These are Top Users in MDSS of dp9 for Core Members.
     104}}}
     105
     106To check top 10 users on ''**/g/data**'' disk of ''dp9'' (output omitted),
    93107{{{
    94108nci_admin.py -p dp9 -r 2b
    95109}}}
    96110
    97 To check top 10 users on ''**/g/data/**'' inode of ''dp9'' (output omitted),
     111To check top 10 users on ''**/g/data**'' inode of ''dp9'' (output omitted),
    98112{{{
    99113nci_admin.py -p dp9 -r 2c
     114}}}
     115
     116To check top 10 users on ''**/scratch**'' disk of ''dp9'' (output omitted),
     117{{{
     118nci_admin.py -p dp9 -r 2d
     119}}}
     120
     121To check top 10 users on ''**SU**'' inode of ''dp9'' (output omitted),
     122{{{
     123nci_admin.py -p dp9 -r 2e
    100124}}}
    101125
     
    105129nci_admin.py -p dp9 -r 2b -d 20200401
    106130nci_admin.py -p dp9 -r 2c -d 20200401
     131nci_admin.py -p dp9 -r 2d -d 20200401
     132nci_admin.py -p dp9 -r 2e -d 20200401
     133}}}
     134
     135For guest users,
     136{{{
     137nci_admin.py -p dp9 -r 2a -d 20200401 -c guest
     138nci_admin.py -p dp9 -r 2b -d 20200401 -c guest
     139nci_admin.py -p dp9 -r 2c -d 20200401 -c guest
     140nci_admin.py -p dp9 -r 2d -d 20200401 -c guest
     141nci_admin.py -p dp9 -r 2e -d 20200401 -c guest
    107142}}}
    108143
     
    119154Max Individual Share of /g/data Disk = 15% (33T)
    120155Max Individual Share of /g/data inode = 15% (6.75M)
     156Max Individual Share of /scratch Disk = 30% (240T)
    121157Emergency Level of mdss Tape Usage = 90%
    122158Emergency Level of /g/data Disk Usage = 85%
    123159Emergency Level of /g/data inode Usage = 85%
     160Emergency Level of /scratch Disk Usage = 90%
    124161Emergency mdss Big User Threshold = 50T
    125162Emergency /g/data Disk Big User Threshold = 5T
    126163Emergency /g/data inode Big User Threshold = 2M
     164Emergency /scratch Disk Big User Threshold = 200T
    127165Guest /g/data/dp9 Usage <= 500G
    128166Guest /g/data/dp9 inode <= 1M
    129 Guest dp9 mdss Usage = 0T
    130 }}}
    131 The detailed explanation of those items,
    132  * Max Individual Share of mdss Tape = 10% (170T): for core users
    133  * Max Individual Share of /g/data Disk = 15% (33T): for core users
    134  * Max Individual Share of /g/data inode = 15% (6.75M): for core users
    135  * Emergency Level of mdss Tape Usage = 90%: trigger for sending auto notice to users by email
    136  * Emergency Level of /g/data Disk Usage = 85%: trigger for sending auto notice to users by email
    137  * Emergency Level of /g/data inode Usage = 85%: trigger for sending auto notice to users by email
    138  * Emergency mdss Big User Threshold = 50T: users with equal or higher usage will be notified by email at emergency
    139  * Emergency /g/data Disk Big User Threshold = 5T: users with equal or higher usage will be notified by email at emergency
    140  * Emergency /g/data inode Big User Threshold = 2M: users with equal or higher usage will be notified by email at emergency
    141  * Guest /g/data/dp9 Usage <= 500G: guest users can use up to 500G on ''**/g/data/dp9**''
    142  * Guest /g/data/dp9 inode <= 1M: guest users can have up to 1M files on ''**/g/data/dp9**''
    143  * Guest dp9 mdss Usage = 0T: guest users should not use ''**mdss**'' tape drive of ''dp9''
     167Guest /mdss dp9 Usage = 0T
     168Guest /scratch/dp9 Usage <= 5000G
     169}}}
     170You may go to [wiki:access/BomAccessDocumentation/bom-dp9-management/policy Policy: dp9 Resource Management] for detailed meaning of those settings.
     171
    144172
    145173[[BR]]
     
    149177nci_admin.py -p dp9 -r 4
    150178}}}
    151 will do the job. This command produces the total ''**/g/data/**'' disk usage and inode usage, and ''**mdss**'' tape usage by all ''dp9'' core users, followed by a table with four columns,
     179will print out the usages of all users in all categories,
    152180 * 1^st^: User NCI ID
    153  * 2^nd^: User's ''**/g/data/**'' disk usage (this column is used to order users in the table)
    154  * 3^rd^: User's ''**/g/data/**'' inode usage
     181 * 2^nd^: User's ''**/g/data**'' disk usage (this column is used to order users in the table)
     182 * 3^rd^: User's ''**/g/data**'' inode usage
    155183 * 4^th^: User's ''**mdss'' tape usage
     184 * 5^rd^: User's ''**/scratch**'' disk usage
     185 * 6^th^: User's ''**SU**'' usage
     186
    156187
    157188Here is a sample output,
    158189{{{
    159 === Core User Info for dp9 ===
    160 Total /g/data/dp9 Core Usage = 171T
    161 Total /g/data/dp9 Core inode = 25M
    162 Total dp9 mdss Core Usage = 1131T
    163 
    164   USER   GDATA_DISK      GDATA_INODE     MDSS TAPE
    165 jtl548   64422.0G          5362K          32933G
    166 cnf599   31641.6G           256K          75300G
    167 hhz548   18432.0G          1627K         202931G
    168 fs5892   12108.5G            88K           8271G
    169 ycx548   9431.8G           3527K         136102G
    170 mxk548   7017.0G            732K          62479G
    171 vvk548   4340.0G           1415K            247G
    172 iib548   4066.0G           2763K           2307G
    173 cit548   3878.0G            130K         118411G
    174 glr548   2608.0G           1404K           1529G
    175 wml548   2424.0G            344K           3856G
    176 xds548   2183.0G            186K          15527G
    177 jrf548   1994.0G            333K          44995G
    178 sjr548   1731.5G           1610K          74139G
    179 yxm548   1694.0G            147K          13575G
    180 slc548   1690.0G            133K           6682G
    181 ixd548   1134.9G            491K           2182G
    182 as2291   1050.3G            311K            631G
    183 mjn548   1007.0G           3317K              0G
    184 chs548   800.0G              23K          33640G
    185 zxs548   392.0G             821K          38162G
    186 mjw548   357.0G              22K              0G
    187 zhl548   342.0G             160K         113898G
    188 ttl548   166.4G             343K         166576G
    189 hqz548   151.0G               0K             98G
    190 sss548    65.2G               5K              0G
    191 pjs548    22.7G              35K           4154G
    192 jfl548     0.0G               0K              0G
     190<<<<<< Core User Info for dp9 >>>>>>
     191Core /g/data/dp9 Usage  = 163T
     192Core /g/data/dp9 inode  = 27M
     193Core /mdss dp9 Usage    = 1096T
     194Core /scratch/dp9 Usage = 333T
     195==================================================================================
     196USER     GDATA_DISK      GDATA_INODE     MDSS_TAPE       SCRATCH              SU
     197==================================================================================
     198jtl548    51314G           5475K          35839G           2938G              7KSU
     199cnf599    31641G            256K          81622G          11366G             12KSU
     200hhz548    18534G           1633K         156605G          43724G            841KSU
     201fs5892    12108G            264K           8271G              2G              0KSU
     202...
     203iib548     4068G           2803K           3599G           2188G           1240KSU
     204cit548     3904G            220K         118411G              0G              1KSU
     205...
     206...
    193207}}}
    194208
     
    204218}}}
    205219
    206 
     220[[BR]]
     221==== Produce SU Quarterly History
     222To get the history of ''**SU**'' usage from the beginning of the quarter to the current date,
     223{{{
     224nci_admin.py -p dp9 -r 5
     225}}}
     226{{{
     227YYYYYMMDD       USAGE           DAILY_AVG       QUOTA   PROJECTION
     228------------------------------------------------------------------
     22920200401        0.00MSU         2.29KSU         6.0MSU  0.2MSU
     23020200402        0.02MSU         11.54KSU        6.0MSU  1.1MSU
     23120200403        0.02MSU         7.20KSU         6.0MSU  0.7MSU
     232...
     233...
     23420200621        8.09MSU         98.66KSU        9.0MSU  9.0MSU
     23520200622        8.24MSU         99.28KSU        9.0MSU  9.0MSU
     23620200623        8.37MSU         99.64KSU        9.0MSU  9.1MSU
     237}}}
     238The last four columns denote the total ''**SU**'' used, average daily usage, ''**SU**'' quota and projected usage at the end of the quarter.
     239
     240[[BR]][[BR]]
    207241===== END OF THE GUIDE