Changes between Version 6 and Version 7 of access/JEDIAtNCI


Ignore:
Timestamp:
May 18, 2020 9:35:15 AM (3 years ago)
Author:
Jin Lee
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • access/JEDIAtNCI

    v6 v7  
    33In early 2020 UKMO formally commenced and committed substantial resources to the Next-Generation OPS (NG-OPS) and Next-Generation DA (NG-DA) projects. Both projects rely heavily on JEDI with UKMO concentrating development on components that it requires to implement its specific data assimilation schemes and observation processing. Their aim is to have JEDI-based DA system which is comparable to the current operational system ready by mid-2023 (the Bureau has not had access to the final project plans as of May 2020).
    44
    5 At time of this writing (May 2020) the Bureau's position is uncertain as to how to respond to the development. Anticipating that the Bureau will commit commensurate resources to JEDI-based NG-OPS and NG-DA projects the Data Assimilation Team has begun familiarisation with various components of JEDI itself. This is to enable early adopters to explore opportunities to collaborate with their UKMO colleagues so that when the Bureau formally decides to participate in the NG projects as a part of a partnership-wide effort we are ready to make contributions with minimum spin-ups.
     5At the time of this writing (May 2020) the Bureau's position is uncertain as to how to respond to the development. Anticipating that the Bureau will commit commensurate resources to JEDI-based NG-OPS and NG-DA projects the Data Assimilation Team has begun familiarisation with various components of JEDI itself. This is to enable early adopters to explore opportunities to collaborate with their UKMO colleagues so that when the Bureau formally decides to participate in the NG projects as a part of a partnership-wide effort we are ready to make contributions with minimum spin-ups.
    66
    77In this wiki page and the links in it we attempt to record various JEDI-related work undertaken by DA Team members.
    88
    9 == Software stack for running JEDI applications
     9== Building and installing software stack for running JEDI applications
    1010
    11 JEDI applications depend on a large number of software packages. The group of packages needed to run various JEDI applications is referred to as jedi-stack. The scripts used to build the packages are host in one of the JCSDA github repositories,
     11This section is for scientific computing support who will be maintaining the software stack used in running JEDI. Developers and users of JEDI subsystems and applications can skip this section.
    1212
    13 Jin LEE and Wenming LU did the initial build and installation of jedi-stack on Gadi. Building of the sotware packages is documented in a site-spefic document (????). The installation step is documented in jedi-stack github issue [https://github.com/JCSDA/jedi-stack/issues/93 #93]. Some of the the decisions made at the time of the initial build/installation were provisional and there is an ample room for improvement. We list below some of these decisions which are not documented in the github issue.
     13JEDI applications depend on a large number of software packages. The group of packages needed to run various JEDI applications is referred to as jedi-stack. The scripts used to build the packages are hosted in one of the JCSDA github repositories (https://github.com/JCSDA/jedi-stack).
    1414
    15 * The tcl modulefiles do not have tag names. This is ????
    16 *
     15Jin LEE and Wenming LU completed the initial build and installation of jedi-stack on Gadi. The name of the github branch used to build the software packages is [https://github.com/JCSDA/jedi-stack/tree/feature/gadi feature/gadi]. The site-specific instruction for building jedi-stack for Gadi can be found in a [https://github.com/JCSDA/jedi-stack/blob/feature/gadi/doc/platforms/gadi.md markdown]. The initial installation is documented in the jedi-stack github issue [https://github.com/JCSDA/jedi-stack/issues/93 #93]. Some of the the decisions made at the time of the initial build/installation were provisional and there is an ample room for improvement. We list below some of these decisions which are not documented in the github issue.
    1716
    18 == Running JEDI subsystems
     17* The environment variable, OPT determines the top-level of the installation directory. We decided to use,
     18  {{{
     19  export OPT=/projects/access/da/jedi
     20  }}}
     21
     22* The official releases of jedi-stack are now tagged. However we decided not to affix to modulefiles the tag as our jedi-stack pre-dates the decision to start tagging. We have need to use tags later to improve traceability (see further discussion in the github issue [https://github.com/JCSDA/jedi-stack/issues/93 #93] .
     23  * Mark Miesch (core development team, JCSDA, `miesch@ucar.edu`) suggested that a minor change to one or more packages of jedi-stack may not warrant a new release (see the github issue [https://github.com/JCSDA/jedi-stack/issues/93 #93]). Hence he recommended upgrading those specific packages manually and then creating a meta-module with a different name. We see a potential problem with this approach: deciding whether an upgrade to a package is minor or not may not be so straighforward as the minor change may affect other packages. We will have to see how well this initial jedi-stack installation supports the running of JEDI applications and prepared to revise the way our installations are managed and named.
     24
     25* We recommend scientific computing who will be maintaining jedi-stack to keep a close eye on the JEDI Teams page (https://github.com/orgs/JCSDA/teams/jedi) for any announcement related to jedi-stack. The official jedi-stack releases are in the github jedi-stack [https://github.com/JCSDA/jedi-stack/releases releases].
     26
     27== Running JEDI subsystems and applications
    1928
    2029First obtain a github account. However, this will not allow you to access all the private repositories.