Opened 4 years ago

Last modified 3 years ago

#203 assigned

Ops & Var refer to internal Met Office repos

Reported by: Scott Wales Owned by: Jin Lee
Priority: major Component: ACCESS model
Keywords: TIWG Cc: Jin Lee, Susan Rennie, Vinod Kumar, Fabrizio Baordo, Yi Xiao, Zhihong Li, Peter Steinle

Description

Jin has found issues with references to internal met office repos - blas, lapack, odb, odb-api, jules, drhook, nemo &c

He is discussing with UKMO on how to resolve

Change History (9)

comment:1 Changed 4 years ago by Scott Wales

Lapack etc to use centrally installed library e.g. MKL

ODB etc to have ACCESS copy of the relevant repositories from UKMO

FCM script modified to point to local repositories and installed libraries

Work with Met Office to provide as much interop as possible for above changes

comment:2 Changed 4 years ago by Jin Lee

Jin's email correspondence with David Davies (UKMO) below:

Hi,

Regarding ODB and ODB-API, these are just the tarballs (perhaps with a few modifications) that we get from Netlib or ECMWF. You should have these, if not I can send them, or in the case of Lapack/Blas? you can just download them from www.netlib.org. How you incorporate them into OPS is up to you. I think in the past BOM have built them separately and linked to them as libraries rather than tried to build them as part of FCM make; for us we mostly build ODB-API separately as a library. If you wish to have them available from a subversion repository the best thing would be to make a local subversion repository for them and use that.

I don't really want to dictate that any of these things are used from repositories. Yes I am aware that the current OPS config files could be more flexible from the point of view of allowing these things to be linked from external installations rather than built via fcm; I am happy to make changes to facilitate that. The reason we have built them via fcm in the past is that we build on several different compilers and a library built with one Fortran compiler cannot safely be linked to code built with a different Fortran compiler. Often these libraries are available on disk for some compilers but not all, just letting fcm build the code means we don't have to worry about libraries being available. Also we sometimes have to do development work on these codes, even if it is just inserting some write statements in the name of debugging, and it is easier to do this if the code is built via fcm than to have to constantly rebuild a separate library. These are reasons to build via fcm but many people prefer to just link to a library. GCOM is another example that falls in the category; we build GCOM via fcm but many codes, including here at the Met Office, use an on-disk library instead.

The Lapack and Blas dependency is there purely to satisfy an esoteric corner of RTTOV, it has no mainstream use. If it did we would almost certainly not compile like we do because we are using very unoptimised versions; we would use properly optimised libraries such as are available on most platforms.

Nemo is a slightly different case, my suggestion would be to not use it at all; unless you are doing ocean data assimilation and require the very particular output files that this code is used to produce you will not need it. It is possible to set the config files up to not require access to the Nemo code. If you *do* really need it I will need to have a chat with some people about the best way forward.

The surf revision you are using is a little older than current, the current version of surf uses the MetDB_RPC/BUFR/Bufr_Retrieval code in the current OPS trunk. So if you get a more recent version of surf you will not need access to those metdb_* repositories.

Finally I am nervous of going down the route of putting 3rd party codes in repositories available on MOSRS due to licensing and support issues. There is a difference between saying "OPS uses X third party software, get it from www...." and actually distributing it (which is what putting it on MOSRS amounts to). Long term if there are issues with these packages (and ODB in particular has required some modification for our use) we should be focused on getting our changes into the proper versions so that future official releases are easier to use, more functional and more portable, instead of trying to maintain our own patched versions. I appreciate that this will take time though.

David Davies

From: Jin Lee [J.Lee@…]
Sent: 29 July 2015 13:30
To: Bovis, Keir
Cc: Davies, David; Susan Rennie; Vinod Kumar
Subject: Met Office internal SVN repositories for building OPS, VAR and SURF [SEC=UNCLASSIFIED]
Hi Keir,

How are you? Hope things are not too hectic with PS36 and other matters.

While trying to build OPS32.0.0, VAR32.0.0 and SURF31.2.0 Susan, Vinod and I came across references to a Met Office internal repository in FCM config files that specify FCM extract.

For example, following lines are present in "common/core.cfg" for OPS,
$odb_branch = svn://fcm4/downloads_svn/odb/branches/dev/odb/r30_downloads11/Odb-1.0.0-Source
$odb_api_branch = svn://fcm4/downloads_svn/odb_api/branches/dev/frwd/r166_downloads37
$lapack_branch = svn://fcm4/downloads_svn/lapack/branches/dev/frwd/r113_LapackForOPS/lapack-3.5.0/SRC
$blas_branch = svn://fcm4/downloads_svn/blas/branches/dev/frwd/r112_BlasForOPS/BLAS-2011-04-19

...

$OPS_EXT_LOC_NEMO{?} = branches/test/frwd/vn3.4_FixNemo@$nemo_rev

So the source code packages which are not available to collaboration partners but which are required for OPS build are: ODB, ODB API, LAPACK, BLAS and NEMO

For SURF "common/options/extract_on.cfg" has following lines,

$metdb_rpc_br = svn://fcm4/downloads_svn/metdb_rpc/branches/dev/frwd/r128_MetDBRPCForOPS
$metdb_bufr_br = svn://fcm4/downloads_svn/metdb_bufr/branches/dev/frwd/r139_MetDBBufrForOPS
$metdb_bufr_retrieval_br = svn://fcm4/downloads_svn/metdb_bufr_retrieval/branches/dev/frbt/r187_r187_LongerLenRepForSurf

So for SURF the non-MOSRS software packages are "metdb_rpc", "metdb_bufr" and "metdb_bufr_retrieval".

We've been discussing how we handle this. The optimal solution is for us to be able to access the Met Office internal repository, "svn://fcm4/downloads_svn" but I guess this is not impossible. But before we start thinking about workarounds at our end - from a collaboration partners' point of view - I would like to hear your opinion. Do you have any suggestion which is possible to implement and which will serve us in the long run?

BTW a big thank-you for making the (unofficial) OPS and VAR MOSRS repositories available!

Cheers,

Jin
Satellite Data Assimilation
Centre for Australian Weather and Climate Research
Australian Bureau of Meteorology
Tel: +61 3 9669 4279

comment:3 Changed 4 years ago by Jin Lee

Cc: Jin Lee Susan Rennie Vinod Kumar Fabrizio Baordo added

comment:4 in reply to:  description Changed 4 years ago by Jin Lee

Replying to saw562:

Jin has found issues with references to internal met office repos - blas, lapack, odb, odb-api, jules, drhook, nemo &c

He is discussing with UKMO on how to resolve

David Davies offered a suggestion (see his email reply above).

Last edited 4 years ago by Jin Lee (previous) (diff)

comment:5 Changed 4 years ago by Yi Xiao

lapack was downloaded from netlib, and now is available at
https://access-svn.nci.org.au/svn/var/branches/local/extras/lapack/lapack-3.5.0

comment:6 Changed 4 years ago by Yi Xiao

Cc: Yi Xiao Zhihong Li Peter Steinle added

comment:7 Changed 3 years ago by Scott Wales

OPS and VAR repositories are now on the NCI mirror & available on accessdev as fcm:ops.xm and fcm:var.xm

comment:8 Changed 3 years ago by Scott Wales

Peter to raise issue with technical advisory TIDA project, will probably be affecting other sites

Jin continuing to work with UKMO to resolve

comment:9 Changed 3 years ago by Scott Wales

Owner: set to Jin Lee
Status: newassigned
Note: See TracTickets for help on using tickets.