Opened 4 years ago

Last modified 4 years ago

#213 new

APS3 Suite ODB tasks

Reported by: Yi Xiao Owned by: Yi Xiao
Priority: major Component: BOM ACCESS
Keywords: APS3 Suite ODB Cc: Tan Le, Zhihong Li, Peter Steinle, Gary Dietachmayer, Robin Bowen, Jin Lee, Fabrizio Baordo, Susan Rennie


Need DA group to work on OPS rose stem

  • add bombufr apps for all obs types (can copy from kma's)
  • use pre-build OPS32 exe to create ODB from bufr files (for G & R)
  • create known good outputs (KGO)

Background: Adam Maycock and I have tried to create ODB from BOM bufr files, with limited success.
Now we have OPS32 available, it is time to revisit this. It is important that we have this task done
asap, estimated two weeks.

Attachments (3)

bufrSubtypes.xlsx (13.1 KB) - added by Fabrizio Baordo 4 years ago.
Updated table bufr/obs type
CreateODB_update.docx (35.4 KB) - added by Yi Xiao 4 years ago.
CreateODB_update.20151203.docx (43.7 KB) - added by Yi Xiao 4 years ago.

Download all attachments as: .zip

Change History (28)

comment:1 Changed 4 years ago by Yi Xiao

Component: ACCESS modelBOM ACCESS

Moved this ticket to "BOM ACCESS"
Peter pointed out some phase in original request may not be very clear.

two weeks is the amount time required to do this work.
Not within 2 weeks of the request is made. Peter will organize a meeting on this.

comment:2 Changed 4 years ago by Yi Xiao

EWG has built OPS32 exe with rose stem, and the bin directory is in
Currently, the built was done using odb/30.0.0. However some limited test indicated
exe here does allow for some basic testing, such as createODB.
In the coming week, once Tan is happy with new ODB module (see ticket 212 for details)
the bin directory will be updated.

EWG will be asking UKMO for a shared OPS32 branch to be created.

comment:3 Changed 4 years ago by Peter Steinle

Cc: Jin Lee Fabrizio Baordo Susan Rennie added

comment:4 Changed 4 years ago by Yi Xiao

The OPS branch is
(As emailed to Fabrizio about 10 days ago.

comment:5 Changed 4 years ago by Yi Xiao

To assist the work along this line, I have created tasks about creating odb and process obs in the rose stem. I also tested ATMS and surface.
ATMS appears to have works, surface clearly failed. Other types are not tested, and apps needs further modification. However with examples provided it should be much easier to modify the apps.
[20/11/2015]] Added an attachment to Summarize the status of CreateODB in OPS32
Additional fixes (See Tan's comments below) have been accepted by UKMO, and merged into trunk. A Shared branch has been created after the merge from trunk@858 at It is used to build
APS3 OPS excutables. The build directory is under /projects/access/nwpdir/share/APS3/OPS/vn32.1.0
Remaining problems:

  • sonde and amv's problem in ODB may have been fixed by Tan. Need to check in OPS (Xiao)
  • Disabled obs types: CrIS,MtsatClear,Windsat are to be fixed by Jin

Big Thanks to Tan has worked out (very complicated) all Obstypes for creating ODB (not Mtsatclear), plus a bug fix in OPS code.
Xiao has tested and continued to test in collaboration with UKMO.
Part of our modification has passed review at UK, and changes have been put on the OPS trunk. I have created a branch under Share (ticket #165)
the bin directory is under ~access/nwpdir/APS3/OPS This directory will be updated when our coed passed review fully.

I have spent sometime on rose-stem createODB tasks, and tidied up apps, and added a few more.
I have also started to create data for KGO. Inputs and KGO are under /g/data/access/OPS
This is where UM KGO are at.
For obs types that you are responsible for, please check if the created ODB are valid under KGO, and let others know outcomes

Last edited 4 years ago by Yi Xiao (previous) (diff)

comment:6 Changed 4 years ago by Fabrizio Baordo

Managing rose-stem tests

To make more flexible the use of rose-stem tests the following changes should be made:

1) OPS_CYCLE_TIME must be defined in "rose-stem/site/nci/common/suite-env.rc" as done by Met Office and not in the rose-app.conf of the single app.

This means that change the basedate is simply localized in 1 file only and we do not have to do multiple times in every single rose-app.conf

2) OPS_BUFR_INPUT_DIR is now specified in the rose-app.conf of the single app as follows:


Changes which we must introduce:
--> the path = "/g/data1/dp9/itf/arbufr/" should be writable for everyone
--> OPS_CYCLE_TIME should be introduced instead of manually set the basedate

Changed 4 years ago by Fabrizio Baordo

Attachment: bufrSubtypes.xlsx added

Updated table bufr/obs type

comment:7 Changed 4 years ago by Susan Rennie

Possibly better to have a rose stem input data archive separate from other archives, with all data copied to it from other archives. Archives used to provide input for routine cycling, like /g/data1/dp9/itf/arbufr, probably shouldn't be universal writable for safety.

comment:8 in reply to:  7 Changed 4 years ago by Peter Steinle

Replying to sjr548:

Possibly better to have a rose stem input data archive separate from other archives, with all data copied to it from other archives. Archives used to provide input for routine cycling, like /g/data1/dp9/itf/arbufr, probably shouldn't be universal writable for safety.

Agree - and am uneasy about world write in general. Can understand the ease of use, but need to be meticulous about recording changes and which commands are issued within these directories. This is not something we have a good record of doing. Is there another option (sudo?) which helps inadvertent destruction?

comment:9 Changed 4 years ago by Fabrizio Baordo

Replying to sjr548 and pjs548

I agree with the solution you describe when our rose-stem tests will be well-established and hopefully we will also have an official OPS manager.

My general thought is that in any development phase everyone should have a more flexible environment to manage input/output

comment:10 Changed 4 years ago by Fabrizio Baordo

Managing rose-stem tests

Also OPS_UMBACK_LIST is a path we mitgh manage in a more general way:


  • No user data dir
  • Using OPS_CYCLE_TIME wherever is possible

example from Met Office:


comment:11 Changed 4 years ago by Fabrizio Baordo

rose stem-tests

I introduced some changes in my OPS branch "r272_ops_at_bom" (copy of Xiao's branch r175_bom_nci) to localize the set up of the rose-stem tests in 1 file only. More adoptable to changes and more user-oriented (in my opinion).

Changes are:

1) 2 new env variables to set up base data and data source for the tests:

in "rose-stem/site/nci/common/suite-env.rc", I defined:

ROSE_STEM_DATA_NCI = /g/data/dp9/da/rosestem_data
OPS_CYCLE_TIME_nci = 2015031200

2) env variables in 1 are used in the "rose-app.conf" of the single rose-stem test to define the following necessary inputs:


3) To implement 1 and 2, I arbitrary created a data dir which contains the nessary data input for the rose-stem tests. The data dir is (ROSE_STEM_DATA_NCI):


accessible to any user and organized as follows:







At any time a new base date (OPS_CYCLE_TIME_nci) can be tested in rose-stem simply coping the right source of data in ROSE_STEM_DATA_NCI. For instance APS2 bgerr/cx_bkg can be copied from /g/data3/rr4/samnmc/access-g/

If you are happy with these changes, r175 Xiao's branch should be updated (merged with r272), e.g.:

Last edited 4 years ago by Fabrizio Baordo (previous) (diff)

comment:12 Changed 4 years ago by Yi Xiao

I have edited the ticket yesterday [12/10/2015] about the progress I made on this and further work required
for experts to check ODB's KGO. But no email was sent out about my update. So now I try modify the ticket
see if message can be sent out.

comment:13 Changed 4 years ago by Susan Rennie

Further information on this issue can be found linked off MOSRS ticket 1, in

comment:14 Changed 4 years ago by Fabrizio Baordo

My OPS branch has been merged with Susan's in order to have 1 branch which contains the rose-stem tests for ODBcreate developed so far:

They are

Developed/tested by Susan:

Developed/tested by Fabrizio:

I have also created a common dir where to store the logs which are generated by each rose-stem ODBcreate test.
The dir is as follows:


These logs might provide a reference to list the problems we have with each ODBCreate.
In my test cases:

Looks OK == I check that data are written into the ODB. My basic check is as follows

Number of obs in job.stats MUST BE THE SAME of the result of the following query
odbsql -q 'select count(lat) from hdr, sat where satellite_identifier == 224'

Does not work == ODB empty

ATMS looks OK
IASI looks OK, but a strange error in job.out (TO BE CHECK)
AIRS does not work
ATOVS does not work

comment:15 Changed 4 years ago by Tan Le

Jin approached me late last week and asked for the status of my work re createODB, with a suggestion that we could share the workload (Jin also kindly volunteered to look at MTSatClear).

from Xiao:
The branch that included our work at the time and been merged on OPS truck, and I have tested
has been informed all of you via ticket 165 and it is
the bin directory is under ~access/nwpdir/APS3/OPS

and summary of my work,

  1. There is a bug in the OPS code that prevents bufr with section 2 to be processed. Fixed with


!BOM 14/10/2015 Section2Present = bit 1 of octet 8 set to 1,
!BOM ie. 8th byte = 128 dec
!BOM IF (ICHAR(SingleBUFR(BufrPointer? + 10:BufrPointer? + 10)) > 128) THEN
IF (ICHAR(SingleBUFR(BufrPointer? + 8:BufrPointer? + 8)) >= 128) THEN

Section2Present = .TRUE.


This fixed up most sat data except AMV, CRIS, MTSATCLEAR, WINDSAT

  1. sfc/aircraft/sonde

These are BOM's specific bufr format retrieved from rtdb. For each obs type we will need to add a definition in



  1. CRIS

in pp/*nci_cris_global/rose-app.conf


metdbkeywords='PLATFORM 224'

in 2. MetDB_Bufr_Retrieval/apps/create_mdblseq/

CRIMSS ${METDB_BASE_DIR}/tables/bufr_localseq/iasi

in 3. OpsMod_TURBO/OpsMod_TURBO.f90

! MetDB BUFR Extraction details
!BOM INTEGER, PARAMETER :: LenRep? = 28672 !- size of CREP (o/p from


!BOM 28/10/2015 CRIMSS msg len can exceed 28K, need to ignore restriction in
INTEGER, PARAMETER :: LenRep? = 50000 !- size of CREP


! To ensure a new subtype is BUFR extracted add it below.
!BOM 26/10/2015 added def for CRIMSS
CHARACTER(len=8),PARAMETER :: BUFRSubType(NumBUFRSubtypes) = &



","AIREPS ", &


With CRIS (399 radiance channels) the number of elements in a bufr message is 1057, and with 30 obs packed to a message the array size required is 30*1057 > 28672, hence the need to increase LenRep?.

Note however this is not accepted by the MetOffice?. We will either exclude assimilating CRIS and wait for the met office for a better solution, or to re-encode the bufr file with less obs packed to a message (say 15).

  1. amv

in addition to elements_index/SATWIND, we will also need to change a few entries in tableb as some bufr ids are in conflict with ECMWF table

  1. status.

amv: createODB is OK but the content of ODB is wrong
sonde: high rejection rate
cris: required alternative solution
mtsatclear/windsat: yet to be done

comment:16 Changed 4 years ago by Jin Lee

Jin tested OpsProg_CreateODB for obs_type=mtsatclear:

  1. the build used is from the branch, fcm:ops.x/branches/dev/Share/r686_165_bom_nci_ops_32.0.0@709
  2. Base date/time is 20150615T0600Z

The task fails with the following message in stderr,

Bufr file or directory input not supported for subtype JMACSR
Error Summary: There has been 1 WARNING message.
gc_abort (Processor     0): Bufr file or directory input not supported for subtype JMACSR

The fact that the array variable, BUFRSubType(:) defined in OpsMod_TURBO/OpsMod_TURBO.f90 doesn't contain "JMACSR" seems to be causing this failure.

comment:17 Changed 4 years ago by Tan Le

I had a closer look at element_index/SATWIND and realized that the met office have already defined the bufr sequences for both JMA/EUMETSAT (sequence 2) and GOESBUFR (sequence 5).

So I believe a proper solution is to re-produce the amv data in their original format, rather than the current format as a direct mapping from the content of rtdb.

This approach has the advantage of

  1. Consistency in bufr input format for data retrieved from rtdb, and that of direct bufr input from their original met centres.
  2. There is no need to change element_index/SATWIND as the bufr sequences have been defined by UKMO
  3. There is no need to use our own tableb, as the re-constructed bufr uses bufr defs from UK's table

There is however an extra step to convert our amv bufr data to the original format used by other centres.

The program to do the amv conversion is on ngamai:$CWSHARE/bin/bufr_amv

Usage : bufr_amv inout.bufr output.bufr

The src code is in twister:~ttl/mars/bufr_amv. Let me know the appropriate place on accessdev/svn to put the code in.


bufr_amv - converts BOM's amv bufr as retrieved from rtdb to their
original bufr format


bufr_amv <input_bufr_file> <output_bufr_file>


converts Bureau's amv bufr files as retrieved from rtdb to their
original bufr formats

  1. amv254msg.* -> EUMETSAT's MSGWINDS: IUV???_EUMG_yyyymmddhhmn
  1. amv254.* -> EUMETSAT's ESACMW: IXC???_EUMS_yyyymmddhhmn
  1. amv160.* -> NESDIS's GOESBUFR: J?CX??_KNES_yyyymmddhhmn
  1. amv34.* -> JMA's JMAWINDS: IUC???_RJTD_yyyymmddhhmn
  1. local amv encoded as NESDIS's

as required by OPS32.0 elements_index/SATWIND bufr sequence 2 (jma/
eumetsat), and bufr sequence 5 (local/nesdis)


The program requires link to ECMWF's bufr decoding software or

comment:18 Changed 4 years ago by Susan Rennie

A while back, in an earlier version of the suite, I tried including WINDSAT in the OPS supported subtypes for bufr, just to see what happened.
I just ran a comparison between the bufr file, and the ODB. The observation values are the same in both files. So it doesn't look like any further changes to the OPS should be necessary for WINDSAT.

Changed 4 years ago by Yi Xiao

Attachment: CreateODB_update.docx added

comment:19 Changed 4 years ago by Susan Rennie

An issue with JMAWINDS has been observed. Some observations with longitude of 180° are excluded due to precision error resulting in a value > 180° on conversion from single (in bufr) to double (in OPS). More information can be found here

comment:20 Changed 4 years ago by Tan Le

I have been looking at the issue with AIRS for the case 20150428 00z and here is what I found:

number of messages = 2529
number of obs in a message = 35
=> total number of obs = 2529*35 = 88515
with average length of each message = 23000 (bytes/words?)

Now if we have LenRep? = 28672 as required by MetDB, and set

we have a total array size of 28672*200 = 5734400

which can accommodate 5734400/23000 = 250 messages

or 250*35 = 8750 obs

or roughly one tenth of total obs as per Xiao's summary.

so for this case, the required bufrbuffersize = (2529*23000)/28672 = 2028

or solution is to set bufrbuffersize to 2500

in summary, set bufrbuffersize to

AIRS: 2500
IASI: 4000
ATOVS: 2500
GPSRO: 500

with the exception of CRIS, the average length of bufr message is 29000 > 28672. splitting the obs does not seem to work, so the only solution is to increase LenRep?, or to use bufr2odb.

comment:21 Changed 4 years ago by Tan Le

I had another attempt at bufr_split to fix the problem with CRIS (message size > 28672).

This time I break down the number of obs in a message, thus effectively cut down the message size.

bufr_split [-n N] input.bufr

with N = 2 we have CRIMSS_1 and CRIMSS_2 as output, each of which has message size roughly reduced by half (from ~30000 to ~15000)

createODB worked with all obs accounted for.

Note that with N > 12 you will need to increase maxbufrbatchessubtype in rose-app.conf as well (currently set to 12)

Changed 4 years ago by Yi Xiao

comment:22 Changed 4 years ago by Jin Lee

BUOY elements index file fixed

There was a problem with the BUOY elements index file, which meant the values of dewpoint temperature at 2m and relative humidity in ODB were different to those in input Bufr files. This problem is now fixed by updating the elements index file for BUOY,


I tested the change by running my secondary suite, u-aa858 and then testing using Metview and odbsql. E.g. I used Metview to examine Bufr messages 2462 (buoy identifier 31053) and then compared the values in the Bufr message with what's in ODB by running following ODB SQL queries,

odbsql -q 'select buoy_identifier,initial_obsvalue from hdr,body,conv where buoy_identifier=31053 and varno=$td2m'


odbsql -q 'select buoy_identifier,initial_obsvalue from hdr,body,conv where buoy_identifier=31053 and varno=$rh2m'

The changeset is,

comment:23 Changed 4 years ago by Susan Rennie

At 20151207, pending changes to OPS branch to run tasks
1) BUOY/surface. pressure tendency/characteristic see MOSRS OPS ticket 222, changeset 1113 (Susan's branch) - will be put in trunk.
2) BUOY elements_index file modified, see MOSRS OPS changeset 1114 (Jin's branch).
3) WINDSAT, need to add WINDSAT to OpsMod_TURBO.f90 subtypes. Windsat looks ok.
3) CRIS, need to add CRIMSS to OpsMod_TURBO.f90 subtypes.

comment:24 Changed 4 years ago by Tan Le


added fix to MetDB_Bufr_Retrieval/lib/source/retpos.f90

! Counts for higher levels



!BOM 09/12/2015 MCOUNT can be zero for nested levels in GPSRO
!BOM check to avoid division by zero

If (ICOUNT > 0) Then



IREP(J) = 1
I = 0

End If


to fix a divide by zero error when the nested level number is zero. The obs is still written out to ODB but the data content is all issing.


to follow the procedure as per CRiS, ie.

  1. set metdbsubtype='JMACSR' in rose-app.conf
  1. add entry for JMACSR in MetDB_Bufr_Retrieval/apps/create_mdblseq/

JMACSR ${METDB_BASE_DIR}/tables/bufr_localseq/jmawinds

  1. add entry for JMSCSR in BUFRSubType OpsMod_TURBO/OpsMod_TURBO.f90, note: need to increase size of NumBUFRSubtypes as well.
  1. bufr_split modified to handle MTSATClear. Note some message lengths can be as large as 60K so need N = 4. Also suggest to set bufrbuffersize = 500

comment:25 Changed 4 years ago by Susan Rennie

GPSRO gives different results depending on the version of the mdb elements file. in APS2

Total number of GPSRO            =      696
 Total ODB GPSRO obs              =      696

 2015  6 15  6  GPSRO               696 obs 

GPSRO    696 observations
         139 had prior reject of whole report and are not included below.
           0 were surplus (near-duplicate) reports and are not included below.
           0 had other reject of whole report, they are included below.


Total number of GPSRO            =      696
 Total ODB GPSRO obs              =      696

 2015  6 15  6  GPSRO            185849 obs 

GPSRO 185849 observations
       28843 had prior reject of whole report and are not included below.
           0 were surplus (near-duplicate) reports and are not included below.
           0 had other reject of whole report, they are included below.

APS3 has 3 additional level elements LEVL_LTTD LEVL_LNGD LEVL_STLT_AZMH

Note: See TracTickets for help on using tickets.