Diagnosis of Inner Loop Minimisation in Global 4DVAR
Introduction
Past study
Some papers of interest,
 Courtier, Thepaut and Hollingsworth, 1994, QJ. A strategy for operational implementation of 4DVar, using an incremental approach
 Lawless and Nichols, 2006, MWR. InnerLoop Stopping Criteria for Incremental FourDimensional Variational Data Assimilation
Related work
Xingbao has a ticket which documents his investigation of 4DVAR convergence for ACCESSTCX (ticket #279).
After looking at its VAR settings there are a few questions which may require some thought:
 In our global 4DVAR  both nonhybrid and hybrid  multiincremental technique is used. The first minimisation at a coarser resolution tries to analyse the larger scales.
 4DVAR which analyses larger scales would better fit the assumptions of 4DVAR: PFM and its adjoint would more likely be a better approximation at larger scales; at larger scales errors in initial condition would grow more slowly so the nonincremental, fullfield (observation) cost function would suffer less from the effect of nonlinearity (Q. How does this relate to the cost function in incremental form? See Fisher (2012), lecture notes from Les Houches Summer School).
 Consequence of (a) is that the first multiincremental (observation) cost function would be exactly quadratic in the control variables (Q. Is this true?). This means there's exactly 1 minimum.
 The humidity control variable used in ACCESSTCX is relative humidity (Fortran namelist varaible, optionvp_mu=1 under Fortran namelist group, "var_transformnl"). This may not be suitable at higher resolution where moist processes become more important.
 At higher resolution moist processes become more dominant (Q. Is this true?) This means the assumptions about moisture control variable in deriving 4DVAR break down more at higher resolution.
 One of the assumptions of 4DVAR is that the state variables (consequently control variables????) have error distributions which are Gaussian. Relative humidity certainly doesn't obey Gaussian distribution (But the forecast error of relative humidity does????).
Questions
As a preliminary to the main investigation the penalties and the norms of their gradients are plotting for a typical 4DVAR run.
 suite ID: uac651
 cycle time: 20150621T12
 resolution of PFM: N108 and N216
 script used to plot: raijin6:/home/548/jtl548/da/var_diag/convergence/scripts/python/pen_gradpen/J_gradJ.py
The first of the following 2 plots shows the values of 3 penalties (total, background and observation) as a function of inner loop iteration count for N216. The second, norms of the gradients of corresponding penalty functions:
Following shows same quantities but for N108:
Note that the convergence behaviour is not that different from that of UKMO suite.
Aspects of the plots I don't understand:
 Why does the observation cost dominate the total cost?
 When the cost function seems to have converged to a local minimum why does the observation cost decrease so little from its value at the start of the inner loop? (Peter)
 At N216 why at certain iterations the cost and the norm of its gradient increase?
 At its face value it suggests that the minimisation moved away from a local minimum. So this means at that iteration the direction and step size were "wrong" and so this resulted in an increase in Jo,
Sum[(modelob[i]ob[i]^T) R^1 (modelob[i]ob[i]),{i,0,num_obs}]
modelobs and modelob^T are computed by PF and its adjoint. Does this suggest any problem with the linearised models?
 At its face value it suggests that the minimisation moved away from a local minimum. So this means at that iteration the direction and step size were "wrong" and so this resulted in an increase in Jo,
 What is the effect of misspecified background error covariance on convergence?
A posteriori statistics  Jo/Jb at minimum
Talagrand (Les Houches, 2012) discusses a posteriori statistics from a data assimilation system.
Michel (2014, NPG) has an alternative derviation of BennettTalagrand criterion.
Talking to Peter the stdev's in R for satellite observations may be larger than B. In our global 4DVAR satellite obs dominiate. So I did a test whether the ratio, Jo/Jb at minimum is changed when all satellite obs are removed.
 suite ID: uae335
 cycle time: 20150615T12
 resolution of PFM: N108
Gordon Inverarity and Marek Wlasak commented on the plots (see here).
Single observation test
Background information
A singleobservation test is a useful way to understand the characteristics of 4DVAR.
 An instruction from Sun on how to set up a singleob test
 VTDP 27  Single Ob Tests with the VAR & AC Schemes
My setup
 suite ID: uae207
Relevant VAR code and settings
This link describes what I think are the relevant parts of VAR code and some of VAR settings.
ToDo
 Try preliminary convergence plots for N106. J would be more smooth for the N106 resolution so search direction/step size would be less prone to be "caught" in "wrong" direction/step (?). So the cost function is more likely to decrease monotonically (?)
 Try global 4DVAR at a higher resolution: UM at N768, coarse PFM at N144, finer PFM at N320. If my hypothesis about nonlinearity(=nonquadratic cost function?) is right then at higher PFM resolution we would see poor convergence. Run the same base date/time as ACCESSTCX.
Attachments (4)

norm_grad_penalties.png (48.1 KB)  added by 5 years ago.
Norms of various gradients of penalties

penalties.png (26.4 KB)  added by 5 years ago.
penalties

penalty.n108.png (29.6 KB)  added by 5 years ago.
penalties for N108 PFM resolution

norm_grad_penalties.n108.png (40.2 KB)  added by 5 years ago.
Norms of various gradients of penalties for N108 PFM resolution
Download all attachments as: .zip