Understanding Mathematical Regularization Techniques

Singular Value Decomposition

PEST: the book www.pesthomepage.org

Highly parameterized inversion Under predictive conditions: Under calibration conditions: s = w wtk k h h = Zk Zk +

Highly parameterized inversion h h = Zk Zk + = + h h k k Z Z

A reminder: What is the goal of regularization? To obtain a solution to the inverse problem that is: 1. unique 2. results in a parameter field of minimized error variance 3. (ideally) renders all predictions made by the model to be of minimized error variance

Manual regularization

Define a parsimonious parameter field p through the equation p1 k k = Np Np p2 h = h = Zk Zk + + 1 0 0 0 1 0 0 0 1 0 0 0 . 0 1 0 0 0 1 0 0 0 1 0 0 . 0 0 1 0 0 0 1 0 0 0 1 0 . 0 0 0 1 0 0 0 1 0 0 0 1 h = h = ZNp ZNp + + p3 = h = h = Xp Xp + + p p k k N N

h h = Xp Xp + = + X X p p h h

History-matching workflow for a well-posed inverse problem From another video h h = Xp Xp + Q Q = r2C-1( ) ) = (wiri)2 = = r rtQr Xp) )tQ Q(h = Qr = (h = (h Xp (h- -Xp Xp) ) p p = (X XtQX QX) )-1X XtQh Qh r2 = min/(n m) C(p p p p) = r2(X XtQX QX) )-1

Mathematical regularization

Two basic types of mathematical/numerical regularization: 1. Tikhonov 2. Singular value decomposition

Tikhonov Regularization h h = Zk Zk + = + Information on parameters and/or on relationships between parameters

Singular Value Decomposition h h = Zk Zk + = +

Singular Value Decomposition h h = Zk Zk + = + 1. Determine what combinations of parameters are estimable 2. Estimate only those parameter combinations 3. Leave the values of inestimable parameter combinations unchanged

Mathematical/Numerical Regularization Remarks Both Tikhonov regularization and SVD can provide a solution of minimum error variance to the inverse problem if certain conditions are met. Which to use? Normally both together SVD for numerical stability and Tikhonov to pursue a solution of minimum error variance in more flexible ways.

The Null Space

Equations are derived on a tablet

Singular Value Decomposition

Basic Equation h h = Zk Zk + = +

Singular Value Decomposition Z = Z = USV USVt = V Vt S S Z Z U U

The U U matrix u u1u u2u u3 etc Orthonormal matrix etc U UtU U = UU UUt = I I U U3 U U2 U U1 spanning model output space Orthogonal unit vectors .

Singular Value Decomposition Z = Z = USV USVt = V Vt S S Z Z U U

The V V matrix Z = Z = USV USVt V Vt V V

The V V matrix v v1v v2v v3 etc Orthonormal matrix v v3 etc V VtV V = VV VVt = I I k k3 k k1 k k2 v v1 v v2 spanning parameter space Orthogonal unit vectors .

Singular Value Decomposition Z = Z = USV USVt = V Vt S S Z Z U U

The S S matrix Off diagonal elements are zero.

The S S matrix Elements are positive, decreasing down the diagonal until they become zero or until the diagonal runs out. 0 0

Singular Value Decomposition V Vt S S Z Z U U = V V v v vectors span solution space v v vectors span null space

Orthogonal vector spaces v3 Null space v1 v2 Solution space

Singular Value Decomposition V Vt S S Z Z U U = S S1 S S2 Z = Z = USV USVt t ?1 ?2 ? = ? ?1 ?2 t V V Z = US Z = US1V V1t + US US2 2V V2 2t V V1 V V2 Z = US Z = US1V V1t

Minimum Error Variance Solution to Inverse Problem

Equations are derived on a tablet

Conclusions

Benefits of Singular Value Decomposition Solution of the inverse problem will never encounter a matrix that cannot be inverted. So the solution process will always be numerically stable. k k = V V1S S1-1U Uth h

Problem with Singular Value Decomposition To guarantee minimum error variance, its use should be preceded by a Karhunen-Lo ve transformation EFEt C C(k k) = EFE j j = F- E Etk k where Hence we estimate basis functions which are the columns of E E. These are eigencomponents of the covariance matrix of k k. But what if we are wrong about C(k) (as we always will be)? How can we learn something about spatial heterogeneity from the inversion process?

Solutions to these problems Use singular value decomposition to calculate k Use Tikhonov regularization to formulate inverse problem

The End