Numerical Analysis I
AMSC/CMSC 666, Fall 2021
Course Information
(*) Additional notes on online class format are found here.
Course Description

Approximation Theory
 General overview. Leastsquares vs. the uniform norm
 Orthogonal polynomials
 Least squares I. Fourier expansions.
• Assignment
[ #1 ]
... with [ answers]
 Least squares II  the finitedimensional case. QR, SVD, PCA (LS rank approximations)
• Lecture notes:
[ SVD ]
(with additional notes on QR factorization )
• Additional reading:
► Gilbert Strang
[on SVD]
(and also here)
► Lijie Cao
[SVD applied to digital image processing]
• Assignment [ #2 ] with answers [ pdf file]
 Gauss quadrature

Numerical Solution of ODEs: InitialValue Problems

Preliminaries. Stability of systems of ODEs.
• Assignment
[ #3 ]
with answers [ pdf file]

Examples of basic numerical methods: Euler's method, LeapFrog, Milne.
• Additional notes:
►
[On Duhanel's principle]
► Why implicit Euler?
[On gradient flows]
• Assignment
[ #4 ]
with answers [ pdf file]

Consistency and stability imply convergence
• Assignment
[ #5 ]
... with answers [ pdf file]
 PredictorCorrector methods: AdamsBashforthMoulton schemes
• Assignment
[ #6 ]
 RungeKutta methods
 Local time stepping and error estimates. RK4 and RKF5.
 Stability and convergence of RungeKutta methods

Stiff systems and absolute stability
• Assignment
[ #7 ]
... with answers [ pdf file]

StrongStability Preserving (SSP) methods:
• Additional reading:
► SSP methods for
linear problems:[coercive problems] and [beyond coercivity];
and on SSP methods for [nonlinear problems]
Iterative Methods for Solving Systems of Linear Equations
 Stationary methods: Jacobi, GaussSeidel, SOR, ...
• Additional reading:
► The original work of [D. Young Thesis (1950)]
► On the Fourier approach to the SOR:
[Garabedian (1956)]
[LeVeque & Trefethen (1988)]
• Assignment
[ #8 ] ... with answers [ pdf file]
 Energy functionals and gradient methods
 Steepest descent. Conjugate gradient.
• Assignment [ #9 ]
... with answers [ pdf file]
 Acceleration method.
• Additional reading:
► Y. Saad [Iterative Methods for Sparse Linear Systems]
► Recent review of Krylovbased methods
[pdf file]
Numerical Optimization
 General overview. Fixed point iterations, loworder and Newton's methods
 Finding roots of nonlinear equations: scalar and systems. Homotopy
• Assignment [ #10 ]
... with answers [ pdf file]
 Computation of minimizers. Numerical Optimization
 Gradient flows
 Line search methods. Trust regions and Wolfe conditions
• Additional reading:
► J. Nocedal S. Wright
[Line search methods] and [Trustregion methods]
 Newton and quasiNewton: SR1 and BFGS methods
• Assignment [ #11 ]
Eigensolvers
 The power method
 The symmetric case: Rayleigh quotient. minmax
 Acceleartion. Inverse power method. Deflation
 Similaritybased methods
 Rotations and refelctions
 Jacobi method
 Reduction to Hessenberg and tridiagonal forms
 QR with a shift
• Additional reading:
► P. Deift, T. Nanda & C. Tomei
[Toda flows and the symmetric e.v. problem]
 Divide and Conquer method
• Additional reading:
► J.J.M. Cuppen
[Divide and conquer to tridiagonal eigenproblem] and [O(N^{2}) method for eigenvectors of Divide and conquer method]
• Additional reading
► Nick Trefethen
[
Numerical Analysis]
References
GENERAL TEXTBOOKS
W. Gautschi, NUMERICAL ANALYSIS, Birkhauser, 2012
K. Atkinson, An INTRODUCTION to NUMERICAL ANALYSIS, Wiley, 1987
S. Conte & C. deBoor, ELEMENTARY NUMERICAL ANALYSIS, McGrawHill
User friendly; Shows how 'it' works; Proofs, exercises and notes
G. Dahlquist & A. Bjorck, NUMERICAL METHODS, PrenticeHall,
User friendly; Shows how 'it' works; Exercises
A. Ralston & P. Rabinowitz, FIRST COURSE in
NUMERICAL ANALYSIS, 2nd ed., McGrawhill,
Detailed; Scholarly written; Comprehensive; Proofs exercises and notes
J. Stoer & R. Bulrisch, INTRODUCTION TO NUMERICAL ANALYSIS, 2nd ed., Springer
detailed account on approximation, linear solvers & eigensolvers,
ODE solvers,..
APPROXIMATION THEORY
E. W. Cheney, INTRODUCTION TO APPROXIMATION THEORY
Classical
P. Davis, INTERPOLATION & APPROXIMATION, Dover
Very readable
T. Rivlin, AN INTRODUCTION to the APPROXIMATION of FUNCTIONS
Classical
R. DeVore & G. Lorentz, CONSTRUCTIVE APPROXIMATION, Springer
A detailed account from classical theory to the modern theory; everything; Proofs exercises and notes
NUMERICAL INTEGRATION
F. Davis & P. Rabinowitz, NUMERICAL INTEGRATION,
Everything...
NUMERICAL SOLUTION Of INITIALVALUE PROBLEMS
E. Hairer, S.P. Norsett and G. Wanner, SOLVING ODEs I: NONSTIFF PROBLEMS,
SpringerVerlag, Berlin. 1991, (2nd ed)
Everything  the modern version
A. Iserles, A FIRST COURSE in the NUMERICAL ANALYSIS of DEs,
Cambridge Texts
W. Gear, NUMERICAAL INITIAL VALUE PROBLEMS in ODEs, 1971
The classical reference on theory and applications
Lambert, COMPUTATIONAL METHODS for ODEs, 1991
Detailed discussion of ideas and practical implementation
Shampine and Gordon, COMPUTER SOLUTION of ODES, 1975
Adams methods and practial implementation of ODE "black box" solvers
Butcher, NUMERICAL ANALYSIS of ODEs, 1987
Comprehensive discussion on RungeKutta methods
(mainly) ITERATIVE SOLUTION OF LINEAR SYSTEMS
A. Householder, THE THEORY OF MATRICES IN NUMERICAL ANALYSIS
The theoretical part by one of
the grand masters; Outdated in some aspects
G. H. Golub & Van Loan, MATRIX COMPUTATIONS,
The basic modern reference
Y. Saad, ITERATIVE METHODS for SPARSE LINEAR SYSTEMS,
PWS Publishing, 1996. (Available on line at
http://wwwusers.cs.umn.edu/~saad/books.html)
R. Varga, MATRIX ITERATIVE ANALYSIS,
Classical reference for the theory of iterations
James Demmel, APPLIED NUMERICAL LINEAR ALGEBRA,
SIAM, 1997
Kelley, C. T., ITERATIVE METHODS for LINEAR and NONLINEAR EQUATIONS,
SIAM 1995
NUMERICAL OPTMIZATION
J. Nocedal, S. Wright, NUMERICAL OPTIMIZATION
Springer, 1999
T. Kelly, ITERATIVE METHODS for OPTIMIZATION
SIAM
EIGENSOLVERS (mainly)
B. Parlett, THE SYMMETRIC EIGENVALUE PROBLEM
PrenticeHall, 1980
James Demmel, APPLIED NUMERICAL LINEAR ALGEBRA,
SIAM, 1997
J. H. Wilkinson The ALGEBRAIC EIGENVALUE PROBLEM, 1965
The classical reference
Eitan Tadmor

