Optimization Methods with Applications

This course aims to provide algorithmic and numerical tools for solving common computational optimization problems in science and engineering. The course will include: linear algebra reminder: norms, least squares, eigenvalue and singular value decompositions, optimization of quadratic problems. Convexity, iterative methods for unconstrained optimization (Steepest Descent, Newton, Quasi-Newton, Conjugate Gradients, subspace methods, BFGS), and line-search methods. Constrained optimization with equality and inequality constraints (method of Lagrange multipliers and KKT optimality conditions), linear and quadratic programming, penalty, barrier and projection methods. Duality. Splitting methods and ADMM. Introduction to stochastic optimization (SGD). Introduction to non-smooth and sparse optimization. Robust statistics for dealing with outliers. The assignments in this course will include writing computer programs for practically implementing and demonstrating the algorithms taught in this course.