Advanced Optimization: Theory and Applications

Course Outcomes

• Learn additional theory needed from calculus and linear algebra for optimization.
• Learn to model various applications from data science as an optimization problem.
• Learn to prove convergence estimates and complexity of the algorithms.
• Learn to code optimization solvers efficiently using Python.
• Demonstrate expertise in applying optimization methods in research problems.

This course teaches numerical optimization techniques to UG and PG students with applications from machine learning.

• Unit 0: Theoretical Foundation of Machine Learning and Role of Optimization Methods. (10 classes)
• Unit 1: Matric spaces, Topological Spaces, Hilbert Spaces, Convex Sets, Non expansiveness, Fejer Monotonocity and Fixed point iterations, Convex Cones
• Unit 2: Convex Functions, Lower Semi-continuity, Convex Minimization Problems, Conjugation, FR-Duality, Subdifferentiability
• Unit 3: Applications of advanced optimization: recommender systems, extreme classification, generative adversarial methods (4 classes)

References:

• Understanding Machine Learning, By Shai Shalev-Shwartz and Shai Ben-David, Cambridge University Press.
• Convex Analysis and Monotone Operator Theory in Hilber Space, Springer.
• Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press, 2004.
• Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016.
• Prateek Jain and Purushottam Kar, Non-convex Optimization for Machine Learning, 2017, arXiv.
• W. Hu, Nonlinear Optimization in Machine Learning.

Weightages:

• Assignments in theory: 15 marks, Mid Semester Examination: 25 marks, End Semester Examination: 30 marks, Assessment of four projects:  30 marks 

---