Puppy on the slide,
Down steep slopes, with joyous glide,
Descent's swift, fun ride.
--ChatGPT
6. Optimization theory and algorithms#
In this chapter, we develop some basic optimization theory and algorithms for supervised learning.
The chapter has three main objectives:
To derive basic optimality conditions, including in the presence of convexity.
To implement optimization algorithms in the unconstrained case, in particular gradient descent.
To apply these methods to supervised learning.
Image credit: Made with Midjourney