TRAINING AS OPTIMIZATION.

GRADIENT DESCENT ITERATIVELY UPDATES PARAMETERS TO MINIMIZE A LOSS FUNCTION.

L(\theta)

\theta_1

\theta_2

OBJECTIVE

FIND PARAMETERS $\theta$ THAT MINIMIZE A LOSS FUNCTION $L(\theta).$

\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)

$\theta$: Parameters
$L(\theta)$: Loss function
$\nabla L(\theta)$: Gradient (slope)
$\eta$: Learning rate
$t$: Iteration step

01INITIALIZATION

L(\theta_0) = 2.49

Start from an initial parameter position.

02GRADIENT

\nabla L(\theta_0) = [1.2,\ -0.8]

Compute the gradient at the current position.

03PARAMETER UPDATE

\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)

Move a small step in the negative gradient direction.

04ITERATION

Repeat the update to gradually descend the loss surface.

05CONVERGENCE

L(\theta^*) = 0.03

Converge to a (local) minimum where $\nabla L(\theta^*) \approx 0.$

Training as Optimization

Gradient Descent in Parameter Space

TRAINING A MODEL CAN BE SEEN AS AN OPTIMIZATION PROBLEM. GRADIENT DESCENT ITERATIVELY UPDATES PARAMETERS TO MINIMIZE A LOSS FUNCTION, FOLLOWING THE LOCAL SLOPE OF THE LOSS LANDSCAPE.

LOSS OVER TIME

10^{1}

10^{0}

10^{-1}

10^{-2}

020406080100Iteration (

t

)

L(\theta)

LEARNING RATE

\theta_{t+1} = \theta_t - \eta \nabla L(\theta_t)

$\eta$ too largemay overshoot
$\eta$ too smallslow convergence
$\eta$ well chosenstable and efficient

LOSS LANDSCAPE (CONTOURS)

Contours show levels of equal loss. Gradient descent moves orthogonally to the contours toward lower values.