Linear Regression with Gradient Descent

How Gradient Descent Works for Linear Regression

Linear Regression Model

The model predicts: h_θ(x) = θ₀ + θ₁x

Cost Function (Mean Squared Error)

J(θ) = 1/(2m) ∑_i=1^m (h_θ(x⁽ⁱ⁾) - y⁽ⁱ⁾)²

Gradient Descent Algorithm

Repeat until convergence:

θ_j := θ_j - α ∂J(θ)/∂θ_j (for j = 0,1)

Partial Derivatives

∂J(θ)/∂θ₀ = 1/m ∑_i=1^m (h_θ(x⁽ⁱ⁾) - y⁽ⁱ⁾)

∂J(θ)/∂θ₁ = 1/m ∑_i=1^m (h_θ(x⁽ⁱ⁾) - y⁽ⁱ⁾)x⁽ⁱ⁾

Key Parameters

• Learning Rate (α): Controls step size (too small = slow convergence, too large = may diverge)
• Iterations: Number of optimization steps