Linear Regression — exercises

With $n=5$: $\sum x=10,\bar{x}=2$, $\sum y=3.10,\bar{y}=0.62$, $\sum x^2=30,\sum xy=6.83$.

${\hat{\beta }}_1=(6.83-5\cdot 2\cdot 0.62)/(30-5\cdot 4)=0.63/10=0.063$ °C/year.

${\hat{\beta }}_0=0.62-0.063\cdot 2=0.494$ °C.

Warming rate: 0.63 °C/decade.

Reparametrize as $h={\beta }_0+{\beta }_1{t}^2$ with ${\beta }_0=h_0$ and ${\beta }_1=-g/2$. The design matrix has columns $[1,t^2]$, so $t^2\in \{0,0.04,0.16,0.36,0.64,1.0\}$.

Solving $X^{\top }X\hat{\beta }=X^{\top }h$ gives ${\hat{\beta }}_0\approx 5.00$, ${\hat{\beta }}_1\approx -4.892$.

So $h_0\approx 5.00$ m and $g=-2{\hat{\beta }}_1\approx 9.78$ m/s² — within 0.2% of the textbook value.

Design matrix $X$ with columns $[1,x,x^2]$:

X = [[1, -2, 4],
     [1, -1, 1],
     [1,  0, 0],
     [1,  1, 1],
     [1,  2, 4]]

Solving the normal equations gives $\hat{\beta }\approx (1.03,-2.03,1.00)$, close to the true $(1,-2,1)$ from $(x-1{)}^2=x^2-2x+1$.

The minor deviation is just the OLS estimator absorbing the noise. With noise-free data it would recover the true coefficients exactly.

Let $z_i={\log }_{10}({\text{income}}_i)$ and $w_i={\log }_{10}({\text{food}}_i)$. Compute $\sum z=8.584$, $\sum w=5.088$, $\sum z^2=15.042$, $\sum zw=8.964$.

Plug into the one-feature formula: ${\hat{\beta }}_1\approx 0.751$, ${\hat{\beta }}_0\approx -0.272$.

Elasticity ≈ 0.75: a 1% increase in income produces about a 0.75% increase in food spending. Sub-unitary, just as Engel observed in 1857. Anything $<1$ here is what makes food a "necessity good" in econ jargon.

(a) $\sum P=100,\bar{P}=20$, $\sum V=12.50,\bar{V}=2.50$, $\sum P^2=3000$, $\sum PV=351.4$. Plug in: $\hat{\beta }=(351.4-5\cdot 20\cdot 2.50)/(3000-5\cdot 400)=101.4/1000=0.1014$ V/kPa. $\hat{\alpha }=2.50-0.1014\cdot 20=0.472$ V.

(b) Invert: $P=(V-0.472)/0.1014$. For $V=3.0$: $P\approx 24.9$ kPa.

This is how every calibration curve in a lab works — fit once, use the inverse forever.

import numpy as np

rng = np.random.default_rng(0)
n  = 200
x1 = rng.normal(0, 1, n)
x2 = rng.normal(0, 1, n)
x3 = x1 + x2 + rng.normal(0, 1e-7, n)   # x3 ≈ x1 + x2

X  = np.column_stack([np.ones(n), x1, x2, x3])
y  = 1.0 + 2.0 * x1 + 1.0 * x2 + rng.normal(0, 0.1, n)

print("cond(XᵀX) =", f"{np.linalg.cond(X.T @ X):.2e}")
print("β (normal eq):", np.linalg.solve(X.T @ X, X.T @ y))
print("β (lstsq)   :", np.linalg.lstsq(X, y, rcond=None)[0])
# Predictions, not coefficients, are what's stable.

$\kappa (X^{\top }X)\sim 10^{14}$, near double-precision limits. The two solvers give different coefficients for $x_1,x_2,x_3$ — the per-column splits are wildly unstable, often differing by factors of $10^3$. But the predictions $\hat{y}=X\hat{\beta }$ agree to $\sim 10^{-4}$ between methods.

Reason: the data constrains linear combinations of $({\beta }_1,{\beta }_2,{\beta }_3)$ that lie in the row-space of $X$, not the individual coefficients. The null direction ${\beta }_3-{\beta }_1-{\beta }_2$ is essentially unconstrained, so any solver picks an arbitrary value there. The takeaway: when $\kappa$ is large, look at predictions, not coefficients.

Write the weighted loss as $\mathcal{L}(\beta )=(y-X\beta {)}^{\top }W(y-X\beta )$. Expand:

$\mathcal{L}=y^{\top }Wy-2{\beta }^{\top }{X}^{\top }Wy+{\beta }^{\top }{X}^{\top }WX\beta$

Gradient: ${\nabla }_{\beta }\mathcal{L}=-2X^{\top }Wy+2X^{\top }WX\beta =0$.

Solving: ${\hat{\beta }}_{\text{WLS}}=(X^{\top }WX{)}^{-1}{X}^{\top }Wy$. ✓

Equivalent recipe in practice: rescale each row $i$ of $X$ and $y$ by $1/{\sigma }_i$, then run plain OLS on the rescaled system. The rescaling makes the residuals equal-variance, which is what OLS optimality requires.

Aside: this is also the MLE under iid Gaussian noise with known per-point variance ${\sigma }_i^2$. Same algebra, different framing — the same answer drops out of the likelihood derivation.