Non-linear Regression

Not every relationship is a straight line. A thrown ball follows a parabola, a viral video’s views grow exponentially, and hours of daylight rise and fall with the seasons. Non-linear regression lets your GDC find the curve of a chosen type that fits the data best, just as linear regression finds the best line. On this page you’ll learn how the “best” curve is defined, how to measure how well it fits using $SS_{res}$ and $R^2$, and why the best number isn’t always the best model.

Key ideas

Least squares regression curves

For each data point $(x_i, y_i)$, the residual is the vertical distance from the point to the model:

$$\text{residual} = y_i - f(x_i) \qquad (\text{observed} - \text{predicted})$$

The sum of square residuals is

$$SS_{res} = \sum_{i=1}^{n} \big(y_i - f(x_i)\big)^2$$

A least squares regression curve is the curve of the chosen type with the smallest possible $SS_{res}$. You choose the type; technology finds the parameters. The types examined in IB AI are:

Regression	Model	GDC hint
linear	$y = ax + b$	linear regression
quadratic	$y = ax^2 + bx + c$	quadratic regression
cubic	$y = ax^3 + bx^2 + cx + d$	cubic regression
exponential	$y = ab^x$ (or $y = ae^{kx}$)	exponential regression
power	$y = ax^b$	power regression
sine	$y = a\sin(bx + c) + d$	sine regression (radian mode)

On most GDCs: enter the data in two lists, open the statistics calculation menu, choose the regression type, and read off the parameters (store the equation as a function so you can graph it and use it).

Two practical notes:

• Many GDCs find exponential and power regressions by fitting a straight line to the logged data, as on the linearizing data with logarithms page. Different software can therefore give slightly different parameters. The data on this page is chosen so that the methods agree to 3 s.f.
• Sine regression needs radian mode, and the answer can come out in different but equivalent forms (for example with a different $c$). On this page, $a \gt 0$ and $b \gt 0$.

SS_res as a measure of fit

For the same data, a smaller $SS_{res}$ means the model’s predictions are closer to the observed values. $SS_{res} = 0$ only if the curve passes through every point. Because $SS_{res}$ is in squared units of $y$ and grows with the number of points, compare $SS_{res}$ values only between models of the same data.

The coefficient of determination

The coefficient of determination $R^2$ is a number between $0$ and $1$ that your GDC reports with the regression (you may need to turn on “diagnostics” or “stat diagnostics”). Interpretation:

$R^2$ is the proportion of the variability in $y$ that is accounted for by the model.

For example, $R^2 = 0.94$ means $94\%$ of the variation in $y$ is explained by the model, and $6\%$ is not.

• For a linear model, $R^2 = r^2$, the square of Pearson’s correlation coefficient.
• It can help to know that $R^2 = 1 - \dfrac{SS_{res}}{SS_{tot}}$, where $SS_{tot} = \sum (y_i - \bar{y})^2$ measures the total variation in $y$. So $R^2 = 1$ exactly when $SS_{res} = 0$. (This formula helps understanding but isn’t examined.)

Choosing between models

$R^2$ is useful, but by itself it is not a good way to decide between models. Also think about:

• Context. Does the shape make sense? A drug concentration should decrease toward $0$, not rise again; a population can’t be negative.
• Behaviour beyond the data. Polynomials eventually head to $\pm\infty$. If you need to predict, the model must behave sensibly there.
• Simplicity. A cubic always has an $R^2$ at least as high as a quadratic on the same data (it has an extra parameter), but that doesn’t make it a better model.
• Residuals. A good model’s residuals are small and show no pattern.

Worked examples

Example 1: Quadratic regression for a thrown ball

The height $h$ metres of a ball is recorded $t$ seconds after it is thrown.

$t$ (s)	$0$	$0.5$	$1$	$1.5$	$2$	$2.5$
$h$ (m)	$1.6$	$7.7$	$11.5$	$13.1$	$11.8$	$8.4$

• (a) Find the quadratic regression model.
• (b) Find $SS_{res}$ and $R^2$, and interpret $R^2$.
• (c) Use the model to find the maximum height and when the ball lands.

Solution.

(a) Quadratic regression on the GDC gives

$$h = -4.85t^2 + 14.9t + 1.55 \qquad (\text{full values } {-4.85},\ 14.86214,\ 1.55357)$$

(b) The residuals (observed minus predicted, using full values) are

$t$	$0$	$0.5$	$1$	$1.5$	$2$	$2.5$
residual	$0.046$	$-0.072$	$-0.066$	$0.166$	$-0.078$	$0.004$

$$SS_{res} = 0.046^2 + (-0.072)^2 + \cdots + 0.004^2 \approx 0.0452$$

(using the unrounded residuals; the GDC can also compute this sum from a list of residuals)

The GDC gives $R^2 = 0.999$ (to 3 s.f.; more precisely $0.99948$). So $99.9\%$ of the variability in the height is accounted for by the quadratic model: an excellent fit.

(c) Graph the model on the GDC. The maximum is at $t \approx 1.53$ s, with $h \approx 12.9$ m. The ball lands when $h = 0$: the positive zero is $t \approx 3.17$ s. (The other zero, $t \approx -0.101$, is outside the domain $t \ge 0$.)

The leading coefficient $-4.85$ is close to $-4.9$, half the acceleration due to gravity, which supports the model.

Example 2: Exponential regression for video views

A new video’s daily views are recorded.

Day $x$	$1$	$2$	$3$	$4$	$5$	$6$
Views $y$	$216$	$331$	$512$	$786$	$1214$	$1867$

• (a) Find an exponential regression model $y = ab^x$ and interpret $b$.
• (b) Predict the views on day $8$.
• (c) On which day will the daily views first exceed $5000$, if the trend continues?

Solution.

(a) Exponential regression gives $a \approx 140$ and $b \approx 1.54$, with $R^2 = 1.00$ (to 3 s.f.):

$$y = 140(1.54)^x$$

$b = 1.54$ means the daily views are multiplied by about $1.54$ each day: an increase of about $54\%$ per day.

(b) Using the stored model with full values, $y(8) \approx 4430$ views (3 s.f.).

(c) Solve $140(1.54)^x = 5000$ with the GDC (using full values): $x \approx 8.28$. Since $x$ counts whole days, the views first exceed $5000$ on day 9. This is an extrapolation: real viral growth always slows down eventually, so this prediction should be treated with caution.

Example 3: Two models with almost the same R²

A patient’s blood concentration $C$ (mg/L) of a drug is measured every hour after an injection.

$t$ (h)	$0$	$1$	$2$	$3$	$4$	$5$	$6$	$7$	$8$
$C$ (mg/L)	$40.1$	$31.9$	$25.7$	$20.4$	$16.5$	$13.0$	$10.6$	$8.3$	$6.8$

• (a) Find a quadratic regression model and an exponential regression model, with $R^2$ for each.
• (b) Which model should be used to estimate the concentration at $t = 12$? Justify your answer.

Solution.

(a) Quadratic regression: $C = 0.440t^2 - 7.55t + 39.5$, with $SS_{res} \approx 1.60$ and $R^2 = 0.998$.

Exponential regression: $C = 40.0(0.800)^t$, with $SS_{res} \lt 0.1$ and $R^2 = 1.00$ (to 3 s.f.; a GDC’s exponential regression shows about $0.9998$).

(b) Both models fit the data extremely well: their $R^2$ values differ by only about $0.0015$. The deciding factors are the context and the behaviour beyond the data:

• The quadratic has its minimum at $t \approx 8.58$ h and then increases, predicting the concentration will rise again with no new dose. That’s not reasonable.
• The exponential keeps decreasing toward $0$, as the body steadily clears the drug. A constant ratio of $0.800$ means $20\%$ is removed each hour, which is a standard model for drug clearance.

So use the exponential: with the full values from a GDC, $C(12) \approx 2.76$ mg/L. (Software that minimizes $SS_{res}$ directly gives $2.75$, so about $2.8$ mg/L either way.) Even then, $t = 12$ is an extrapolation, so it’s an estimate.

Inside the data the two curves are almost identical; beyond $t = 8$ only the exponential behaves sensibly.

Example 4: Sine regression for daylight hours

The average number of daylight hours $y$ in a Canadian city in month $x$ (with $x = 1$ for January) is shown.

$x$	$1$	$2$	$3$	$4$	$5$	$6$	$7$	$8$	$9$	$10$	$11$	$12$
$y$ (h)	$9.1$	$10.4$	$11.9$	$13.5$	$14.9$	$15.6$	$15.3$	$14.1$	$12.5$	$10.9$	$9.5$	$8.8$

• (a) Find a sine regression model $y = a\sin(bx + c) + d$.
• (b) Find the period of the model and comment.
• (c) Estimate the daylight hours in mid-April ($x = 4.5$).

Solution.

(a) With the GDC in radian mode, sine regression gives

$$y = 3.37\sin(0.516x - 1.62) + 12.2$$

with $SS_{res} \approx 0.0470$ and $R^2 = 0.999$.

(b) The period is $\dfrac{2\pi}{b} = \dfrac{2\pi}{0.51607} \approx 12.2$ months. That’s very close to $12$ months, the length of a year, so the model makes sense. The amplitude $3.37$ h is close to half the difference between the longest and shortest days, and $d = 12.2$ h is the average daylight over the year.

(c) $y(4.5) \approx 14.3$ hours (using full values).

Common mistakes

Choosing the type from $R^2$ alone. A higher $R^2$ doesn’t guarantee a better model. A cubic always fits at least as well as a quadratic on the same data, and a model can fit the data well but behave absurdly just outside it. Use context, the shape of the data and the behaviour of the curve as well.

Comparing SS_res across different data sets. $SS_{res}$ depends on the units and on how many points there are. Only compare $SS_{res}$ for different models of the same data.

Misinterpreting R². $R^2 = 0.85$ means $85\%$ of the variability in $y$ is accounted for by the model. It does not mean $85\%$ of the points lie on the curve, or that the predictions are $85\%$ accurate.

Using degree mode for sine regression. GDC sine regression works in radians. In degree mode you’ll get nonsense (or an error).

Rounding the parameters before predicting. Store the regression equation in your GDC and use the full values. Rounding $b = 1.53995$ to $1.54$ and then raising it to a large power changes the answer.

Trusting extrapolation. Regression describes the data you have. Predictions far outside the range of $x$ (especially with polynomial or exponential models) can be wildly wrong.

Practice

1. (Warm-up) The model $y = x^2 + 2$ is proposed for the data below. Find $SS_{res}$.

$x$	$1$	$2$	$3$	$4$
$y$	$2.8$	$6.3$	$11.2$	$17.9$

Solution

Predicted values: $3$, $6$, $11$, $18$. Residuals: $-0.2$, $0.3$, $0.2$, $-0.1$.

$$SS_{res} = 0.04 + 0.09 + 0.04 + 0.01 = 0.18$$

2. (Warm-up) A power regression of a bird’s wingspan on its mass gives $R^2 = 0.91$. Interpret this value.

Solution

$91\%$ of the variability in wingspan is accounted for by the power model of wingspan against mass; the other $9\%$ is due to other factors (or random variation).

3. (Warm-up) Which type of regression would you try first for each situation?

• (a) Daily maximum temperature over two years.
• (b) The number of bacteria in a dish over the first few hours.
• (c) The height of a basketball during a shot.
• (d) The time for a ball to fall from different heights, where $t$ appears to be proportional to a power of $h$.

Solution

(a) Sine: the temperature repeats every year.

(b) Exponential: bacteria multiply at a constant rate at first.

(c) Quadratic: projectile motion.

(d) Power: $t = ah^b$.

4. (Core) A café tests different prices $p$ (dollars) for a lunch special and records its weekly profit $P$ (dollars).

$p$ ($)	$10$	$15$	$20$	$25$	$30$	$35$
$P$ ($)	$820$	$1490$	$1750$	$1900$	$1610$	$1170$

• (a) Find the quadratic regression model and $R^2$.
• (b) Use the model to find the price that gives the maximum profit, and the maximum profit.

Solution

(a) $P = -5.54p^2 + 262p - 1230$ (full values $-5.53571$, $262.021$, $-1232.71$), with $R^2 = 0.992$ (and $SS_{res} \approx 6740$).

(b) The vertex (using the GDC maximum, or $p = -\dfrac{b}{2a}$ with full values) is at $p \approx 23.67$, so a price of about $23.70 (3 s.f.), giving a maximum weekly profit of about $1870.

5. (Core) The number of users $U$ (thousands) of a new app $m$ months after launch is shown.

$m$	$0$	$1$	$2$	$3$	$4$	$5$
$U$ (thousands)	$12.0$	$15.1$	$19.1$	$24.0$	$30.3$	$38.1$

• (a) Find an exponential regression model $U = ab^m$, and the monthly percentage growth.
• (b) Use the model to find when the app will reach $100$ thousand users.

Solution

(a) $U = 12.0(1.26)^m$ (both parameters to 3 s.f.; $R^2 = 1.00$). Since $b \approx 1.26$, the number of users grows by about $26\%$ per month.

(b) Solve $12.0(1.26)^m = 100$ with full values: $m \approx 9.17$ months after launch (during the tenth month).

6. (Core) The time $t$ (s) for a ball to fall from height $h$ (m) is measured.

$h$ (m)	$1$	$2$	$5$	$10$	$20$
$t$ (s)	$0.452$	$0.639$	$1.010$	$1.428$	$2.019$

• (a) Find a power regression model $t = ah^b$.
• (b) Predict the time to fall $45$ m.

Solution

(a) $t = 0.452h^{0.500}$ (3 s.f.), with $R^2 = 1.00$. So $t$ is proportional to $\sqrt{h}$.

(b) $t(45) \approx 0.452 \times 45^{0.500} \approx 3.03$ s (using full values).

7. (Core)

• (a) A linear model has Pearson’s correlation coefficient $r = -0.92$. Find $R^2$ and interpret it.
• (b) Two models are fitted to the same data. Model A has $SS_{res} = 12.4$ and model B has $SS_{res} = 3.1$. Which fits the data better?
• (c) For a data set, $SS_{tot} = 250$ and a model has $SS_{res} = 20$. Use $R^2 = 1 - \dfrac{SS_{res}}{SS_{tot}}$ to find $R^2$.

Solution

(a) For a linear model $R^2 = r^2 = (-0.92)^2 = 0.8464 \approx 0.846$. About $84.6\%$ of the variability in $y$ is accounted for by the linear model.

(b) Model B: its smaller $SS_{res}$ means its predictions are closer to the data. (Whether it’s the better model also depends on context.)

(c) $R^2 = 1 - \dfrac{20}{250} = 1 - 0.08 = 0.92$.

8. (Challenge) A sunflower’s height $H$ (cm) is measured weekly.

Week $w$	$0$	$1$	$2$	$3$	$4$	$5$	$6$	$7$	$8$
$H$ (cm)	$10$	$18$	$33$	$58$	$92$	$130$	$162$	$183$	$195$

• (a) Find quadratic and cubic regression models, with $R^2$ for each.
• (b) Use each model to predict the height at week $12$.
• (c) Which model would you choose for predicting the height at week $12$? Explain.

Solution

(a) Quadratic: $H = 0.596w^2 + 21.3w - 0.879$, $R^2 = 0.980$.

Cubic: $H = -0.749w^3 + 9.59w^2 - 5.81w + 11.7$, $R^2 = 0.999$.

(b) Using full values: quadratic $H(12) \approx 341$ cm; cubic $H(12) \approx 27.9$ cm.

(c) Neither. The cubic has the higher $R^2$ but predicts the sunflower shrinks from $195$ cm to about $28$ cm, which is impossible. The quadratic predicts growth speeding up forever, but the data shows growth already slowing down (the weekly increases fall from $38$ cm to $12$ cm). The shape suggests the height is levelling off, so a model with a horizontal asymptote, such as a logistic model, would be more suitable. This is a clear example of why $R^2$ alone is not enough.

9. (Challenge) The height of the tide $y$ metres at a harbour is recorded every $2$ hours, starting at midnight ($t = 0$).

$t$ (h)	$0$	$2$	$4$	$6$	$8$	$10$	$12$	$14$	$16$	$18$	$20$	$22$
$y$ (m)	$3.9$	$3.1$	$1.4$	$0.6$	$1.2$	$2.8$	$3.9$	$3.3$	$1.6$	$0.6$	$0.9$	$2.5$

• (a) Find a sine regression model $y = a\sin(bt + c) + d$ with $a \gt 0$ and $b \gt 0$, and its $R^2$.
• (b) Find the period of the tide according to the model.
• (c) Predict the height of the tide at 1 a.m. the next day.

Solution

(a) In radian mode: $y = 1.69\sin(0.508t + 1.58) + 2.19$, with $R^2 = 0.998$.

(b) Period $= \dfrac{2\pi}{0.50837} \approx 12.4$ hours (a typical period for ocean tides).

(c) 1 a.m. the next day is $t = 25$: $y(25) \approx 3.86$ m (using full values). This is a short extrapolation of a regular periodic pattern, so it’s reasonably reliable.