Linear Regression

Once a scatter plot shows a linear pattern, you can draw the line of best fit through it. That line is a model: it summarizes the relationship in an equation and lets you make predictions. Linear regression is how technology finds the best possible line, and residuals tell you how well it fits.

Key ideas

The line of best fit

The line of best fit (or regression line) is written

$$y = ax + b$$

where $a$ is the slope and $b$ is the $y$-intercept. Technology finds it using the least squares method: of all possible lines, it picks the one that makes the sum of the squared vertical distances from the points to the line as small as possible.

• Spreadsheet: =SLOPE(B2:B9, A2:A9) and =INTERCEPT(B2:B9, A2:A9) (note: $y$-values first), or add a linear trendline to a chart.
• Graphing calculator: STAT, CALC, LinReg(ax+b).
• Desmos: y1 ~ a x1 + b.

The line always passes through the point $(\bar{x}, \bar{y})$, the means of the two variables.

Interpreting slope and intercept

• Slope $a$: for each increase of $1$ unit in $x$, the model predicts $y$ changes by $a$ units.
• Intercept $b$: the predicted $y$ when $x = 0$. This only makes sense if $x = 0$ is reasonable and close to the data.

Always use the units and the context: “each extra degree predicts about $4$ more cones sold”, not just “the slope is $4$”.

Interpolation and extrapolation

• Interpolation: predicting for an $x$-value inside the range of the data. Usually reliable when the correlation is strong.
• Extrapolation: predicting outside the range. Risky, because the pattern may not continue.

Residuals

A residual is how far a point is above or below the line:

$$\text{residual} = \text{observed } y - \text{predicted } y$$

A positive residual means the point is above the line; negative means below.

A residual plot graphs the residuals against $x$. If the line is a good model, the residuals look randomly scattered around $0$. A clear pattern, like a U shape, means a straight line isn’t the right model.

Outliers

An outlier can pull the line toward itself, changing the slope and intercept, and usually lowering $|r|$. Outliers far from the others in the $x$-direction have the most pull.

Answers on this page are rounded; technology may differ slightly in the last digit.

Worked examples

Example 1: Finding and interpreting the line

A student runs an ice-cream stand and records the afternoon temperature and the number of cones sold on eight days.

Temperature (°C)	$16$	$19$	$21$	$23$	$24$	$26$	$28$	$30$
Cones sold	$52$	$70$	$66$	$85$	$80$	$98$	$93$	$112$

Find the line of best fit and $r$. Interpret the slope and the intercept.

Solution. With technology, using $x$ for temperature and $y$ for cones sold:

$$y = 3.957x - 10.500, \qquad r \approx 0.958$$

The correlation is strong and positive, so a line is a good model.

• Slope: each $1$ °C rise in temperature predicts about $4$ more cones sold.
• Intercept: at $0$ °C the model predicts $-10.5$ cones. That’s impossible! The data only go from $16$ °C to $30$ °C, so $x = 0$ is far outside the data and the intercept has no real meaning here.

Example 2: Interpolation and extrapolation

Use the line from Example 1 to predict sales at (a) $25$ °C (b) $40$ °C. How reliable is each prediction?

Solution. Use the rounded equation $y = 3.957x - 10.500$ from Example 1.

(a)

$$y = 3.957(25) - 10.500 = 88.425$$

About $88$ cones. Since $25$ °C is inside the data range ($16$ °C to $30$ °C), this is interpolation, and with $r \approx 0.958$ it’s fairly reliable.

(b)

$$y = 3.957(40) - 10.500 = 147.78$$

About $148$ cones. But $40$ °C is well outside the data, so this is extrapolation. On a very hot day, people might stay indoors, or the stand might run out of ice cream. Treat this prediction with caution.

Example 3: Residuals

Find the residuals for the days at $21$ °C and $26$ °C.

Solution.

At $21$ °C: predicted $y = 3.957(21) - 10.500 \approx 72.6$. Observed: $66$.

$$\text{residual} = 66 - 72.6 = -6.6$$

At $26$ °C: predicted $y = 3.957(26) - 10.500 \approx 92.4$. Observed: $98$.

$$\text{residual} = 98 - 92.4 = 5.6$$

The $21$ °C day sold about $6.6$ cones fewer than predicted (below the line); the $26$ °C day sold about $5.6$ more (above the line).

The line of best fit, with two residuals drawn as vertical segments.

Example 4: The effect of an outlier

On one $20$ °C day, a summer camp group stopped by and the stand sold $115$ cones. Add this point to the data and find the new line and $r$.

Solution. With all nine days:

$$y = 3.099x + 14.395, \qquad r \approx 0.661$$

One outlier pulls the line toward it.

The single outlier lifted the left side of the line, so the slope fell from about $3.96$ to about $3.10$, and $r$ dropped from strong ($0.958$) to moderate ($0.661$). Because the camp visit was a one-time event, it makes sense to leave it out when modelling ordinary sales, and to say that you did.

Common mistakes

Mixing up the variables. If you swap $x$ and $y$, technology gives a different line. Put the independent variable in the $x$ list.

Subtracting the wrong way for a residual. It’s observed minus predicted. A point above the line has a positive residual.

Trusting extrapolations. The model is only supported by the data you have. Predicting far outside that range, or interpreting an intercept far from the data, can give nonsense (like $-10.5$ cones).

Interpreting the slope without context or units. Say what changes, by how much, and for each unit of what.

Rounding the equation too early. Rounding $a$ and $b$ to one decimal before predicting can noticeably change your answers. Keep at least three significant digits.

Deleting outliers without a reason. Remove a point only if you have a reason (an error, or an unusual event), and always report that you did.

Practice

1. (Warm-up) A line of best fit is $y = 2.5x + 12$. Predict $y$ when $x = 8$.

Solution

$y = 2.5(8) + 12 = 32$

2. (Warm-up) A point has observed $y = 47$, and the line predicts $y = 50.3$. Find the residual. Is the point above or below the line?

Solution

Residual $= 47 - 50.3 = -3.3$. It’s negative, so the point is below the line.

3. (Warm-up) The ice-cream data in Example 1 range from $16$ °C to $30$ °C. Is each prediction interpolation or extrapolation? (a) $18$ °C (b) $35$ °C (c) $10$ °C

Solution

(a) Interpolation. (b) Extrapolation. (c) Extrapolation.

4. (Core) A driver records distance driven and fuel used on six trips.

Distance (km)	$50$	$120$	$180$	$250$	$300$	$410$
Fuel (L)	$4.3$	$8.6$	$13.9$	$18.1$	$23.0$	$29.8$

• (a) Find the line of best fit and $r$.
• (b) Interpret the slope.
• (c) Predict the fuel used on a $200$ km trip.

Solution

(a) $y = 0.0723x + 0.508$, with $r \approx 0.998$, a very strong positive correlation.

(b) Each extra kilometre uses about $0.0723$ L of fuel, or about $7.2$ L per $100$ km.

(c) $y = 0.0723(200) + 0.508 \approx 15.0$ L. This is interpolation, and with $r$ so close to $1$, it’s very reliable.

5. (Core) For one house, the monthly heating bill $y$ (in dollars) and the average outdoor temperature $x$ (in °C) for the winter months are modelled by $y = -4.2x + 135$.

• (a) Interpret the slope and the intercept.
• (b) Predict the bill for a month averaging $-8$ °C.

Solution

(a) Slope: each $1$ °C warmer average temperature predicts a bill about $4.20 lower. Intercept: a month averaging $0$ °C predicts a bill of about $135. Since $0$ °C is within the range of typical Ontario winter temperatures, this intercept is meaningful.

(b) $y = -4.2(-8) + 135 = 33.6 + 135 = 168.6$, so about $168.60.

6. (Core) Using your line from Question 4, find the residuals for the $120$ km and $300$ km trips. Which of the two trips used more fuel than predicted?

Solution

$120$ km: predicted $0.0723(120) + 0.508 \approx 9.18$ L, so the residual is $8.6 - 9.18 \approx -0.58$ L.

$300$ km: predicted $0.0723(300) + 0.508 \approx 22.20$ L, so the residual is $23.0 - 22.20 \approx 0.8$ L. (Using unrounded coefficients gives about $0.82$ L.)

The $300$ km trip had a positive residual, so it used more fuel than predicted.

7. (Core) The hours-studied data from scatter plots and correlation are:

Hours	$1$	$2$	$2$	$3$	$4$	$4$	$5$	$6$	$7$	$8$	$8$
Mark (%)	$55$	$52$	$64$	$60$	$71$	$63$	$74$	$70$	$85$	$80$	$45$

The last student was sick on test day. Find the line of best fit with and without that student, and compare.

Solution

With: $y = 1.973x + 56.395$, $r \approx 0.402$.

Without: $y = 4.143x + 50.000$, $r \approx 0.900$.

The outlier flattens the line a lot (the slope drops by more than half) and weakens the correlation from strong to moderate. Since there’s a clear reason the point is unusual, it’s reasonable to leave it out and say so.

8. (Challenge) For the points $(1, 2)$, $(2, 5)$, $(3, 10)$, $(4, 17)$, $(5, 26)$, $(6, 37)$:

• (a) Find the line of best fit and $r$.
• (b) Find all six residuals. What pattern do you see, and what does it tell you?

Solution

(a) $y = 7x - 8.333$, with $r \approx 0.979$.

(b) The predicted values are $-1.333$, $5.667$, $12.667$, $19.667$, $26.667$, $33.667$, so the residuals are about

$$3.33,\ -0.67,\ -2.67,\ -2.67,\ -0.67,\ 3.33$$

They go positive, negative, then positive again: a U shape. Even though $r$ is very high, the data curve upward, so a straight line is not the best model. (In fact $y = x^2 + 1$ fits exactly.)

9. (Challenge) Show that the line of best fit from Example 1 passes (very nearly) through the point $(\bar{x}, \bar{y})$.

Solution

$$\bar{x} = \frac{16 + 19 + 21 + 23 + 24 + 26 + 28 + 30}{8} = \frac{187}{8} = 23.375$$$$\bar{y} = \frac{52 + 70 + 66 + 85 + 80 + 98 + 93 + 112}{8} = \frac{656}{8} = 82$$

At $x = 23.375$: $y = 3.957(23.375) - 10.500 \approx 81.995 \approx 82$. The tiny difference is from rounding the slope and intercept; with unrounded values the line passes through $(23.375, 82)$ exactly.