Margin of Error

You’ve probably seen a news story that ends with a line like “accurate within $\pm 3$ percentage points, 19 times out of 20.” That line is the margin of error, and it’s an honest admission that a poll of a sample can’t give the exact answer for a whole population. Understanding it helps you tell the difference between a real result and one that’s too close to call.

Key ideas

Statistics estimate parameters

A parameter is a number that describes a whole population, like the true percentage of all voters in a town who support a new arena. We usually can’t ask everyone, so we take a sample and calculate a statistic, like the percentage of the sample who support it. (See sampling methods.)

Different random samples give slightly different statistics, just by chance. This is called sampling variability. The margin of error measures how big that chance variation is likely to be.

Margin of error and confidence interval

A poll result is reported as

$$\text{statistic} \pm \text{margin of error}$$

The range from (statistic $-$ margin) to (statistic $+$ margin) is called a confidence interval. For example, ”$58\%$, accurate within $\pm 4$ percentage points” gives the interval $54\%$ to $62\%$.

Confidence level

The confidence level says how reliable the method is. ”$19$ times out of $20$” means

$$\frac{19}{20} = 0.95 = 95\% \text{ confidence}$$

What it means: if the poll were repeated many times with new random samples of the same size, about $95\%$ of the intervals made this way would contain the true population value.

What it doesn’t mean:

• It doesn’t mean $95\%$ of people agreed, or that $95\%$ of the population was surveyed.
• It doesn’t guarantee the true value is in the interval. About $1$ poll in $20$ will miss.
• It doesn’t cover bias. The margin of error only accounts for random sampling variability. A biased sample (for example, an online poll where anyone can vote) can be far off no matter how small its margin of error looks. (See bias in sampling.)

The three-way relationship

Sample size, margin of error, and confidence level are linked:

• Bigger sample → smaller margin of error. More data gives a more precise estimate. But to cut the margin of error in half, you need about four times the sample size.
• Higher confidence → wider margin of error. To be more sure your interval catches the true value, you have to make the interval wider.

A formula for investigating (proportions at 95% confidence)

For a sample proportion $p$ (written as a decimal) from a random sample of size $n$, the margin of error at $95\%$ confidence is approximately

$$\text{ME} \approx 1.96\sqrt{\frac{p(1 - p)}{n}}$$

This formula is a tool for exploring the relationships above. It assumes a simple random sample and a fairly large $n$.

The $1.96$ comes from the normal distribution: the middle $95\%$ of a standard normal curve lies between $z = -1.96$ and $z = 1.96$. For other confidence levels, replace $1.96$ with a different z-score:

Confidence level	$z$
$90\%$	$1.645$
$95\%$	$1.96$
$99\%$	$2.576$

Quick estimate. The product $p(1 - p)$ is largest when $p = 0.5$, and then $1.96\sqrt{\dfrac{0.25}{n}} = \dfrac{0.98}{\sqrt{n}}$. So at $95\%$ confidence, a quick (slightly generous) estimate is

$$\text{ME} \approx \frac{1}{\sqrt{n}}$$

For example, a poll of $1000$ people has a margin of error of at most about $\dfrac{1}{\sqrt{1000}} \approx 0.032$, or about $\pm 3$ percentage points.

Worked examples

Example 1: Reading a poll

A random sample of $600$ residents of a town found that $58\%$ support building a new skatepark. The result is “accurate within $\pm 4$ percentage points, $19$ times out of $20$.”

• (a) What is the confidence interval, and what is the confidence level?
• (b) Can the town council be fairly confident that a majority of residents support the skatepark?

Solution. (a) The interval is $58\% - 4\% = 54\%$ to $58\% + 4\% = 62\%$. ”$19$ times out of $20$” is a $95\%$ confidence level.

(b) Yes. The whole interval is above $50\%$, so the council can be $95\%$ confident that a majority of all residents (not just the $600$ surveyed) support the skatepark.

Example 2: Calculating a margin of error

In a random sample of $400$ students at a large high school, $35\%$ said they’d prefer a later start time. Find the margin of error at $95\%$ confidence, and compare it with the quick estimate.

Solution. Use $p = 0.35$ and $n = 400$:

$$\begin{aligned} \text{ME} &\approx 1.96\sqrt{\frac{0.35(0.65)}{400}} \\ &= 1.96\sqrt{0.000\,568\,75} \\ &\approx 1.96(0.0238) \\ &\approx 0.047 \end{aligned}$$

The margin of error is about $\pm 4.7$ percentage points, so the interval is about $30.3\%$ to $39.7\%$.

The quick estimate gives $\dfrac{1}{\sqrt{400}} = \dfrac{1}{20} = 0.05$, or $\pm 5$ points. That’s close, and a little larger, as expected.

Example 3: How sample size matters

Using $p = 0.5$ and $95\%$ confidence, find the margin of error for samples of size $100$, $400$, $1600$, and $6400$. What pattern do you see?

Solution. With $p = 0.5$, $\text{ME} \approx 1.96\sqrt{\dfrac{0.25}{n}}$:

$n$	$\text{ME}$
$100$	$0.098$, or $\pm 9.8$ points
$400$	$0.049$, or $\pm 4.9$ points
$1600$	$0.0245$, or about $\pm 2.5$ points
$6400$	$0.012\,25$, or about $\pm 1.2$ points

Each time the sample size is multiplied by $4$, the margin of error is cut in half. That’s because $n$ is under a square root: $\sqrt{4} = 2$. Bigger samples are more precise, but each improvement costs four times as many people. This is why most national polls stop at around $1000$ to $2000$ people.

Example 4: How confidence level matters

A random sample of $500$ people found that $60\%$ have a library card. Find the margin of error at $90\%$, $95\%$, and $99\%$ confidence.

Solution. First, $\sqrt{\dfrac{0.6(0.4)}{500}} = \sqrt{0.000\,48} \approx 0.021\,91$. Multiply by each $z$:

Confidence	$\text{ME}$	Interval
$90\%$	$1.645(0.021\,91) \approx 0.036$	$56.4\%$ to $63.6\%$
$95\%$	$1.96(0.021\,91) \approx 0.043$	$55.7\%$ to $64.3\%$
$99\%$	$2.576(0.021\,91) \approx 0.056$	$54.4\%$ to $65.6\%$

Same data, different intervals: the more confident you want to be, the wider the interval has to be. It’s a trade-off between being sure and being precise.

Common mistakes

Thinking “95% confidence” means 95% of people agree. The confidence level describes the polling method, not the opinions. It says how often intervals made this way catch the true value.

Treating the interval as a guarantee. About $1$ in $20$ polls at $95\%$ confidence will miss the true value, through no fault of the pollster.

Ignoring the margin of error when comparing results. If two options’ intervals overlap, the poll can’t tell you who is really ahead. Roughly, a lead needs to be more than twice the margin of error before it’s clear. Otherwise, call it “too close to call”.

Thinking a big sample fixes bias. A huge, self-selected online poll can have a tiny margin of error on paper and still be badly wrong. The margin of error only applies to properly random samples.

Doubling the sample to halve the margin. Because of the square root, you need four times the sample to halve the margin of error.

Mixing up percent and percentage points. If support goes from $40\%$ to $44\%$, that’s an increase of $4$ percentage points, but a $10\%$ increase. Margins of error for percentages are given in percentage points.

Practice

1. (Warm-up) A school survey finds that $42\%$ of students eat breakfast every day, accurate within $\pm 3$ percentage points, $19$ times out of $20$. Write the confidence interval.

Solution

$42\% - 3\% = 39\%$ to $42\% + 3\% = 45\%$, at $95\%$ confidence.

2. (Warm-up) What confidence level does each phrase describe?

• (a) ”$19$ times out of $20$”
• (b) ”$9$ times out of $10$”
• (c) ”$99$ times out of $100$“

Solution

(a) $95\%$. (b) $90\%$. (c) $99\%$.

3. (Warm-up) A poll of students finds that $62\%$ want a longer lunch break, accurate within $\pm 5$ percentage points, $19$ times out of $20$. Is each statement true or false?

• (a) We can be $95\%$ confident that between $57\%$ and $67\%$ of all students want a longer lunch.
• (b) Exactly $62\%$ of all students want a longer lunch.
• (c) $95\%$ of the students in the school were surveyed.
• (d) The true percentage is definitely between $57\%$ and $67\%$.

Solution

(a) True. (b) False: $62\%$ is the sample’s result, an estimate of the true value. (c) False: the $95\%$ is the confidence level, not the share surveyed. (d) False: the method catches the true value about $19$ times out of $20$, so there’s a small chance it’s outside the interval.

4. (Core) In a random sample of $600$ people, $70\%$ said they recycle every week. Find the margin of error at $95\%$ confidence, and write the confidence interval.

Solution

$$\text{ME} \approx 1.96\sqrt{\frac{0.7(0.3)}{600}} = 1.96\sqrt{0.000\,35} \approx 1.96(0.0187) \approx 0.037$$

The margin of error is about $\pm 3.7$ percentage points, so the interval is about $66.3\%$ to $73.7\%$.

5. (Core) Use the quick estimate $\text{ME} \approx \dfrac{1}{\sqrt{n}}$.

• (a) What is the margin of error for a poll of $2500$ people?
• (b) About how many people would you need for a margin of error of $\pm 4$ percentage points?

Solution

(a) $\dfrac{1}{\sqrt{2500}} = \dfrac{1}{50} = 0.02$, or $\pm 2$ percentage points.

(b) Solve $\dfrac{1}{\sqrt{n}} = 0.04$: $\sqrt{n} = 25$, so $n = 625$ people.

6. (Core) A poll of $1000$ randomly chosen voters in a small town shows the mayoral candidates at $48\%$ (Candidate A) and $44\%$ (Candidate B), accurate within $\pm 3$ percentage points, $19$ times out of $20$. A headline says “A leads B.” Is that fair?

Solution

A’s interval is $45\%$ to $51\%$ and B’s is $41\%$ to $47\%$. The intervals overlap (from $45\%$ to $47\%$), so it’s quite possible that B actually has as much support as A, or more. The poll doesn’t give strong evidence that A is ahead. A fairer headline would be “Race too close to call.”

7. (Core) Two schools ran the same survey question. At School P, $250$ randomly chosen students were asked; at School Q, $1000$ were asked. Both found that $50\%$ said yes. Find each margin of error at $95\%$ confidence and explain the difference.

Solution

School P: $1.96\sqrt{\dfrac{0.25}{250}} \approx 0.062$, or about $\pm 6.2$ points.

School Q: $1.96\sqrt{\dfrac{0.25}{1000}} \approx 0.031$, or about $\pm 3.1$ points.

School Q’s sample is four times as big, so its margin of error is half as big. Its estimate is more precise.

8. (Challenge) A school council wants to estimate the percentage of students who support a new dress code to within $\pm 2$ percentage points at $95\%$ confidence. Using $p = 0.5$ (the “safest” choice, because it gives the largest margin), how many students should they survey? If they only want $\pm 4$ points, how many?

Solution

Solve $1.96\sqrt{\dfrac{0.25}{n}} = 0.02$:

$$\begin{aligned} \sqrt{\frac{0.25}{n}} &= \frac{0.02}{1.96} \\ \frac{0.5}{\sqrt{n}} &= \frac{0.02}{1.96} \\ \sqrt{n} &= \frac{0.5(1.96)}{0.02} = 49 \\ n &= 2401 \end{aligned}$$

They need $2401$ students. For $\pm 4$ points (twice the margin), they need a quarter as many: $\sqrt{n} = 24.5$, so $n = 600.25$, and they should survey $601$ students. (A school would need more than $2401$ students for the first plan to even be possible!)

9. (Challenge) A website invites readers to vote online on whether a city should ban cars downtown. Of $5000$ votes, $81\%$ say yes. The website reports a margin of error of $\pm 1.4$ percentage points. Where does the $\pm 1.4$ come from, and why is the result still not trustworthy?

Solution

The quick estimate gives $\dfrac{1}{\sqrt{5000}} \approx 0.014$, or $\pm 1.4$ percentage points.

But the margin of error only applies to a random sample. Here, people chose to vote themselves (a voluntary response sample). People with strong opinions, and readers of that particular website, are much more likely to vote, and some people may vote more than once. The sample is biased, so the true percentage of all city residents could be very different from $81\%$, no matter how many votes there were.