Survey and Experiment Design

Good data doesn’t happen by accident. Whether you’re testing if a study app improves marks or asking students what they think of a new schedule, the way you design your experiment or survey decides whether the results mean anything. This page covers the principles that make primary data reliable, and how to write survey questions that are fair, clear, and respectful.

Key ideas

Experiments and observational studies

In an observational study, you record what already happens without changing anything. Surveys are observational.
In an experiment, you deliberately apply a treatment (something you change) to some subjects and measure the effect.
The treatment group gets the treatment. The control group doesn’t (or gets the usual treatment, or a placebo, a fake treatment like a sugar pill), so you have something to compare with.

Only a well-designed experiment can show that one thing causes another. An observational study can show that two things are related, but other factors might explain the link (see correlation and causation).

Three principles of experimental design

Principle	What it means	Why it matters
Randomization	use chance to decide which subjects get which treatment	stops the researcher (even accidentally) putting certain kinds of subjects in one group, so the groups are similar at the start
Replication	use enough subjects, or repeat the experiment	one or two subjects could be unusual; many subjects let real effects stand out from natural variation
Control	compare with a control group, and keep all other conditions the same for every group	if the only difference between groups is the treatment, a difference in results can be blamed on the treatment

In a blind experiment, subjects don’t know which group they’re in. In a double-blind experiment, the people measuring the results don’t know either. This reduces response bias and measurement bias.

Reliable data

Data is reliable if repeating the study would give similar results. To make data more reliable:

use a large enough sample: larger random samples give results that vary less (see margin of error)
ask every respondent the same questions, worded the same way
run a pilot test on a few people first to find confusing questions

Open and closed questions

	Open question	Closed question
Example	“What would improve the cafeteria?”	“How often do you buy lunch in the cafeteria? (never / 1–2 days a week / 3–4 days a week / every day)”
Strengths	gives detailed, unexpected ideas	quick to answer; easy to count, graph, and compare
Weaknesses	slow to answer; hard to organize and summarize	can miss answers you didn’t think of; choices can push people

For closed questions, the answer choices must not overlap and must cover every possibility (add “other” or “none” if needed).

Writing fair questions

Problem	Example	Fix
Leading question (suggests the “right” answer)	“Don’t you agree that the school day starts too early?”	“Do you think the school day should start earlier, at the same time, or later?”
Loaded question (uses emotional words or assumes something)	“Should the town waste money on a fancy new arena?”	“Should the town build a new arena?”
Double-barrelled question (asks two things at once)	“Are the coaches knowledgeable and friendly?”	Ask two questions, one about each.
Unclear or technical wording	“How often do you exercise?” (what counts? how often is “often”?)	“In the past 7 days, on how many days did you do at least 30 minutes of physical activity?”
Cultural bias (assumes one background)	“What did your family do for Christmas?”	“Did your family celebrate any holidays in December? If so, what did you do?”

Ethics, privacy, and honest responses

Informed consent: tell people what the study is for and how their answers will be used, and get their agreement. For students under $18$ , a parent or guardian may also need to agree.
Voluntary participation: people can skip any question or stop at any time.
Privacy: collect only the personal information you need. Keep responses anonymous (no names) or confidential (names kept private), and store data securely.
Avoid harm: experiments must not put people at risk, and sensitive questions need extra care.

People answer more honestly when responses are anonymous, the wording is neutral, and the person asking isn’t someone they want to impress (like their teacher or coach).

Worked examples

Example 1: Experiment or observational study

Decide whether each study is an experiment or an observational study. For any experiment, name the treatment and control groups.

(a) A researcher compares the marks of students who already eat breakfast with those who don’t.
(b) A coach randomly assigns $15$ swimmers to a new warm-up routine and $15$ to the usual one, then compares their race times.

Solution.

(a) Observational study. The researcher didn’t decide who eats breakfast. Students who eat breakfast might also sleep more or have different routines, so this study can’t prove breakfast causes better marks.

(b) Experiment. The treatment group is the $15$ swimmers on the new warm-up; the control group is the $15$ on the usual warm-up.

Example 2: Designing an experiment

A student wants to know whether listening to music while studying affects scores on a vocabulary quiz. She has $40$ volunteers. Design an experiment that uses randomization, replication, and control.

Solution.

Randomization: list the $40$ volunteers in a spreadsheet, add a column with =RAND(), sort by it, and put the first $20$ in the music group and the other $20$ in the silence group. Chance, not the student, decides the groups.
Control: both groups study the same $25$ words for the same $15$ minutes, in the same kind of room, at the same time of day, then take the same quiz. The only difference is the music. The silence group is the control group.
Replication: $20$ people per group (rather than $2$ or $3$ ) means a few unusually strong or weak students won’t decide the result. She could also repeat the experiment with a different word list.
Compare the mean quiz scores of the two groups.

Example 3: Fixing bad questions

Identify the problem with each question and rewrite it.

(a) “Most experts agree that homework is harmful. Do you think there’s too much homework?”
(b) “How satisfied are you with the bus schedule and the cost of a pass?”
(c) “Do you support the council’s reckless plan to cut library hours?”

Solution.

(a) Leading. Quoting “experts” pushes people to agree. Rewrite: “How would you describe the amount of homework you get? (too little / about right / too much)”

(b) Double-barrelled. Someone could like the schedule but not the price. Rewrite as two questions: “How satisfied are you with the bus schedule?” and “How satisfied are you with the cost of a bus pass?”

(c) Loaded. “Reckless” is an emotional word. Rewrite: “Do you support or oppose the council’s plan to reduce library hours?”

Example 4: Closed-question answer choices

A survey asks, “How many hours do you spend on social media each day?” with the choices $0$ – $1$ , $1$ – $2$ , $2$ – $3$ . What’s wrong with the choices? Rewrite them.

Solution. Two problems:

They overlap. Someone who spends exactly $1$ hour could pick $0$ – $1$ or $1$ – $2$ .
They don’t cover everything. There’s no choice for more than $3$ hours.

Better choices: “less than 1 hour”, “1 hour to less than 2 hours”, “2 hours to less than 3 hours”, “3 hours or more”. Every possible answer now fits in exactly one choice.

Common mistakes

Letting people choose their own group. If volunteers pick whether to be in the music group, the groups may differ in other ways (people who like studying with music might study differently). Use random assignment.

Forgetting the control group. If everyone gets the new warm-up and times improve, you can’t tell whether it was the warm-up or just more practice. You need a group to compare with.

Changing more than one thing. If the music group also studies in a different room, you can’t tell whether the music or the room made the difference. Keep everything else the same.

Writing a double-barrelled question without noticing. Watch for the word “and” in a question. If someone could honestly answer “yes” to one part and “no” to the other, split it.

Collecting more personal information than needed. Names, addresses, and birth dates make people less honest and raise privacy concerns. Only ask for what your question needs.

Practice

1. (Warm-up) Is each study an experiment or an observational study?

(a) A student records the number of cars that pass the school each hour.
(b) A gardener grows $10$ plants under a grow light and $10$ in normal light and compares their heights.
(c) A survey asks students how many hours they sleep.

Solution

(a) Observational. (b) Experiment. (c) Observational.

2. (Warm-up) Classify each question as open or closed.

(a) “What is your favourite thing about this school?”
(b) “Do you take the bus to school? (yes / no)”
(c) “Rate the cafeteria food from 1 (poor) to 5 (excellent).”

Solution

(a) Open. (b) Closed. (c) Closed.

3. (Warm-up) Name the problem with each question: leading, loaded, or double-barrelled.

(a) “Do you enjoy math and science?”
(b) “Wouldn’t a longer lunch period be great?”
(c) “Should students be forced to wear ugly uniforms?”

Solution

(a) Double-barrelled. (b) Leading. (c) Loaded.

4. (Core) A company claims its new energy drink improves reaction time. Design an experiment with $60$ volunteers that uses randomization, replication, and control. Explain why a placebo would be useful.

Solution

Sample answer: randomly assign $30$ volunteers to drink the energy drink and $30$ to drink a placebo, a drink that looks and tastes the same but has no active ingredients (randomize by sorting a =RAND() column, for example). After $30$ minutes, everyone takes the same online reaction-time test in the same conditions. Compare the mean reaction times.

Randomization makes the groups similar at the start. Using $30$ per group is replication. The placebo group is the control: since nobody knows which drink they had, any “I feel energized” effect happens in both groups, so a difference in results can be credited to the drink itself.

5. (Core) Rewrite each question to fix its problem.

(a) “Since our park is so run-down, should the town fix it?”
(b) “How often do you read?”
(c) “Do you think the hockey league should lower fees and add more practice time?”

Solution

Sample answers:

(a) The question is leading (it also assumes the park is run-down, so you could call it loaded too). Rewrite: “How would you rate the condition of the park? (poor / fair / good / excellent)”

(b) The question is vague. Rewrite: “In the past 7 days, on how many days did you read for pleasure for at least 15 minutes?”

(c) The question is double-barrelled. Split it: “Should the league lower fees?” and “Should the league add more practice time?”

6. (Core) A survey asks, “How old are you?” with the choices $13$ – $15$ , $15$ – $17$ , $17$ – $19$ . Explain the two problems and fix them, assuming the survey is for high school students.

Solution

The choices overlap: a $15$ -year-old and a $17$ -year-old each fit in two choices. They also miss anyone younger than $13$ or older than $19$ .

Fix: ” $13$ or younger”, ” $14$ – $15$ ”, ” $16$ – $17$ ”, ” $18$ or older”.

7. (Core) A student plans to survey classmates about vaping by handing out a form that asks for each student’s name, homeroom, and whether they have ever vaped. Give two ethical or design problems with this plan, and suggest improvements.

Solution

Privacy: asking for names on a sensitive question isn’t necessary and could expose students to consequences. Make the survey anonymous.
Honest responses: students who have vaped may answer “no” if their name is attached (response bias). Anonymous forms collected in a sealed box encourage honesty.

The student should also explain the purpose of the survey, make it voluntary, and check whether school or parental permission is needed.

8. (Challenge) A researcher notices that students who play a musical instrument have higher average math marks than those who don’t. She concludes that learning an instrument improves math skills.

(a) Why can’t this observational study prove her conclusion?
(b) Describe an experiment that could test her claim, and explain one practical or ethical difficulty with it.

Solution

(a) She didn’t assign who plays an instrument. Other factors could explain the link: students in music programs might have more support at home, more time for activities, or already enjoy patterns and practice. Any of these could raise both music participation and math marks.

(b) Sample answer: take a large group of students who don’t play an instrument, randomly assign half to a year of free music lessons and half to no lessons (or a different activity, like art, to control for “extra attention”), and compare the change in math marks.

Difficulties: it takes a long time; some students in the control group may take up an instrument anyway; and withholding lessons from students who want them, or requiring lessons from those who don’t, raises fairness concerns. Students and parents must give informed consent.

9. (Challenge) A student council wants to know what students think about a proposed later start time. Write a short survey (three or four questions) for them that includes at least one closed question and one open question. Then explain how your survey avoids bias and protects privacy.

Solution

Sample survey:

What grade are you in? (9 / 10 / 11 / 12)
If school started $45$ minutes later and ended $45$ minutes later, would you prefer that to the current schedule? (prefer the later schedule / prefer the current schedule / no preference)
How would a later end time affect your after-school activities or job? (a lot / a little / not at all / I don’t have after-school commitments)
Is there anything else you’d like the council to consider? (open)

How it avoids bias: question 2 describes both the later start and the later end, so it doesn’t make the change sound only good; the choices are balanced and include “no preference”. Question 3’s choices don’t overlap and include an option for everyone. Question 4 lets students raise ideas the council didn’t think of.

Privacy: the survey asks only for grade, not names, so it’s anonymous, and answering is voluntary. The council should hand it to a random sample of students (for example, stratified by grade) rather than just posting it online, to avoid a voluntary response sample.