Science · Unit 17

Research Methods and Statistics

Designing rigorous studies and reading the data

Most of what goes wrong in published research goes wrong in the design, before any data is collected. This unit is about the choices that determine whether a study can answer its own question.

It covers experimental design, controls and sampling, the difference between correlation and causation, what statistical significance does and does not mean, and why peer review and reproducibility exist.

This unit breaks down into 21 short steps and 127 questions, starting at difficulty 4 and building to 5. Below you can see exactly what it covers, how the path is structured, and worked examples with explanations.

Steps: 21
Questions: 127
Difficulty: 4-5

What this unit covers

Experimental Design
Controls and Sampling
Correlation vs Causation
Statistical Significance
Peer Review and Reproducibility

Where this fits

The advanced counterpart to The Scientific Method. Psychology's Research Methods covers similar ground for human subjects.

Where people slip

Statistically significant does not mean important, large or true. It means the result would be unlikely if nothing were going on - which is a much weaker claim than it sounds.

How the unit is structured

Research Methods and Statistics runs as 21 short steps that unlock in order. 15 are practice rounds and 6 are challenge rounds that pull together everything before them. Questions start at difficulty 4 and climb to 5 as you progress.

Step 1 · easierStep 21 · harder

Challenge rounds

Example questions

30 real questions from this unit, with the answer and the reason behind it, grouped by what they practise. There are 127 in the unit altogether.

Controls and Sampling

Build the sentenceLevel 4
1. Build a sentence about the role of a control group.
Answer: A control group gives a baseline for comparison
A control group provides the baseline that the treatment group is compared against.
Choose all that applyLevel 4
2. Which of these help make a sample more representative of its population? Pick all that apply.
- Randomly selecting participantscorrect
- Using a large enough samplecorrect
- Building a complete sampling framecorrect
- Recruiting only self-selected volunteers
Random selection, adequate size, and a good sampling frame reduce bias; relying on volunteers adds self-selection bias.
Fact or fibLevel 4
3. Including a placebo control group lets researchers separate a treatment's specific effect from the improvement people expect just from being treated.
Answer: True
A placebo group captures the expectation effect so the treatment's true effect can be isolated.
Fill the blankLevel 4
4. A sample that mirrors the key characteristics of its population is described as ____.
- representativecorrect
- biased
- convenient
- anonymous
A representative sample lets you generalize findings to the population with less bias.
Match the pairsLevel 4
5. Match each sampling method to its description.
Answer: Stratified sampling = Sample each subgroup in proportion; Cluster sampling = Randomly select whole groups; Systematic sampling = Take every nth person on a list; Convenience sampling = Use whoever is easiest to reach
Each scheme selects members differently, with different effects on representativeness.
Multiple choiceLevel 4
6. What is the main purpose of a control group?
- To provide a baseline showing what happens without the treatmentcorrect
- To increase the total sample size
- To guarantee a statistically significant result
- To make the study double-blind by itself
The control group isolates the treatment's effect by showing the outcome when the treatment is absent.

Correlation vs Causation

Build the sentenceLevel 4
7. Build the classic warning about interpreting an observed association.
Answer: Correlation does not imply causation
The phrase reminds us that an observed association is not by itself evidence of a cause.
Choose all that applyLevel 4
8. Besides the idea that X causes Y, which are valid alternative explanations for an observed correlation? Pick all that apply.
- Y actually causes X (reverse causation)correct
- A third variable causes both (confounding)correct
- The association arose by chance in this samplecorrect
- The correlation coefficient is greater than 1
Reverse causation, confounding, and chance can all produce a correlation, but a correlation coefficient can never exceed 1.
Fact or fibLevel 4
9. A negative correlation means that as one variable increases, the other tends to decrease.
Answer: True
Negative (inverse) correlations have a downward trend: higher values of one go with lower values of the other.
Fill the blankLevel 4
10. When two variables correlate only because a third variable influences both, the association is called ____.
- spuriouscorrect
- causal
- linear
- significant
A spurious correlation reflects a lurking third variable, not a real link between the two measured variables.
Guess the numberLevel 4
11. What value of the correlation coefficient r indicates no linear relationship between two variables?
Answer: 0 r
An r of 0 means no linear association; values approach +1 or -1 as the linear relationship strengthens.
Match the pairsLevel 4
12. Match each correlation coefficient to its description.
Answer: r = +0.90 = Strong positive; r = -0.85 = Strong negative; r = +0.15 = Weak positive; r = 0.00 = No linear relationship
The sign shows direction and the magnitude from 0 to 1 shows the strength of the linear relationship.

Experimental Design

Choose all that applyLevel 4
13. Which are defining features of a true experiment? Pick all that apply.
- Manipulation of an independent variablecorrect
- Random assignment to conditionscorrect
- A comparison or control conditioncorrect
- Measuring pre-existing groups as they naturally occur
True experiments manipulate a variable, randomly assign, and compare conditions; measuring natural groups is correlational.
Fill the blankLevel 4
14. An ____ definition states exactly how an abstract variable will be manipulated or measured in a study.
- operationalcorrect
- theoretical
- circular
- nominal
An operational definition turns a concept like stress into concrete procedures so others can replicate it.
Guess the numberLevel 4
15. A fully crossed 2 x 2 x 2 factorial experiment has how many distinct treatment conditions?
Answer: 8 conditions
Each of the three factors has 2 levels, so the number of cells is 2 x 2 x 2 = 8.
Match the pairsLevel 4
16. Match each experimental design term to its role.
Answer: Independent variable = Manipulated cause; Dependent variable = Measured outcome; Confounding variable = Alternative explanation; Control group = Baseline comparison
Distinguishing the manipulated cause, measured outcome, confound, and baseline is the core vocabulary of design.
Multiple choiceLevel 4
17. What is the dependent variable in an experiment?
- The outcome that is measured to detect an effectcorrect
- The variable the researcher manipulates
- A variable held constant across all conditions
- A participant assigned at random
The dependent variable is the measured outcome expected to respond to the independent variable.
Fact or fibLevel 5
18. Random assignment and random sampling are two names for the same procedure.
Answer: False
Random assignment spreads participants across conditions (internal validity), while random sampling draws participants from a population (external validity).

Peer Review and Reproducibility

Build the sentenceLevel 4
19. Build a sentence describing what peer review does.
Answer: Peer review lets experts check a study
Peer review is independent experts checking a study's quality before it is published.
Fact or fibLevel 4
20. Publication bias means studies with null (non-significant) results are less likely to get published.
Answer: True
The file drawer problem hides null results, biasing the visible literature toward positive findings.
Fill the blankLevel 4
21. Publicly registering your hypotheses and analysis plan before collecting data is called ____.
- preregistrationcorrect
- peer review
- replication
- meta-analysis
Preregistration guards against p-hacking and HARKing by fixing the plan in advance.
Guess the numberLevel 4
22. If you run 20 independent significance tests on data with no real effect, using alpha = 0.05, about how many false positives would you expect on average?
Answer: 1 tests
With alpha = 0.05, about 5 percent of the 20 tests, or roughly 1, will be a false positive by chance.
Multiple choiceLevel 4
23. What is the primary purpose of peer review?
- To have independent experts evaluate a study's quality before publicationcorrect
- To guarantee the findings are true
- To repeat the experiment for the authors
- To calculate the study's p-value
Peer reviewers check methods, analysis, and reasoning to filter and improve work before it is published.
Choose all that applyLevel 5
24. Which of these are questionable research practices (QRPs)? Pick all that apply.
- Selectively reporting only the analyses that reached significancecorrect
- HARKing, presenting a hypothesis formed after seeing the results as if predicted
- Trying tests until p drops below 0.05correct
- Preregistering the analysis plan before data collection
Selective reporting, HARKing, and p-hacking bias the literature; preregistration is a safeguard against them.

Statistical Significance

Choose all that applyLevel 4
25. Which factors increase the statistical power of a study? Pick all that apply.
- A larger sample sizecorrect
- A larger true effect sizecorrect
- Lower measurement variability or noisecorrect
- A smaller sample size
Power rises with bigger samples, larger real effects, and less noise, making a genuine effect easier to detect.
Fact or fibLevel 4
26. A statistically significant result is always large enough to be practically important.
Answer: False
With a large sample even a tiny, trivial effect can be significant, so significance is not the same as importance.
Fill the blankLevel 4
27. By common convention, a result is called statistically significant when the p-value is below the alpha level of ____.
- 0.05correct
- 0.5
- 5
- 0.95
An alpha of 0.05 is the traditional threshold, meaning about a 5 percent risk of a false positive.
Guess the numberLevel 4
28. A 95 percent confidence interval corresponds to an alpha level of what percent?
Answer: 5 percent
A 95 percent confidence level leaves 5 percent in the tails, matching an alpha of 0.05.
Match the pairsLevel 4
29. Match each significance-testing term to its meaning.
Answer: Type I error = Rejecting a true null (false positive); Type II error = Failing to reject a false null (false negative); Alpha = Accepted Type I error rate; Statistical power = Chance of detecting a real effect
Type I and Type II errors, alpha, and power are the core trade-offs of hypothesis testing.
Multiple choiceLevel 4
30. What does a p-value represent?
- The probability of data at least as extreme as observed, assuming the null hypothesis is true
- The probability that the null hypothesis is true
- The probability that the result is a fluke and wrong
- The size of the effect that was found
A p-value is computed assuming the null hypothesis holds; it is not the probability the null is true.

Where these questions come from. Each unit starts as a plan of the concepts it should cover and the difficulty it should span. Questions are written against that plan with AI assistance, then checked by a validator that rejects anything without a single defensible answer, an explanation, or plausible wrong options. How we write questions sets out the whole process, and corrections are fixed in the bank and reach the site and the app the same day.

How you practise

This unit mixes 13 different question formats, so you are recalling and applying rather than recognising the same layout every time.

Build the sentence
Choose all that apply
Fact or fib
Fill the blank
Guess the number
Match the pairs
Multiple choice
Odd one out
Put in order
Sort into groups
Spell it
True or false
Type the answer

Practise Research Methods and Statistics

127 questions across 21 steps. Start with step one and crawl at your own pace.

Play this unit

More units in Science

See all 18 units in Science →