P-values, confidence intervals, type I and II errors, and power
Read any confidence interval at a glance, know exactly what a p-value does and does not mean, apply the overlapping-intervals rule, and link type I and II errors to power and sample size.
Reviewed and kept current
Last editorial review: 28 September 2026 · Next scheduled refresh: March 2027.
Clinical vignette
Begin with the patient. Read the case, weigh the options, then work through the unit.
Mr G has read a headline about a new drug for his condition. The headline says that it “reduced hospital admission by 20%”. He asks: “Does it work? Is it proven?” His specialist has suggested that he could start it. He is reluctant to take another tablet unless it makes a real difference.
The trial report gives a relative risk of 0.80, with a 95% confidence interval of 0.65 to 0.98.
Mr G wants a simple yes or no. The interval tells you more than the point estimate alone.
Options to weigh
- Tell him that there is a 95% probability that the drug works.
- Explain that the interval does not include 1, so the result is statistically significant at the 5% level.
- Explain that the true effect could be as small as a 2% relative reduction.
- Tell him that statistical significance proves the drug will keep him out of hospital.
- Explain that a significant result does not by itself show that the effect is clinically important.
Reflect
How would you explain what this interval means, so that Mr G can judge whether the drug is worth taking, and what would you be careful not to say?
What this unit covers
Pocket-sized and self-contained: three AKT-style questions, the answers explained, an expanded discussion with hyperlinked guidance, a decision algorithm and a certificate.
By the end of this unit you will be able to
Five steps, in order
Answer first, then learn. Each section unlocks as you progress, and your progress is saved in this browser.
1Three single best answer questions
Commit to an answer for all three single best answer questions, then submit. Sections 2–4 open on submission; your certificate opens at 2/3 or better.
Question 1 of 3
A trial reports that a new drug reduced hospital admission compared with placebo: relative risk 0.80 (95% confidence interval 0.65 to 0.98).
Which statement is correct?
Question 2 of 3
A small trial of 60 patients finds no statistically significant difference in pain scores between acupuncture and sham acupuncture (p = 0.21). A subsequent trial of 1,200 patients shows a clinically important benefit.
The first trial most likely made which error?
Question 3 of 3
An ICB compares the proportion of patients with hypertension treated to target in two practices. Practice A: 62% (95% CI 58% to 66%). Practice B: 71% (95% CI 68% to 74%).
What can be concluded?
2Answers explained
Question 1
Correct answer D. The result is statistically significant at the 5% level because the interval does not include 1
For ratios (RR, OR, HR) the value meaning “no difference” is 1. The 95% CI 0.65–0.98 lies wholly below 1, so p < 0.05. For differences (mean difference, ARR) the null value is 0 — option B applies the wrong null.
The upper limit (0.98) is close to 1, so the true effect could be as small as a 2% relative reduction — relevant to clinical significance.
Question 2
Correct answer B. Type II error
Failing to detect a difference that really exists is a type II error (β), a false negative. Small trials are underpowered. Power = 1 − β, conventionally 80–90%.
A type I error (α) is the opposite — a false positive, rejecting a true null hypothesis — and its probability is set by the significance level (usually 0.05).
Question 3
Correct answer D. The difference is statistically significant because the confidence intervals do not overlap
If two 95% confidence intervals do not overlap, the difference is statistically significant (p < 0.05). Here A’s upper limit (66%) is below B’s lower limit (68%).
The reverse is not reliable: intervals that overlap slightly can still represent a significant difference. If intervals overlap, you need a formal test of the difference.
Statistical significance does not explain the reason: case-mix, coding and deprivation may all contribute (option B over-interprets).
3Expanded discussion, worked examples & guidance
1. The null hypothesis and the p-value
The null hypothesis (H₀) states that there is no difference. The p-value is the probability of obtaining a result at least as extreme as the one observed if the null hypothesis were true.
- It is not the probability that the null hypothesis is true.
- It is not the probability that the result occurred “by chance” in the everyday sense.
- It says nothing about the size or importance of an effect — a huge trial can make a trivial difference “significant”.
2. Confidence intervals
A 95% CI is the range within which the true population value would lie in 95% of repeated samples. It conveys both significance (does it include the null value?) and precision (how wide is it?).
Worked example 1 — reading five intervals
| Result | Significant? | Interpretation |
|---|---|---|
| HR 0.72 (0.61–0.85) | Yes | 28% lower hazard; precise |
| OR 1.40 (0.95–2.06) | No | Includes 1; possible harm, inconclusive |
| Mean difference in SBP −4.1 mmHg (−6.3 to −1.9) | Yes | Excludes 0 |
| ARR 1.2% (−0.4% to 2.8%) | No | Includes 0 |
| RR 0.50 (0.10–2.50) | No | Very imprecise — underpowered |
Answer. Check the null value first (1 or 0), then the width, then whether the whole interval is clinically important.
3. Errors, power and sample size
| H₀ true (no real effect) | H₀ false (real effect) | |
|---|---|---|
| Reject H₀ (“significant”) | Type I error (α) — false positive | Correct (power = 1 − β) |
| Do not reject H₀ | Correct | Type II error (β) — false negative |
Power increases with: a larger sample size, a larger true effect, less variability in the measurements, and a higher α (at the cost of more false positives). Trials calculate their sample size in advance to achieve 80–90% power to detect the minimal clinically important difference.
Worked example 2 — multiple testing
A trial reports 20 secondary outcomes, and one is “significant” at p = 0.04.
- With α = 0.05, about 1 in 20 tests will be significant by chance alone when there is no true effect.
- Twenty tests → roughly one false positive expected.
Answer. Treat it as hypothesis-generating. Corrections such as Bonferroni (α divided by the number of tests) reduce type I error.
4. Statistical vs clinical significance
A trial of 40,000 patients may show a 1 mmHg blood-pressure difference with p < 0.001: statistically significant, clinically trivial. Conversely, a small trial with a wide interval including 1 may still be compatible with an important benefit (absence of evidence is not evidence of absence). Judge clinical importance from the size of the effect and the whole confidence interval.
- One-tailed tests are rarely justified; two-tailed is the default.
- “Not significant” ≠ “no difference”.
- A 99% CI is wider than a 95% CI from the same data.
References and further reading
Guidance checked at the last editorial review (28 September 2026). Always use the current version.
4Decision algorithm
Interpreting any result with a confidence interval
Start
Identify the effect measure and its 95% CI.
Decision
Is it a ratio (RR, OR, HR)?
Decision
Does the CI include the null value?
Decision
Comparing two groups’ CIs: do they overlap?
Outcome
Is the whole interval clinically important? Consider the minimal clinically important difference, baseline risk and NNT.
Revision summary; apply clinical judgement and local pathways.
5Certificate of completion
You have met the pass mark. Enter your details to issue your personalised certificate of completion, then save it as a PDF for your appraisal or revalidation portfolio.
Your certificate is issued immediately as a PDF. Your details stay in your browser and are not sent to MD Acumen.
Your completion and score have also been recorded by your organisation’s learning platform.
MRCGP AKT · CAS Programmes · For institutions · Legal & regulatory · enquiry@mdacumen.com
Educational content for healthcare professionals; apply clinical judgement and current local guidance. MRCGP is a registered trademark of the Royal College of General Practitioners. These units are independent, curriculum-aligned revision resources and are not endorsed by the RCGP.
© 2026 MD Acumen Ltd · Registered in England and Wales · Company no. 16538952 · ICO registration ZC105897