Significance testing

Effect sizes, standard errors, confidence intervals and p-values

Dr Massimiliano Canzi

From sample to population

Did you know: when George Shaw of the British Museum examined one of the first platypus specimens, around 1799, he suspected a hoax, and cut into it to look for stitches.

Source: Discover Magazine

Today

Samples and populations
How big is an effect? Cohen’s d
How precise is an estimate? Standard errors and confidence intervals
Null hypotheses and the t statistic
What a p-value is, and what it is not

So far

Lectures 3 to 8 described the data we have: means, SDs, slopes, differences between groups.

But we never care about these 120 vowels. We care about vowels before voiced consonants in general.

Today: how far can we trust a number computed from a sample?

Samples and populations

	Population	Sample
What it is	everything we want to draw conclusions about: all speakers of a language, all tokens of a vowel	the part of it we actually measured
Its mean	a parameter, written `μ`, usually unknown	a statistic, written x, which we can compute

We use the statistic as our estimate of the parameter. Today is about how good that estimate is.

A population we know

To see how samples behave, simulate a population whose truth we know:

set.seed(1)
pop <- rnorm(100000, mean = 140, sd = 25)    # 100,000 vowel durations (ms)
c(mean = mean(pop), sd = sd(pop))

     mean        sd 
139.94390  25.08808

Samples differ

Draw a sample of 10, and compute its mean:

set.seed(2)
mean(sample(pop, 10))

[1] 146.7041

mean(sample(pop, 10))

[1] 144.5591

mean(sample(pop, 10))

[1] 133.2498

Every sample gives a different estimate.

Sampling variation

2000 samples each. Dashed: the true mean, 140 ms. Larger samples give less variable estimates.

Effect size

Did you know: a platypus’s bill has about 40,000 electroreceptors. Underwater, it closes its eyes, ears and nose, and finds prey by the weak electric signals of their muscles.

Source: American Museum of Natural History

How big is a difference?

Back to lecture 7’s vowels:

vowels |> summarise(mean = mean(duration), sd = sd(duration), n = n(), .by = voicing)

# A tibble: 2 × 4
  voicing    mean    sd     n
  <fct>     <dbl> <dbl> <int>
1 voiceless  124.  19.7    60
2 voiced     162.  19.8    60

A difference of about 38 ms. Big or small?

That depends on how much durations vary anyway. The same difference is huge if tokens vary by 5 ms, and tiny if they vary by 200 ms.

Cohen’s d

The difference between two means, in units of their standard deviation:

d = (mean₂ − mean₁) / pooled SD

The pooled SD combines the two groups’ SDs, weighted by their sample sizes:

s <- vowels |> summarise(m = mean(duration), sd = sd(duration), n = n(), .by = voicing)

pooled_sd <- sqrt(((s$n[1] - 1) * s$sd[1]^2 + (s$n[2] - 1) * s$sd[2]^2) / (sum(s$n) - 2))
(s$m[2] - s$m[1]) / pooled_sd

[1] 1.911677

The voiced mean is almost 2 SDs above the voiceless mean.

Cohen’s d in R

With the {effsize} package (install once with install.packages("effsize")):

library(effsize)
cohen.d(duration ~ voicing, data = vowels)


Cohen's d

d estimate: -1.911677 (large)
95 percent confidence interval:
    lower     upper 
-2.348058 -1.475295

cohen.d() subtracts the second level from the first, so d is negative here. The size is what matters: 1.91.

Interpreting d

Cohen’s benchmarks: 0.2 small, 0.5 medium, 0.8 large. They are rough conventions: what counts as large depends on the field and the measure.

Standard errors and confidence intervals

Did you know: male platypuses have a venomous spur on each ankle, most active in the breeding season. The venom is not lethal to humans, but very painful.

Source: Australian Museum

The standard error

How much would a sample mean vary from sample to sample? The standard error:

SE = SD / √n

Check it against the simulation behind the plot, 2000 sample means for each sample size:

set.seed(2)
means <- bind_rows(
  tibble(n = "n = 10", mean = replicate(2000, mean(sample(pop, 10)))),
  tibble(n = "n = 50", mean = replicate(2000, mean(sample(pop, 50))))
)
c(simulated_n10 = sd(means$mean[means$n == "n = 10"]), formula_n10 = 25 / sqrt(10),
  simulated_n50 = sd(means$mean[means$n == "n = 50"]), formula_n50 = 25 / sqrt(50))

simulated_n10   formula_n10 simulated_n50   formula_n50 
     7.800182      7.905694      3.562828      3.535534

The SD describes the data, and the SE describes the estimate.

The SE shrinks with n

To halve the SE, you need four times as many observations.

Confidence intervals

A 95% confidence interval: the estimate, plus and minus about two standard errors.

set.seed(3)
x  <- sample(pop, 20)
se <- sd(x) / sqrt(length(x))
t_crit <- qt(0.975, df = length(x) - 1)      # about 2.09 for n = 20

c(lower = mean(x) - t_crit * se, mean = mean(x), upper = mean(x) + t_crit * se)

   lower     mean    upper 
138.2636 151.8480 165.4324

t.test(x)$conf.int

[1] 138.2636 165.4324
attr(,"conf.level")
[1] 0.95

95% of intervals

100 samples of 20. Pink: the 8 intervals that miss the true mean. In the long run, 95% of intervals contain it.

What a confidence interval means

The procedure captures the true value in 95% of samples. Each interval is a range of values that are compatible with the data: narrow intervals come from precise estimates, wide ones from imprecise estimates.

It does not mean “there is a 95% probability that the true value lies in this particular interval”. The true value is fixed, while the interval changes from sample to sample.

Null hypotheses

Did you know: the platypus is one of the few mammals that lay eggs, usually two. The young then drink milk from two patches of fur on their mother’s belly.

Source: Australian Museum

Null and alternative hypotheses

The null hypothesis H₀ says there is no difference: μ_voiced = μ_voiceless
The alternative hypothesis H₁ says there is one: μ_voiced ≠ μ_voiceless

Significance testing asks: how compatible are our data with H₀?

If the data would be very surprising in a world where H₀ is true, we treat that as evidence against H₀.

A small study

Only 8 vowels per group:

set.seed(21)
small <- tibble(
  voicing  = rep(c("voiceless", "voiced"), each = 8),
  duration = round(c(rnorm(8, 120, 20), rnorm(8, 140, 22)))
) |>
  mutate(voicing = fct_relevel(voicing, "voiceless"))

obs_diff <- diff(tapply(small$duration, small$voicing, mean))
obs_diff

voiced 
 15.75

Voiced vowels are about 16 ms longer. Could a difference this big appear by chance, if voicing had no effect at all?

Simulating the null

If H₀ is true, the labels “voiced” and “voiceless” are arbitrary. Shuffle them, and see which differences chance alone produces:

set.seed(5)
null_diffs <- replicate(5000, {
  shuffled <- sample(small$voicing)
  diff(tapply(small$duration, shuffled, mean))
})
mean(abs(null_diffs) >= abs(obs_diff))

[1] 0.2602

In about 26% of shuffles, the difference is at least as large as ours.

Simulating the null

Pink: differences at least as extreme as the observed one (dashed). Our result is not surprising under H₀.

The t statistic

Instead of simulating, we can measure the difference in units of its standard error:

t = difference / SE of the difference

s_small <- small |> summarise(m = mean(duration), v = var(duration), n = n(), .by = voicing)
se_diff <- sqrt(sum(s_small$v / s_small$n))

obs_diff / se_diff

  voiced 
1.207035

A t of about 1.2: the difference is only a little larger than its SE.

p-values

Did you know: platypus fur glows green or cyan under UV light. The glow was found in museum specimens in 2020; nobody knows yet what it is for.

Source: Anich et al. (2020), Mammalia

The t-distribution

If H₀ is true, t follows a known shape: the t-distribution, which depends on the degrees of freedom (roughly, the sample size).

The p-value is the pink area: the probability of a t at least this extreme, if H₀ were true.

Computing p

t_obs  <- t.test(duration ~ voicing, data = small)
df_obs <- t_obs$parameter          # degrees of freedom
t_val  <- abs(t_obs$statistic)     # observed t, ignoring its sign
2 * pt(-t_val, df = df_obs)        # both tails

        t 
0.2477153

Or let t.test() do everything:

t.test(duration ~ voicing, data = small)


    Welch Two Sample t-test

data:  duration by voicing
t = -1.207, df = 13.791, p-value = 0.2477
alternative hypothesis: true difference in means between group voiceless and group voiced is not equal to 0
95 percent confidence interval:
 -43.77618  12.27618
sample estimates:
mean in group voiceless    mean in group voiced 
                124.875                 140.625

Two tails

Two-tailed test: H₁ is “a difference in either direction”; both tails count
One-tailed tests only count one direction. Use them only if the direction was fixed before seeing the data

t.test() is two-tailed by default. Note the p matches the shuffling simulation: 0.25 vs 0.26.

What a p-value is

The probability of obtaining data at least as extreme as ours, assuming H₀ is true.

It is not:

the probability that H₀ is true
the probability that the result is due to chance
a measure of how big or important the effect is

Chapter 10 of Winter is about these misinterpretations: we come back to it in lecture 13.

Significance

By convention, if p < α, with α usually 0.05, the result is called statistically significant and H₀ is rejected.

α = 0.05 is a convention, and “significant” means only “surprising under H₀”, and says nothing about importance
p = 0.049 and p = 0.051 are practically the same evidence

Back to the 120 vowels

t.test(duration ~ voicing, data = vowels)


    Welch Two Sample t-test

data:  duration by voicing
t = -10.471, df = 117.99, p-value < 2.2e-16
alternative hypothesis: true difference in means between group voiceless and group voiced is not equal to 0
95 percent confidence interval:
 -44.90931 -30.62403
sample estimates:
mean in group voiceless    mean in group voiced 
               124.3000                162.0667

With 60 tokens per group, t is about 10.5: a difference this large would be extremely unlikely if voicing had no effect.

Significance is not size

set.seed(4)
big <- tibble(group = rep(c("a", "b"), each = 5000),
              y     = c(rnorm(5000, 100, 15), rnorm(5000, 101, 15)))

study	Cohen’s d	p
Small study: 8 vowels per group	0.60	0.250
Huge study: 5000 per group	0.05	0.022

A medium effect that is not significant, and a tiny effect that is. The p-value mixes effect size and sample size.

Always report the effect size and its confidence interval alongside p.

Discussion

Study A tests a pronunciation training with 3,000 learners: ratings improve by 0.1 points on a 7-point scale, p < .001. Study B tests a different training with 12 learners: ratings improve by 1.2 points, p = .09.

Which result is more convincing? Which training would you try in your own class?
What would you want to know about each study before deciding?

t-tests are linear models

tidy(lm(duration ~ voicing, data = vowels))

# A tibble: 2 × 5
  term          estimate std.error statistic  p.value
  <chr>            <dbl>     <dbl>     <dbl>    <dbl>
1 (Intercept)      124.       2.55      48.7 5.10e-80
2 voicingvoiced     37.8      3.61      10.5 1.53e-18

t.test(duration ~ voicing, data = vowels, var.equal = TRUE)$statistic

        t 
-10.47068

The slope’s t value and p.value in lm() are exactly a t-test (with equal variances assumed). Significance tests for regression coefficients: Chapter 11.

Reporting

With inline R (lecture 3), every number comes from the analysis:

Vowels before voiced consonants were 37.8 ms longer than before voiceless consonants (95% CI [30.6, 44.9]), t(118) = 10.47, p < .001, d = 1.91.

The effect and its CI, in the units of the data
The test statistic, its degrees of freedom, and p
A standardised effect size

Your turn

Exercises

With palmerpenguins::penguins, keep only Adelie and Chinstrap penguins (use filter() and then droplevels()):

Compare their body_mass_g: compute Cohen’s d and run a t-test
Do the same for flipper_length_mm
Which difference is significant? Report each with its 95% CI
Does the first result show that the two species weigh the same?

Solutions

two <- palmerpenguins::penguins |>
  filter(species %in% c("Adelie", "Chinstrap")) |>
  drop_na(body_mass_g, flipper_length_mm) |>
  mutate(species = droplevels(species))

cohen.d(body_mass_g ~ species, data = two)$estimate

[1] -0.07420226

t.test(body_mass_g ~ species, data = two)[c("statistic", "p.value", "conf.int")]

$statistic
         t 
-0.5430902 

$p.value
[1] 0.5878608

$conf.int
[1] -150.38481   85.53284
attr(,"conf.level")
[1] 0.95

Solutions

cohen.d(flipper_length_mm ~ species, data = two)$estimate

[1] -0.8724636

t.test(flipper_length_mm ~ species, data = two)[c("statistic", "p.value", "conf.int")]

$statistic
        t 
-5.780385 

$p.value
[1] 6.049267e-08

$conf.int
[1] -7.880530 -3.859244
attr(,"conf.level")
[1] 0.95

Flipper length differs clearly; body mass does not reach significance. But a non-significant result is not evidence of no difference: the CI for body mass runs from -150 to 86 g. The data are compatible with no difference, but just as compatible with differences of that size.

Summary

We estimate population parameters from sample statistics
Cohen’s d expresses a difference in SD units
The SE (SD / √n) says how much an estimate varies across samples, and a 95% CI gives the values compatible with the data
t = difference / SE; p = how surprising t would be if H₀ were true
Significant ≠ large ≠ important: report effect sizes and CIs

Slides and code

These slides

mcanzi.de/ws26_statistics/09_significance_testing.html

Code, data and all lectures

github.com/mcanzi/ws26_statistics