Power analysis by simulation

Statistical power is the probability that a study detects an effect of a given size, if that effect is real. It depends on the size of the effect, on how noisy the data are, on the number of participants and items, and on the significance level. Once the design and the effect of interest are fixed, only the sample size is still up to us. Green and MacLeod (2016) point out that the usual target of 80% is a convention, with no deeper justification.

For a simple design, a formula gives the power. For the designs common in linguistics and psychology, with participants and items as crossed random effects, there is no simple formula, and power is estimated by simulation (Brysbaert & Stevens, 2018): make up a world in which the effect exists, simulate a study in it, analyse the data as planned, and repeat many times. Since the effect is real in every simulated data set, the proportion of significant results is the power.

Small studies exaggerate

The figure below runs 2,000 studies, each comparing two groups. The true difference between the groups is set by the first slider, in standard deviations (Cohen’s d), and the number of participants per group by the second. Each bar counts the studies whose estimate fell in that range, coloured by whether the estimate was significant.

With 20 participants per group and d = 0.3, about one study in seven is significant, close to the 15% the formula predicts. Only the large estimates pass the significance threshold, so the significant studies overestimate the effect, on average by a factor of two or more. Gelman and Carlin (2014) call this a Type M (magnitude) error. A few significant studies even get the direction wrong, a Type S (sign) error. Increase the sample size and both errors shrink, until the significant estimates centre on the true effect.

If journals mostly publish significant results, the published effects of small studies are too large. Vasishth et al. (2018) showed this for a psycholinguistic finding with seven direct replications. A power analysis that takes its effect size from such a publication then promises more power than the next study will have. A better basis is the smallest effect that would matter for the theory.

How many participants?

The same calculation for every sample size gives a power curve. The highlighted curve follows the effect size of the first figure, and the grey curves show the conventional small, medium and large effects of d = 0.2, 0.5 and 0.8.

Halving the effect roughly quadruples the sample size needed: 64 per group for d = 0.5, but 394 for d = 0.2.

More participants or more items?

In the lecture, we plan a replication of a lexical decision study in which L1 and L2 speakers respond to words of different frequencies. The question is the interaction: is the frequency effect larger for L2 speakers? Participants differ in how strongly frequency affects them (the SD of the slopes), and each response adds trial-level noise (an SD of 70 ms). In R, the {simr} package (Green & MacLeod, 2016) answers the question by simulating from a mixed model, refitting it many times, and adding participants or words with extend() and powerCurve().

A browser cannot refit a mixed model a thousand times while you move a slider. This figure uses a simpler analysis of the same design instead: a regression slope for each participant, then a t-test comparing the slopes of the two groups (Lorch & Myers, 1990). Its numbers differ somewhat from those of the mixed model, while the curves have the same shape.

Adding participants raises the power steadily. Adding words helps at first and then levels off, because more words only make each participant’s slope more precise, while the real differences between participants stay as they are. The ceiling shows the power with infinitely many words. Set the SD of the slopes to zero and the words become as useful as the participants.

Post hoc power

Post hoc power, also called observed or retrospective power, is computed after a study, with the effect size estimated from the study’s own data. Reviewers sometimes ask for it to show whether a non-significant study was “underpowered”. Heinsberg and Weeks (2022) simulated the same mouse experiment at two universities, with 100 animals each, and different true effects:

	Estimate (95% CI)	p	Post hoc power	True effect	True power
A	0.290 (−0.002, 0.582)	.052	0.584	0.402	0.841
B	0.275 (−0.002, 0.553)	.052	0.537	0.201	0.322

The two studies look almost the same, and so does their post hoc power. Their true power, 84% against 32%, cannot be read from either data set.

The figure below repeats this on a larger scale. Each point is one simulated study, all with the same design and the same true power.

Wherever the true power lies, the points fall on a single curve of the p-value: post hoc power is about 50% at p = .05, lower whenever p is larger, and higher whenever p is smaller (Hoenig & Heisey, 2001). Change the true effect and the points slide along the same curve. Because post hoc power is a transformation of the p-value, a non-significant result always comes with low post hoc power, whether the study missed a real effect or the effect does not exist.

What to report instead

To help readers judge a non-significant result, report the estimate with its confidence interval, which shows the effect sizes the data are compatible with. Report the power analysis you ran before the study, with the effect size you chose and the reason for it. If there was none, a sensitivity analysis gives the smallest effect the design had 80% power to detect, computed from the design and the assumed variances, without using the observed effect.

References

Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of Cognition, 1(1), 9. https://doi.org/10.5334/joc.10

Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing Type S (sign) and Type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651. https://doi.org/10.1177/1745691614551642

Green, P., & MacLeod, C. J. (2016). SIMR: An R package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution, 7(4), 493–498. https://doi.org/10.1111/2041-210X.12504

Heinsberg, L. W., & Weeks, D. E. (2022). Post hoc power is not informative. Genetic Epidemiology, 46(7), 390–394. https://doi.org/10.1002/gepi.22464

Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897

Lorch, R. F., & Myers, J. L. (1990). Regression analyses of repeated measures data in cognitive research. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(1), 149–157. https://doi.org/10.1037/0278-7393.16.1.149

Vasishth, S., Mertzen, D., Jäger, L. A., & Gelman, A. (2018). The statistical significance filter leads to overoptimistic expectations of replicability. Journal of Memory and Language, 103, 151–175. https://doi.org/10.1016/j.jml.2018.07.004

All simulations run in your browser: the two-group studies use t-tests, and the power curves use the exact power of the two-sided t-test. The lecture slides and their R code are on the teaching page.