Overview
Statistical power is the probability of correctly rejecting the null hypothesis when it is false in a given sample. For further discussion of Power Analysis, see Jacob Cohen's classic 1992 Psychological Bulletin Power Primer or Statsoft, or Faul, Erdfelder, Lang, and Buchner, 2007.Power analysis is particularly useful when planning a study. There are many trade-offs when designing a study. Awareness of the trade-offs and how they relate to statistical power is particularly important. For example, you might be confronted with the decision of whether to use a subtle experimental manipulation (small effect size) that is of greater applied relevance or a stronger experimental manipulation (large effect size) which is less realistic. Being aware of the relative power of the two options helps to highlight whether the less realistic, but more statistically powerful, option is the better choice.
The website for G-Power 3 contains help material for many common scenarios. G*Power 3 makes it easy to get precise values of your statistical power for your given scenario. If G*Power 3 is insufficient for your needs, you may want to consider writing a simulation in a language such as R (see William Revelle's comments)
The following four plots show examples of the great plots that G*Power 3 can generate. I've also selected these four plots because I think that researchers in psychology should have an intuition about the statistical power of what may be the most common statistical tests: (a) the t-test of differences between independent groups, and (b) the correlation coefficient. Most studies in psychology have many variables and many hypotheses. However, many, but not all, of these hypotheses can be reduced to correlations between two numeric variables (e.g., IQ and job performance) or differences in a numeric variable between two groups (happiness levels between those receiving counselling and those not receiving counselling). Thus, internalising the approximate values of these four plots should help you when you design studies and when you read and evaluate the research literature.
What statistical power do I have given my sample size and assumed population correlation?
The rough rules of thumb can help: r = .1 is small; r = .3 is medium; r = .5 is large.
What statistical power do I have given my sample size and assumed population difference between group means?
Rough rules of thumb: d = .2 is small; r = .5 is medium; r = .8 is large.
What sample size do I need given a desired power an assumed population correlation?
The following two plots are useful when deciding what sample you need.
What sample size do I need given a desired power an assumed population difference between group means?
Related Resources
- Quick-R on Power Analysis in R
- Notes and audio from a talk I gave on Power Analysis, Null Hypothesis Significance Testing, and effect size
- Slides with notes form a talk I gave on descriptive statistics. The latter part of the talk is on power analysis.
