OverviewWhen reporting the results of the study New York Times states that:
"Mothers still do most of the housework, spending 27 percent of their time on it, on average, compared with 18 percent for fathers and 3 percent for children (giving an allowance made no difference)." (New York Times, 2010)
However, this is relatively poor data for making claims about:
- What percentage of housework men and women do
- Whether an allowance is effective at increasing house work
Statistical Power on a difference between group meansThere are also issues with the claims that an allowance is not effective at increasing housework levels of children. First, the study is observational making it difficult to assess causal effects. A randomised control trial would be more appropriate. However, regardless of this, the sample size issue is even more problematic. Presumably something like an independent groups t-test was done at the family-level to see whether families that give an allowance to their children on average had children who did more housework than families that did not give an allowance.
The statistical power for testing such a difference with a sample size of 32 is very low. Assuming that an allowance has a medium effect (i.e., a half standard deviation increase) on household work a standard alpha level of .05, and that there were equal numbers of families that had an allowance or did not have an allowance (16 in each), this would result in statistical power (i.e., the probability of correctly rejecting the null hypothesis of no differences) of .277 (see Figure 1). That is to say, the study had a 27.7% chance of rejecting the null hypothesis. Or more broadly, the study was more likely to accept the null hypothesis than reject it.
Figure 1. Screen shot of G*Power 3 showing statistical power of reported study given the above stated assumptions
In order to have reasonable statistical power (i.e., 80% power) given the above assumptions a sample size of 124 families (64 in each group) would be required (see Figure 2).
Figure 2. Screen shot of G*Power 3 showing sample size required to have 80% power given above stated assumptions
Concluding PointsMy main points are that:
- Improving the validity of social science measurement is important. The use of coded videos of real world settings creates exciting measurement possibilities.
- Even with improved measurement, a small sample size will result in large confidence intervals when estimating effects of interest.