Basic Advice
I teach about carryover effects in a research methods subject. My basic advice could probably be distilled to something like this:
- If order is the focus of the analysis (e.g., skill acquisition looking at effects of practice), then don't worry about order effects
- If order effects are very strong, it may be better to stick to between subjects designs
- if order effects are small or moderate or unknown, typical design strategy depends on the number of levels of the within-subjects factor of interest.
- If there are few levels (e.g., 2,3,4 perhaps), present all orders (counterbalance)
- If there are more levels (e.g., 4+ perhaps), adopt a latin squares approach or randomise ordering
Design and Inference Issues
The following captures some of my approach to assessing, controlling and drawing inferences in studies with potential order effects.
- At study design phase, brainstorm the plausible causal mechanisms that would yield order effects and the likely size of the order effects
- At study design phase, think of as many strategies that could minimise the order effects as possible, such as separating the two conditions by a longer periods of time or reducing the connection in participants' minds between the conditions.
- Assess the effect of order and statistically control its effect (by including it as a predictor in the ANOVA or regression)
- Assess the order by interaction effect. The presence of this is more problematic.
- When assessing order effects and interaction effects involving order, consider effect sizes and confidence intervals around the effect size estimate. If the order effects or interaction effects are small, then the confounding of the conclusion is likely to be small.
- Compare the effect of the within subjects factor with the between subject effect at time 1. If there is no order effect, they should be estimating the same difference between group means; it's just that the within subjects estimate will typically have more statistical power.
- Remember that order effects may interact with participant, and this will be difficult to assess. The result of this is that the order and order by interaction effects may not detect the "order" effect, because the order is affecting individuals in different ways.
- Reflect on the data obtained and update theory about the causal mechanisms influencing order effects and make design recommendations about further minimisation of order effects for future research
- In addition, strategies for minimising order effects can be studied by including them as an independent variable (e.g., time between repeated measures conditions). This represents a way of moving forward on carry over effects.
Finally, in terms of general advice, it's important to remember that the causal effect of an independent variable is an effect on an individual. Between subjects designs allow us to estimate the mean of the individual effects. However, if you want to have some understanding of the variability in the effects, a repeated measures design can be very useful.
Case Study Applying the above Ideas to My Research
I use some of my research looking at personality faking to illustrate the ideas:
I often use a 2 by 2 design, with a within subjects factor of test context (honest versus job), and a between subjects factor of order (honest first or job first).
I also use theory to anticipate any order effects and interpret any order effects that I find. For example, theory might suggest that people will try to be consistent with their prior personality test responses. Thus, if they did the honest test first, they will appear to answer in a less conscientious way on a job test. Or there might be a learning process whereby being exposed to a test allows you to understand how it works and makes it easier to distort responses in a favourable way.
Thus, I design studies that have a period of time, at least a few days, between completing the tests. I also try to reduce the connection between the two tests in participants' minds.
I assess the order and order by condition effects to assess the extent and nature of order effects. The combination of learning and consistency effects might result in an interaction effect.
I can also check that the repeated measures estimates of differences between contexts is broadly similar with the between subjects estimates. If there are no ordering effects, then the estimates of the mean difference in conscientiousness levels between honest and job contexts should be the same whether we compare the groups at time 1 or we compare the differences over time.
If order effects prove to be a nuisance, I can go forward with further research exploring methods of order effect minimisation or I can stick with between subjects designs. However, for me, my research interests are specifically looking at how individuals change. Thus, I would much prefer the order effect minimisation strategy than have to fall back on between subjects designs. At any rate with a 2 by 2 design, I already have the between subjects design, if I just take time 1.
