The audio (17MB) for the talk is available online, as are the Slides, and a PDF version of the content below.
Statistics for a Psychology Thesis
The Big Picture: It all starts with a research question. We design or obtain empirical data that might assist in answering a research question. Statistics is a tool for summarising empirical reality and answering questions. Knowing how to link statistical analysis with the research question is a critical skill. One reason that psychology is special is that it attempts to ground its knowledge in empirical reality. We put our ideas to the test. We are taught to be scientist-practitioners.Staying open minded: There is often a lot of pressure to obtain certain results, support certain hypotheses or test various complex statistical models. My advice: Stuff them all. Be ethical. Stay true to yourself. Let the data speak to you in all its non-conforming brutal honesty. When you analyse data, discard all agendas. If the sample size is too small to say much conclusively, acknowledge this. If the data does not support your hypotheses, accept it and try to understand why. If you have data based on a correlational design, acknowledge that there are many other competing explanations to the particular causal relationship you might be proposing. The whole point of the empirical process is ABSOLUTELY NOT to act as a checkbox for some ill-conceived theory.
Democracy and statistics: Ideologically based positions are common in public debate. Well designed and analysed empirical studies can be powerful in setting out the “facts” that any side of a debate needs to acknowledge. However, empirical research can be biased and hijacked for particular agendas. Having citizens that are able to critically evaluate empirical research and are able to honestly and skilfully conduct and analyse their own research is important for maintaining a healthy democracy. The rhetorical question I ask often is: “Can you create knowledge from empirical observations? Or must you rely on others to digest it for you?”
Statistics as reasoned decision making: Perhaps because of statistics association with mathematics or perhaps because of the way we are taught statistics and associated rules of thumb, it may appear like there is always a right and wrong way to do statistics. In reality, statistics is just like other domains. There are different ways of doing what we do, and the key is to justify our choices based on reasoned decision making. Reasoned decision making involves weighing up the pros and cons of different choices in terms of such factors as the purpose of the analyses, the nature of the data, and recommendations from statistics textbooks and journals. The idea is to explain your reasons in a logical and coherent way just as you would justify any other decision in life.
Null Hypothesis Significance Testing (NHST): a p value indicates the probability of observing results in a sample as or more extreme as those obtained assuming the null hypothesis is true. NHST is a tool for ruling out random sampling as an explanation for the observed relationship. Failing to reject the null hypothesis does not prove the null hypothesis. Statistical significance does not equal practical importance.
A modern orientation to data analysis: Answers to research questions depend on the status of population parameters. Empirical research aims to estimate population parameters (e.g., size of a correlation, size of group differences, etc.). NHST is still relevant. However, confidence intervals around effect sizes and a general orientation of meta-analytic thinking leads to better thinking about research problems, results interpretation and study design than does NHST.
Effect Size: Thinking about effect sizes is a philosophical shift which emphasises thinking about the practical importance of research findings. Effect size measures may be standardised (e.g., cohen’s d, r, odds ratio, etc.) or unstandardised (e.g., difference between group means, unstandardised regression coefficient, etc.). Think about what this means for practitioners using the knowledge. Contextualise the effect size in terms of its statistical definition, prior research in the area, prior research in the broader discipline and only finally using Cohen’s rules of thumb.
Confidence Intervals: Confidence intervals indicate how confident we can be that the population parameter is between given values (e.g., 95% confidence). Confidence intervals focus our attention on population values, which is what theory is all about. They highlight our degree of uncertainty. If the confidence interval includes the null hypothesis value, we know that we do not have a statistically significant result. In this way confidence intervals provide similar information as NHST, but also much more.
Power Analysis: Having an adequate sample size to assess your research question is important. Statistical power is the probability of finding a statistically significant result for a particular parameter in a particular study where the null hypothesis is false. Power increases with larger population effect sizes, larger sample sizes and less stringent alpha. G-Power 3 (just Google G Power 3) is excellent free software for running power analyses.
Accuracy in Parameter Estimation (AIPE): Power analysis is aligned with NHST. AIPE is aligned with confidence intervals around effect sizes and meta-analytic thinking. AIPE attempts to work out the size of the confidence interval we will have for any given sample size and effect size. The aim is to have a sample size that will give us sufficiently small confidence intervals around our obtained effect sizes to draw the conclusions about effect sizes that we want to draw.
Meta Analytic thinking: Meta analytic thinking involves "a) the prospective formulation of study expectations and design by explicitly invoking prior effect size measures and b) the retrospective interpretation of new results, once they are in hand, via explicit, direct comparison with the prior effect sizes in the related literature" (Thompson, 2008, p.28). This approach incorporates the idea that we read the literature in terms of confidence intervals around effect sizes and we design studies with sufficient power to test for the effect size and sufficient potential to refine our estimate of the parameter under study.
Sharing data with the world: Imagine the potential for knowledge advancement if data underlying published articles was readily assessable to be re-analysed. You could learn about data analysis by trying to replicate analyses on data similar to your thesis. You could do meta-analyses using the complete data sets. You could run analyses that the original authors did not report. You could be an active consumer of their results, rather than a passive receiver. Others would be more receptive to your ideas if they could subject your analyses to scrutiny. Such a model fits with the idea of being open minded, distributing knowledge, and emphasising meta-analytic thinking. In many situations concerns about confidentiality, intellectual property, and the data collector’s right to first publish can be overcome. The message: Consider making your data publicly available after you have published it in a journal.
Software: Be aware of the different statistical packages that are available. SPSS is relatively easy to use. “R” (www.r-project.org/) is an open source (i.e., free software) alternative and is worth learning if you want to become a serious data analyst. It has cutting edge features (e.g., polychoric correlations, bootstrapping, reports for psychological tests, meta analysis, multilevel modelling, item analysis, etc.) , amazing potential for automation and customised output, and encourages a better orientation towards running analyses. Results can be fed back into subsequent analyses; graphs and output can be customised to your needs; it forces you to document your analysis process; it generally requires that you know a little more about what you are doing; and it leads to an approach of being responsive to what the data is saying and adjusting analyses accordingly. For an introduction for psychologists, see (personality-project.org/r/r.guide.html).
Learning Statistics: For many people in psychology, statistics is not something done everyday. A strategy is needed to identify and acquire the skills required to analyse your thesis data. Set out a statistical self-development plan possibly in conjunction with a statistical adviser, identifying things such as books and chapters to read, practice exercises to do, formal courses to do, etc. It is important to get practical experience analysing other datasets before you tackle your thesis dataset.
The right books: It is critical to have the right resources. Get a comprehensive multivariate book (Tabachnick & Fiddel – Using Multivariate Statistics or Hair et al – Multivariate Data Analysis). Get a clear, entertaining, insightful and SPSS-focused book (Field – Discovering Statistics Using SPSS). Get an easy to follow SPSS cookbook for doing your thesis (Pallant – SPSS Survival Manual).
Using statistical consultants: be prepared; be clear about your questions; recognise that statistical consultants are there to provide advice about options and that many decisions are intimately tied up with theoretical considerations and should be made by the researcher.
Taking your time: As Wright (2003) so aptly put it: “Conducting data analysis is like drinking a fine wine. It is important to swirl and sniff the wine, to unpack the complex bouquet and to appreciate the experience.” A good dataset often has a lot to say. When we’ve often spent many months designing and collecting data, it is important to give the data the time to speak to us. Often, this will require us to change how we conceptualise the phenomena. Explore the data; produce lots of graphs; consider the individual cases; assess the assumptions; reflect on the statistical models used; reflect on the metrics of the variables used; and value basic descriptive statistics.
Telling a story: The results section should be the most interesting section of a thesis. It should show how your results answer your research question. It should show the reasons for your statistical decisions. It should explain why the statistical output is interesting. You’ve whet the reader’s appetite with the introduction and method, the results section is where you get to convert your empirical observations into a contribution that advances the sum of all human knowledge.
