In 2007 I presented a talk to postgraduate psychology students at The University of Melbourne. As part of the talk I produced a handout which summarised many of the key points that I felt were relevant for such an audience who needed to complete a thesis involving quantitative analysis. Reading over it two years later, I still agree with the ideas, even if my understanding may be a little more nuanced. For example, I'd now see meta-analytic thinking as a simple version of Bayesian statistics. Anyway, I thought I'd post it on the blog.

The audio (17MB) for the talk is available online, as are the Slides, and a PDF version of the content below.

##### Statistics for a Psychology Thesis

**The Big Picture: **It all starts with a research question. We design or
obtain empirical data that might assist in answering a research question.
Statistics is a tool for summarising empirical reality and answering questions.
Knowing how to link statistical analysis with the research question is a
critical skill. One reason that psychology is special is that it attempts to
ground its knowledge in empirical reality. We put our ideas to the test. We are
taught to be scientist-practitioners.

**Staying open minded: **There is often a lot of pressure to obtain certain
results, support certain hypotheses or test various complex statistical models.
My advice: Stuff them all. Be ethical. Stay true to yourself. Let the data speak
to you in all its non-conforming brutal honesty. When you analyse data, discard
all agendas. If the sample size is too small to say much conclusively,
acknowledge this. If the data does not support your hypotheses, accept it and
try to understand why. If you have data based on a correlational design,
acknowledge that there are many other competing explanations to the particular
causal relationship you might be proposing. The whole point of the empirical
process is ABSOLUTELY NOT to act as a checkbox for some ill-conceived theory.

**Democracy and statistics:** Ideologically based positions are common in
public debate. Well designed and analysed empirical studies can be powerful in
setting out the “facts” that any side of a debate needs to acknowledge. However,
empirical research can be biased and hijacked for particular agendas. Having
citizens that are able to critically evaluate empirical research and are able to
honestly and skilfully conduct and analyse their own research is important for
maintaining a healthy democracy. The rhetorical question I ask often is: “Can
you create knowledge from empirical observations? Or must you rely on others to
digest it for you?”

**Statistics as reasoned decision making: **Perhaps because of statistics
association with mathematics or perhaps because of the way we are taught
statistics and associated rules of thumb, it may appear like there is always a
right and wrong way to do statistics. In reality, statistics is just like other
domains. There are different ways of doing what we do, and the key is to justify
our choices based on reasoned decision making. Reasoned decision making involves
weighing up the pros and cons of different choices in terms of such factors as
the purpose of the analyses, the nature of the data, and recommendations from
statistics textbooks and journals. The idea is to explain your reasons in a
logical and coherent way just as you would justify any other decision in
life.

**Null Hypothesis Significance Testing (NHST): **a p value indicates the
probability of observing results in a sample as or more extreme as those
obtained assuming the null hypothesis is true. NHST is a tool for ruling out
random sampling as an explanation for the observed relationship. Failing to
reject the null hypothesis does not prove the null hypothesis. Statistical
significance does not equal practical importance.

**A modern orientation to data analysis: **Answers to research questions
depend on the status of population parameters. Empirical research aims to
estimate population parameters (e.g., size of a correlation, size of group
differences, etc.). NHST is still relevant. However, confidence intervals around
effect sizes and a general orientation of meta-analytic thinking leads to better
thinking about research problems, results interpretation and study design than
does NHST.

**Effect Size: **Thinking about effect sizes is a philosophical shift which
emphasises thinking about the practical importance of research findings. Effect
size measures may be standardised (e.g., cohen’s d, r, odds ratio, etc.) or
unstandardised (e.g., difference between group means, unstandardised regression
coefficient, etc.). Think about what this means for practitioners using the
knowledge. Contextualise the effect size in terms of its statistical definition,
prior research in the area, prior research in the broader discipline and only
finally using Cohen’s rules of thumb.

**Confidence Intervals: **Confidence intervals indicate how confident we can
be that the population parameter is between given values (e.g., 95% confidence).
Confidence intervals focus our attention on population values, which is what
theory is all about. They highlight our degree of uncertainty. If the confidence
interval includes the null hypothesis value, we know that we do not have a
statistically significant result. In this way confidence intervals provide
similar information as NHST, but also much more.

**Power Analysis: **Having an adequate sample size to assess your research
question is important. Statistical power is the probability of finding a
statistically significant result for a particular parameter in a particular
study where the null hypothesis is false.
Power increases with larger population effect sizes, larger sample sizes and
less stringent alpha. G-Power 3 (just Google G Power 3) is excellent free
software for running power analyses.

**Accuracy in Parameter Estimation (AIPE): **Power analysis is aligned with
NHST. AIPE is aligned with confidence intervals around effect sizes and
meta-analytic thinking. AIPE attempts to work out the size of the confidence
interval we will have for any given sample size and effect size. The aim is to
have a sample size that will give us sufficiently small confidence intervals
around our obtained effect sizes to draw the conclusions about effect sizes that
we want to draw.

**Meta Analytic thinking: **Meta analytic thinking involves "a) the
prospective formulation of study expectations and design by explicitly invoking
prior effect size measures and b) the retrospective interpretation of new
results, once they are in hand, via explicit, direct comparison with the prior
effect sizes in the related literature" (Thompson, 2008, p.28). This approach
incorporates the idea that we read the literature in terms of confidence
intervals around effect sizes and we design studies with sufficient power to
test for the effect size and sufficient potential to refine our estimate of the
parameter under study.

**Sharing data with the world: **Imagine the potential for knowledge
advancement if data underlying published articles was readily assessable to be
re-analysed. You could learn about data analysis by trying to replicate analyses
on data similar to your thesis. You could do meta-analyses using the complete
data sets. You could run analyses that the original authors did not report. You
could be an active consumer of their results, rather than a passive receiver.
Others would be more receptive to your ideas if they could subject your analyses
to scrutiny. Such a model fits with the idea of being open minded, distributing
knowledge, and emphasising meta-analytic thinking. In many situations concerns
about confidentiality, intellectual property, and the data collector’s right to
first publish can be overcome. The message: Consider making your data publicly
available after you have published it in a journal.

**Software: **Be aware of the different statistical packages that are
available. SPSS is relatively easy to use. “R” (www.r-project.org/) is an open source
(i.e., free software) alternative and is worth learning if you want to become a
serious data analyst. It has cutting edge features (e.g., polychoric
correlations, bootstrapping, reports for psychological tests, meta analysis,
multilevel modelling, item analysis, etc.) , amazing potential for automation
and customised output, and encourages a better orientation towards running
analyses. Results can be fed back into subsequent analyses; graphs and output
can be customised to your needs; it forces you to document your analysis
process; it generally requires that you know a little more about what you are
doing; and it leads to an approach of being responsive to what the data is
saying and adjusting analyses accordingly. For an introduction for
psychologists, see (personality-project.org/r/r.guide.html).

**Learning Statistics: **For many people in psychology, statistics is not
something done everyday. A strategy is needed to identify and acquire the skills
required to analyse your thesis data. Set out a statistical self-development
plan possibly in conjunction with a statistical adviser, identifying things such
as books and chapters to read, practice exercises to do, formal courses to do,
etc. It is important to get practical experience analysing other datasets before
you tackle your thesis dataset.

**The right books: **It is critical to have the right resources. Get a
comprehensive multivariate book (Tabachnick & Fiddel – Using Multivariate
Statistics or Hair et al – Multivariate Data Analysis). Get a clear,
entertaining, insightful and SPSS-focused book (Field – Discovering Statistics
Using SPSS). Get an easy to follow SPSS cookbook for doing your thesis (Pallant
– SPSS Survival Manual).

**Using statistical consultants:** be prepared; be clear about your
questions; recognise that statistical consultants are there to provide advice
about options and that many decisions are intimately tied up with theoretical
considerations and should be made by the researcher.

**Taking your time: **As Wright (2003) so aptly put it: “Conducting data
analysis is like drinking a fine wine. It is important to swirl and sniff the
wine, to unpack the complex bouquet and to appreciate the experience.” A good
dataset often has a lot to say. When we’ve often spent many months designing and
collecting data, it is important to give the data the time to speak to us.
Often, this will require us to change how we conceptualise the phenomena.
Explore the data; produce lots of graphs; consider the individual cases; assess
the assumptions; reflect on the statistical models used; reflect on the metrics
of the variables used; and value basic descriptive statistics.

**Telling a story:** The results section should be the most interesting
section of a thesis. It should show how your results answer your research
question. It should show the reasons for your statistical decisions. It should
explain why the statistical output is interesting. You’ve whet the reader’s
appetite with the introduction and method, the results section is where you get
to convert your empirical observations into a contribution that advances the sum
of all human knowledge.

Hi Jeromy, I've been finding your information on SEM very helpful. I'm planning on using it for my Masters thesis in Psych, is it worth doing an advanced course on this before attempting it?

ReplyDeleteRegards,

Jemima Coombs

(ex-student of your stats labs/lects)

Hi Jemima,

ReplyDeleteA course in SEM is often a good idea if you are planning on using the technique.

Feel free to send me an email to discuss further.

Please modify the statement on power here so that it's conditional rather than unconditional. Power is the probability of finding a significant effect when it's actually there. The way you've worded it includes error.

ReplyDeleteHi John, I agree completely with the point you are making that in order to have any statistical power, the null hypothesis has to be false.

ReplyDeleteI'm guessing that you are commenting on my statement: "Statistical power is the probability of finding a statistically significant result for a particular parameter in a particular study." These notes were meant to be brief and I was intending to imply by context that there is an actual effect. I'll update the text to make this clearer. Cheers.

Hi Jeromy,

ReplyDeleteThis website is great. As a honours student working on my thesis I am finding that the way you communicate all of this is so clear. Thank you very much. You're great.

Kind regards,

Dominique