Describing Empirical Research and Critical Analysis
Selective description of specific studies: Good literature reviews provide a concise overview of the research on a topic. They balance the need to be concise with the desire to provide deeper analysis. They go into depth on a few articles. The choice of which articles are based on specific reasons. Good reasons include that: 1) the study illustrates a common methodology in the area; and therefore understanding the methodology clarifies the validity of conclusions drawn based on the methodology; 2) the study is particularly influential; 3) the study is one of only a few related to an important issue.Illustrate a methodology: Studies that illustrate a methodology are particularly important. The limitations of a methodology can affect all studies that use the methodology. Understanding and critiquing the methodology allows the author to assess whether the theoretical claims argued by researchers using this methodology are justified and allows the reader to understand this also. Common methodological limitations in I/O psychology include: (a) use of self-report measures (limits objectivity of conclusions); use of a observational designs (limits causal inference); (b) use of university samples (limits generalisability); (c) use of a common statistical analysis technique (e.g., most people researchers using mediation or SEM do not adequately consider alternative causal interpretations); and (d) use of experiments (may lack ecological validity).
Describing an empirical study: When describing a study it is important to present only the important details. Details highly relevant in I/O psychology include: (a) sample size (will the resulting statistics have small standard errors), (b) sample type (university students, employees, etc. - will the results generalise to the target population), (c) study design (experimental, observational, longitudinal - to what extent is causal inference possible), (d) method of measurement (self-report, other report, observational, etc. - to what extent is the measure valid, reliable, measuring something real), and (e) context of research (applied, laboratory, etc. - how ecologically valid is the task) (for more info see the Social Research Methods website). Some details that are typically less relevant include the exact number of males and females in the study, racial breakdowns of the sample, and the exact scale used to measure the variable. If such details are reported, a persuasive argument is given explaining why these details may alter the results. When describing research good literature reviews provide a quick snapshot of the important elements of the research and the important findings, while at the same time not wasting precious words that could be spent on other issues. Good literature review's “don’t lose the forest for the trees”.
Critical Evaluation: Good literature reviews base claims on empirical research. Research is critically evaluated and issues such as the following are discussed: level of analysis; confounding variables; correlation or causation; mediation and moderation; generalisability; limitations of sample size; issues of measurement; implications of design (correlational, experimental); effect size and practical significance; quantitative and qualitative methods; and so on. This examination of the theoretical literature is then integrated into the conclusions reached.
Causation: Good literature reviews demonstrate an understanding of proper causal inferences. For example: It does not matter how many cross-sectional observational studies are conducted looking at the correlation between self-efficacy and performance, this does not prove that one variable causes the other. Before writing about causality consider whether X could cause Y, Y could cause X, or whether a third variables could cause both X and Y to covary. The best evidence for causal claims usually comes from experiments involving direct manipulation of the independent variable. Weaker support can come from research statistically controlling for extraneous variables and longitudinal studies. It’s also important not to trust causal claims made by many researchers, when they are conducting mediation analysis or structural equation modelling on observational data (see my discussion). Authors may argue that their data provides evidence for causal mechanisms, but at the end of the day, their research rarely allows for unambiguous causal inference. To learn about causal inference, see this post.
Causation and control: The four goals of science are sometimes said to be description, prediction, explanation and control. Understanding causation (i.e., explanation) has important implications for making recommendations for I/O psychology practice (i.e., control). For example, if self-efficacy does not causally influence performance, then attempts by organisations to manipulate self-efficacy in the aim of improving performance may be useless.
Effect Size: The research literature is filled with findings of the form, there is a relationship between x and y or there is a difference between group A and group B on some variable. Good literature reviews communicate information about effect sizes (e.g., correlations, Cohen’s d, odds ratio). This is particularly important in literature reviews concerned with the practical importance of findings. For example, in research looking at faking personality tests in selection and recruitment, reporting that (a) "student samples were able to increase their conscientiousness score on average by one standard deviation" (e.g., Cohen’s d = 1) when asked to fake is a lot clearer than reporting that "students were able to increase their conscientious score". The former reports the size of the effect, whereas the latter only reports the direction. In another example looking at the effect of different types of diversity on team performance actual correlations or Cohen’s ds from studies or meta-analyses could be reported. Effect size can also be incorporated into a style of writing which puts relationships and effects in context. Using words like "small", "medium" and "large" to describe relationships in the literature, drawing on Cohen’s recommendations for effect size, can help the reader understand the relative importance of particular relationships (see Lee Becker's notes to learn more about effect sizes).
Integration of empirical studies: Good literature reviews are grounded in the empirical literature. They highlight problems involved in drawing conclusions from empirical research . Common design limitations include: small sample sizes (inadequate power and results that are not robust); poor measures of the theoretical constructs (unreliable or invalid); unrepresentative of the domain of generalisation (e.g., a study on students applied to organisations; a study of racial diversity applied to gender diversity). Good literature reviews are critical of the ways that studies draw conclusions from empirical research.
Weaker literature reviews often treat research claims equivalently. Good literature reviews delve into the reasoning used by authors to make their claims. They critically evaluate the empirical research and develop a reasoned evaluation of the justifiability and generalisability of claims. They explore alternative interpretation of results. They identify limitations of study design.
Presenting an Overview
Good literature reviews provide an overview of the literature relevant to the topic. For most topics in psychology, there are hundred, or even thousands, of articles on the topic. Words are limited. Good literature reviews manage this issue. Some good strategies include:1) Cite meta-analyses: When the question is focused, meta-analyses provide a systematic quantitative summary of a relationship. It’s also important to know how to describe and evaluate a meta-analysis. Some major things to mention include: a) the effect size that was obtained; b) what kinds of studies were included; c) the number of studies and total number of participants; In some instances you also mention d) whether the effect size varied more than was expected from random sampling; e) results of any moderator analysis.
2) Cite review articles: Directing the reader to review articles helps provide an overview of the topic. Good literature reviews often share the conclusions reached by earlier reviews and use these as a launching pad.
3) Cite examples: Good literature reviews carefully select illustrative studies. They vary the depth of explanation given to a study. Sometimes it is sufficient to just highlight the existence of a study. For example, a literature review could include: “The relationship between x and y found in study Z has also been found in several other areas including sport (e.g., citation, citation), music (e.g, citation, citation), and physics (e.g., citation, citation)”. This style provides a review of the literature and gives the reader a sense of the breadth of work on the topic. If the reader is interested they can follow-up on one of the citations.
Synthesis: Dealing with Issues
In relation to many research questions there are findings that go for and against particular claim. For example, on the topic of the effect of diversity on team performance, there are findings suggesting positive, negative and no relationship. Good literature reviews propose plausible explanations for the variability in findings across studies based on the available evidence. Differences in findings can be explained in terms of differences in terms of study design, study conduct, and random sampling. A moderator is a factor that alters the relationship between another two variables. Moderators can be substantive (e.g., different types of participants or contexts) or methodological (e.g., different measures, software, etc.). Random sampling is also an important explanation of differences in results between studies (See my discussion of meta-analytic thinking also).
Good literature reviews weight conclusions by the quality of the evidence for alternative arguments. They recognise the value of meta-analyses in pooling multiple studies in a systematic way.
Poorer literature reviews present one theory after the next without integration. They often include text like: “despite all the research, it can not be said whether job satisfaction is caused more by situational or dispositional factors. The End…” Concluding statements, such as “more research is needed” is typical of poorer literature reviews. They also often tend to jump to conclusions too soon, often declaring a particular idea (e.g., a strong culture leads to high performance) as definitively established when debate existed. Slightly better literature reviews say things like: “This model has not been particularly supported (Smith, 2006)”. A good literature review provides evidence for why a theory has not been supported. A good literature review is aware of the relationship between empirical observations and theoretical claims.
Good literature reviews identify issues in the literature. Issues arise when two proposed ideas conflict. For example, does X cause Y or does Y cause X? Is expertise learnt or innate? Is this theory useful for practitioners or not? Is job satisfaction influenced more by dispositional or situational factors? Good literature reviews accurately and concisely summarise the major competing positions on an issue. They do not rush to conclusion, yet they do strive to reach a nuanced conclusion based on the evidence. If more research is needed, the achievements of existing research are acknowledged and concrete recommendations are made. This might be recommendation for greater use of a particular methodology or particular contexts.
Logic and Reasoning
Good literature reviews are logical. For each statement evidence is given or implied and the evidence is good and persuasive.Structure
Clear Structure: Good literature reviews have a clear structure. Aims and objectives are set out at the beginning of the literature review. The aims are consistent with the broader research needs. These needs may be derived from an assignment question, a thesis aim, or journal publishing requirements. The structure systematically works through the issues. Communicating structure both to oneself and to the reader can be facilitated by a set of headings. The structure of ideas and themes guide the literature review. Research and theory is integrated into the structure. Independent thought is reflected in the structure.
First, weaker literature reviews tend to waste words discussing material unrelated to the aims. For example, a literature review on the role of team diversity might spend many words discussing other factors besides team diversity that predict team performance. Second, weaker literature reviews are a loose listing of ideas. At its extreme the literature review appears like the author has obtained a set of articles and spent a paragraph or two discussing each with no integration. Poorer literature reviews do not demonstrate independent thought. They may mimic a particular journal article or lecture on the topic. They cite the same articles and present the same arguments as adopted in the source. At the extreme they plagiarise (i.e., copying without acknowledgement) the structure of another article.
Quality of Expression
Spelling, Grammar, Composition, Style: Good literature reviews are written well. At a basic level this involves correct spelling, grammar, and punctuation. At the next level this involves following conventions of writing literature reviews. Tense, pronoun usage, and paragraph composition are particularly important. There are many strategies for ensuring that prose reads in a structured and sequenced way (some thoughts). Finally there are many books on writing style that are worth studying from time to time (See here for a list; and here for online grammar help).
Article Deconstruction: To learn the conventions of writing in psychology, the best strategy that I know is what I call Article Deconstruction. It involves: 1) completing a writing needs analysis on yourself; 2) finding an article that demonstrates the writing style that you would like to emulate; 3) deconstructing the journal article to identify the principles that guided its creation; 4) working on your own writing task and attempting to implement the principles. A fifth step can involve reading how-to books on the particular writing task. When selecting a source article to deconstruct Annual Review of Psychology is a good place to start.
Proofreading: Authors of polished literature reviews have a system of proofreading. Simple typographic mistakes detract from the finished product. Standard strategies of proofreading include: 1) putting the literature review aside for a couple of days and returning to it to proofread; 2) reading the literature review aloud; 3) getting someone else to proofread; 4) using spell-checkers and grammar-checkers. (Here's some more proofreading tips).
References and Citations
Making the link clear: Good literature reviews clearly communicate the relationship between a statement and an accompanying citation. Is the citation a baseless assertion, an established theory, or an empirical finding? The following examples tend to be superior to ending a sentence with a mystery citation: “A study by Smith showed”, “Smith has suggested that”, “Smith has theorised”, “Smith obtained results supporting the idea that”. Good literature reviews give a sense of the strength and nature of the evidence provided by the citation. Putting an author’s name at the end of a sentence tends to not fulfil this need. Thus, the words around the citation can explain the reasons why the author asserted the idea in the first place. Did the author base the claim on common sense? Was it an empirical finding? Was it based on the summary of a set of empirical results or meta-analysis?
For example:
Poor: “Perceptual speed, psychomotor and general abilities relate to the three phases of skill acquisition in different ways (Ackerman, 1988).”
Better: “Ackerman (1988) has theorised that perceptual speed, psychomotor and general abilities relate to skill acquisition phases in different ways.”
Even Better: “Ackerman (1988) performed a series of large sample empirical studies using a range of simple psychomotor tasks which provided partial support for his theory that perceptual speed, psychomotor and general abilities relate to skill acquisition phases in different ways.”
Cited In: Good literature reviews do not use "Cited in". Literature reviews which summarise Author B’s citation of Author A’s work write: "as Author A (1999) says as cited in Author B (2002) …" . However, good literature reviews, when they see that Author B cites Author A, go and get Author A’s article, read it , and draw conclusions about it directly.
Quotes: Good literature reviews provide an independent voice on the research of others. Quotes are used sparingly and judiciously if at all. literature reviews with more than two or three quotes typically appear as if the author is unable to express an independent perspective. A simple rule is to not use quotes at all. A better rule is to use quotes sparingly and only when the particular phrasing of the original author adds value, such as in definitions or when the words have a poetic or other literary effect.
Number of References: It is important to have a sufficient number of references. The aim is generally to give an overview of the topic. The nature of the coverage and the selection procedure should be made explicit.
Independent voice: Good literature reviews carry a thread of the author’s voice. An independent voice can be facilitated by following the principles of citation mentioned above. Good literature reviews present an independent perspective on the issues. Good literature reviews answer questions like: does the theory make sense with my understanding of the world? Is my argument logically consistent? What would be the implication of what I am saying for interventions, other theories, future research, etc.?
Additional Resources:
- Meltzoff (1997, Critical Thinking About Research: Psychology and Related Fields)
- Ruscio (2005, Critical Thinking in Psychology: Separating Sense from Nonsense)
- University of Washington document on Writing a literature review
