25 Identifying Limitations and Structure of the Abstract
25.1 Identifying limitations
Last week, we outlined the structure of the discussion. A key component of the discussion is outlining limitations which are features of your study that may threaten the validity of your findings or how these features affect how much confidence you have in your findings/conclusions. It is not about tearing your study apart or the reader will just think "why did you bother in the first place", but if you can own your limitations, it shows you have carefully considered what conclusions you can and cannot make, rather than overselling your findings.
It takes time and experience to build an understanding of what limitations affect a given area of study or methods, so we wanted to spend extra time on identifying limitations in addition to the chapter on the discussion section.
Before we start outlining common types of limitations, it is important for the future when you design your own study that you should consider limitations from the design phase. Once you have conducted the study, there is nothing you can do about them. However, no individual study provides the definitive answer to a research question. You must make decisions and often you must make compromises when designing a study. So, try and avoid as many limitations as you can in the design phase, and for the limitations you cannot avoid, make sure you can justify why you accepted the limitation.
25.1.1 The four validities
Limitations typically focus on your methods as they are elements you no longer have any control over. Clarke and colleagues have released some excellent articles on limitations in psychology articles and reinforce the idea of the four validities. These are not exhaustive and there are some limitations which cannot be classified this way, but we think it is a good way to develop an appreciation of what limitations are and provide inspiration for what features you might identify as limitations.
We covered validity in lecture 2 alongside research design and reliability. Validity relates to the truth of an inference (Clarke et al., 2023a), meaning how confident you can be in the conclusions you made from a given finding. The four validities relate to different elements that affect that confidence:
External validity - how generalisable you think a finding is.
Construct validity - how well a measure or manipulation represents a given psychological construct.
Internal validity - whether there is evidence of a causal relationship between your variables.
Statistical conclusion validity - whether your statistical approach can provide valid inferential statistics.
Before we outline the kind of features that you could question as a limitation, Clarke et al. (2023b) provide guidance on how to approach identifying limitations:
Focus on your most important limitations - avoid the temptation to list every limitation you can think of. It is not about undermining your study or distracting the reader, but highlighting one or two in detail and do them justice.
Be specific - avoid generic limitations that could be applied to almost any study. It is important to contextualise the limitation you identified and explain how it affects what you can conclude from your specific study.
Explain, don't excuse - You do not need to spin your limitations as a strength (think of a job interview where someone says "I'm too much of a perfectionist"), the critical reader is interested in how you can explain the implications of your limitations for the conclusions you make.
If you have time, we recommend reading the article from Clarke et al. (2023b) in full as they provide advice on better practice in outlining limitations, but for this chapter, we will summarise their key categories.
25.1.1.1 External validity
Selection bias - Did you use a representative sample of your target population in your study? Does your sample not include traditionally underrepresented or understudied populations?
Limited generalisability across cultures - Can you apply the findings from the cultural context of your study to another cultural context?
Situation effects - Would you expect the findings to be consistent across new contexts or time periods?
Low realism - Could your sample setting and choice of measure generalise from your study to your target setting and construct?
Theoretical generalisability - Does the theory behind your study apply universally to all humans or would it only apply locally to a specific population?
25.1.1.2 Construct validity
Construct definition errors and ambiguities - Did you clearly define your psychological construct and choose a measure accordingly?
Insufficient validity evidence - Is there sufficient validity evidence for your measures and/or manipulations?
Using measures/manipulations with little or no validation - If you developed a measure/manipulation specifically for your study, is there evidence it is valid and reliable?
Short measures - If a measure is too short, is this a limitation or is there sufficient validation evidence?
Over-reliance on self-reports - If you measure your construct with a self-report method, is this the best way to measure your construct?
25.1.1.3 Internal validity
Observational study - If you did not conduct an experiment or provide specific inferential evidence for a causal relationship, how could your inferences be biased by potential confounding variables?
Reverse causation - If you did not provide specific inferential evidence for a causal relationship, how plausible would each direction of the relationship be?
Unobserved confounding - If you did not provide specific inferential evidence for a causal relationship, what plausible additional confounding variables could there be?
25.1.1.4 Statistical conclusion validity
Exploratory research - Are you making strong conclusions based on unplanned analyses with an unknown false positive error rate?
Underpowered research - Did you use a sufficient number of participants to detect your smallest effect size of interest?
Weak statistical evidence - Do your main conclusions rely on weak statistical evidence, such as a p-value very close to your alpha?
Small effect sizes - What is the practical significance of your effect size estimate?
25.1.2 Summary
Hopefully, dedicating extra space to identifying limitations provides inspiration for what features of your study could affect confidence in your findings. Remember this is not about intentionally undermining your study, but instilling trustworthiness by realistically adding caveats for what you can and cannot conclude in your study. The four validities popularised by the work of Clarke et al. are meant to provide a starting point and help calibrate your understanding of what a legitimate limitation is. If you can identify a limitation applicable to your study that is not covered here, you can still outline it, but it is important you are specific and support your evaluation with evidence wherever possible.
25.2 Structure of the abstract
The abstract is a summary of the entire report. It comes right at the very start and is the first thing people will read. That said, it is normally the last thing that people write as it summarises all the different sections of the report you will have already written.
The abstract highlights the key point or points from each section of the report: the introduction, the methods, the results, and the discussion. In other words, Gernsbacher (2018) describes it as “A clear abstract states the study’s primary hypothesis; its major methodology, including its sample size and sampled population; its main findings, along with their summary statistics; and its key implications". Koopman (1997, cited in Derntl, 2014) suggests the following structure:
Motivation (General Area): Why do we care about the problem and the results?
Problem (Aim of Paper - Research Question): What problem is the paper trying to solve and what is the scope of the work?
Solution (Methods): What was done to solve the problem?
Results: What is the answer to the problem?
Implications: What implications does the answer imply?
Ideally, you are looking for one or two sentences from each of the different sections. You do not normally just copy and paste a sentence from each part, rather summarising the key information from each part. Remember this is a rough guide for how much detail you need. It would not be a problem if you need three sentences to outline a key component, but you probably do not need six sentences for each. We will now elaborate on those key areas.
Start off with the background and rationale
- Here you are looking for one or two sentences giving the brief background of the study for what we know, then the main problem you wanted to address or issues in the current field - basically your rationale for the current study. This provides an insight into the research question for your study.
The method
- This is not a highly detailed summary, rather an overview of your approach, stating things like the number of participants, who were they, and what they did.
The Results
- State the main findings and what tests you used. We tend not to outline all the numbers in an abstract, but it can be useful to report key effect sizes. Think of it like a short version of how you would start the discussion section.
Brief interpretation or implications of the study
You are looking to cover the main outcome of your study, trying to relate that back to theory or the main issue you wanted to address - your rationale. See it as “closing the loop” to see if you addressed your research question.
Try and avoid ending on limitations as these will be covered in your report. The main point of this final component is to briefly relate your findings to previous research and theory.
It is also a waste of words to end on something uninformative like "the implications of the results will be discussed". We would consider this a given, so try and offer the reader something concrete.
For some final formatting advice, it is normally written as one paragraph approximately 200-250 words long. For an APA style report:
It will appear on the first page after your title page.
We recommend the single paragraph approach, but some journals request a "structured" abstract where the key components are broken down into little headers. For this course, we would like you to start learning the key components and write it as a single paragraph, but we want to make you aware there are differences in style.
It should have the title at the top of the abstract. The title does not count in the 200-250 words.
You do not need to add keywords at the end of the abstract. You will see these included in some journal articles, but you do not need them for a student report.
25.2.1 Breaking down an example abstract
Looking at examples can help show how the ideas build up. Here is an adapted example from Tsantani et al (2016) broken down to highlight the key components.
25.2.1.1 Background and rationale
Vocal pitch has been found to influence judgments of perceived trustworthiness and dominance from a novel voice. However, the majority of findings arise from using only male voices and it is unclear how the pitch of female voices affects judgements of trust and dominance. Here we explore the influence of average vocal pitch on first-impression judgments of perceived trustworthiness and dominance and hypothesise that there will be a preference for low-pitched voices in both genders regardless of judgement.
25.2.1.2 Methods
Vocal pitch has been found to influence judgments of perceived trustworthiness and dominance from a novel voice. However, the majority of findings arise from using only male voices and it is unclear how the pitch of female voices affects judgements of trust and dominance. Here we explore the influence of average vocal pitch on first-impression judgments of perceived trustworthiness and dominance and hypothesise that there will be a preference for low-pitched voices in both genders regardless of judgement.40 pairs of high- and low-pitched temporally reversed recordings of male and female vocal utterances were presented in a two-alternative forced-choice task to 40 participants.
25.2.1.3 Results
Vocal pitch has been found to influence judgments of perceived trustworthiness and dominance from a novel voice. However, the majority of findings arise from using only male voices and it is unclear how the pitch of female voices affects judgements of trust and dominance. Here we explore the influence of average vocal pitch on first-impression judgments of perceived trustworthiness and dominance and hypothesise that there will be a preference for low-pitched voices in both genders regardless of judgement.40 pairs of high- and low-pitched temporally reversed recordings of male and female vocal utterances were presented in a two-alternative forced-choice task to 40 participants. Through a series of t-tests and Wilcoxon signed-rank tests, we found a tendency for participants to select the low-pitched voice over the high-pitched voice as more trustworthy, for both genders, and more dominant, for male voices only.
25.2.1.4 Implications
Vocal pitch has been found to influence judgments of perceived trustworthiness and dominance from a novel voice. However, the majority of findings arise from using only male voices and it is unclear how the pitch of female voices affects judgements of trust and dominance. Here we explore the influence of average vocal pitch on first-impression judgments of perceived trustworthiness and dominance and hypothesise that there will be a preference for low-pitched voices in both genders regardless of judgement. 40 pairs of high- and low-pitched temporally reversed recordings of male and female vocal utterances were presented in a two-alternative forced-choice task to 40 participants. Through a series of t-tests and Wilcoxon signed-rank tests, we found a tendency for participants to select the low-pitched voice over the high-pitched voice as more trustworthy, for both genders, and more dominant, for male voices only. We propose that an overall preference for low pitch is a default prior in male voices irrespective of context, whereas pitch preferences in female voices are more context- and situation-dependent. The present study confirms the important role of vocal pitch in the formation of first-impression personality judgments and advances understanding of the impact of context on pitch preferences across genders.
25.2.2 Conclusion
This is our advice to learn the key components to include in an abstract. However, you will see a lot of variation in how people approach writing abstracts and standards have changed over time. For example, it was rare to include the sample size of your study, but this is now normally considered essential to include in the abstract. So, while we recommend looking over abstracts in articles from your literature review to develop a sense of style, try and do so critically as they might not be great examples.