Psychology has got some trust issues. It used to be the case that when authors wrote they had a specific hypothesis or they were interested in a specific dependent variable, you just took their word for it. However, there has been a series of high profile articles highlighting how researchers engage in questionable research practices (QRPs) such as hypothesising after the results are known (HARKing; Kerr 1998) and selectively reporting only the variables that are statistically significant (Simmons, Nelson and Simonsohn 2011). To regain some trust and reduce the impact of QRPs, many authors are now pre-registering their studies. Pre-registration is where authors state in advance what their aims and hypotheses are for the study, along with sample size calculations and which measures are going to be used. When the study is complete and written up for publication, anyone can cross-reference what they have reported in comparison to their pre-registration. Although it has supposedly been compulsory for clinical trials to be pre-registered for decades, pre-registration in psychology is relatively new and the emphasis has increased since the replicability of psychological research has been called into question (Open Science Collaboration 2015).
The focus of this post is on how you can effectively pre-register your study with a case-study of how not to do it…my own first attempt. I will point out what is wrong with my first attempt and highlight some resources that have been particularly useful in improving my own knowledge. I was inspired by Katherine Wood’s blog post on her experience of starting her graduate programme after the fall-out of the replication crisis. She identified that current graduate students have the opportunity to start their research career following an open science approach, as opposed to changing practices further into their career. I too started my graduate programme after becoming aware of the issues of replicability in psychology, but I did not know anyone else at my institution who pre-registered their research and there was not exactly a “pre-registration for dummies” to turn to. However, there are now several articles and blog posts explaining how you can pre-register your research (e.g. Research Pre-registration 101). This post is aimed at those who are yet to pre-register their research, or are perhaps in the same boat as me where you have tried and not quite got there with the first attempt. There are some resources that I have found invaluable in improving how I pre-register my research, and a suggested template that you can follow.
Note: this is about what van ‘t Veer and Giner-Sorolla (2016) call unreviewed pre-registration, where the author time stamps their pre-registration document on a platform such as the Open Science Framework (OSF). However, there are also reviewed pre-registrations or Registered Reports. van ‘t Veer and Giner-Sorolla (2016) and Dablander (2016) have previously written about the distinction so I would direct you there for a more elaborate discussion. I have also previously interviewed Dr. Hannah Hobson about her experience of publishing a Registered Report during her PhD if you are interested in the experience of publishing one. There is also a (growing) list of journals that accept Registered Reports which can be found via the Centre for Open Science.
My first attempt
The pre-registration for the first study of my PhD can be found on the OSF. I cannot remember the exact point in time, but I was not aware of pre-registration in psychological research until I was planning the first study of my PhD. I had followed the discussions on Twitter and read some eye-opening articles about why it was important. However, I had not read someone else’s pre-registration and none of my supervisory team were aware of it. I was determined to do some good science so I did what I thought at the time was a good job as I had never seen another pre-registration document.
If you have looked at my pre-registration, you will probably notice some immediate issues. I will break it down into a list of what I now see as far from ideal.
- The aim of my study was to investigate differences between three smoking sub-groups: light smokers, heavy smokers, and a non-smoking control group. A good point is that I outlined a sampling plan to collect 20 non-smokers and 40 smokers as optional stopping has been identified as hugely problematic for increasing type one errors (Simmons et al. 2011). However, I did not justify how I calculated the sample size and I did not outline the inclusion/exclusion criteria.
- I originally stated I was unclear about the hypotheses due to a combination of rushing before I started collecting data, conflicting previous research, and I was still in undergraduate mode of being terrified to stick my neck out and make actual predictions. I uploaded an updated version later on but if anything this makes it look worse, as despite being transparent on where I was with collecting data, people have to take my word on whether I had not looked at the data yet.
- There was no information on the procedure and how the data collection would be taking place.
- Although it is not always included in pre-registration, a plan of data-screening could be included to avoid researcher degrees of freedom in data-analysis.
All in all, it’s not great. However, I was still in the first six months of my PhD and I did not have anyone to learn from who had done it before. Now being in the second year of my PhD and having had the time to read around, there are several resources that have helped me to improve dramatically as I write a more thorough pre-registration for my second study.
Useful resources
Improving your Statistical Inferences MOOC
Firstly, week seven of Daniël Lakens’ MOOC on Coursera has an assignment which focuses on how to conduct a study following an open science approach. There is a useful guide for leading you through pre-registration either through the template on aspredicted.com or directly through the OSF. There is also a handy guide for how you can organise folders for each section of your project on the OSF.
Pre-registration in social psychology – a discussion and suggested template
I only came across this article by van ‘t Veer and Giner-Sorolla (2016) when I first started writing this post. My original idea was to write something like this article, but they have done a much better job than I ever would. The article can be briefly summarised as the historical antecedents of pre-registration in psychology, the benefits it can provide, some common apprehensions and limitations, and finally their suggested pre-registration template.
Other pre-registrations
One of the difficulties I encountered when I wrote my first pre-registration is that I had nothing to base it on. This will probably be less of a problem if your supervisor can show you an example of theirs, but I did not know what it was supposed to look like. Therefore, another valuable resource I learned from is having a look at the pre-registrations of other researchers or institutions. You can browse the OSF to find a project that is in a similar field to your own research. A particularly good example of how to pre-register in my field of research (psychopharmacology) is the Tobacco and Alcohol Research Group at the University of Bristol. Quite admirably, they have a research group wide policy of pre-registration everything from undergraduate dissertations to grant funded projects. Reading some of their examples helped me to improve the way in which I write my pre-registrations and know what information to include.
My suggested template for pre-registration
These are the headings and sub-headings that I have used to pre-register my current study. They are an amalgamation of all the resources I have read whilst I have been preparing my next study. There is no universally agreed upon template so you could add additional headings if there is further information that you think is important for your project.
1. Background
This is pretty much a very short introduction section. What previous research has influenced your study and what problem are you trying to address?
2. Objectives and hypotheses
2.1. Aims
The aim(s) of your study. What are you trying to achieve?
2.2. Hypotheses
What are your predictions, what do you think is going to happen?
3. Study design
A brief outline of the design of the study. What are the IVs and what are the DVs?
4. Participants and recruitment
I start off with a summary of the sample I am trying to obtain. How many groups do you have, how many participants will be in each group, and how are you going to recruit them?
4.1. Inclusion criteria
What characteristics are you interested in? What makes them eligible to take part in your research.
4.2. Exclusion criteria
What characteristics could your participant have so that it would not be a good idea for them to take part. You just need to be transparent on the criteria you are using to exclude people to reduce researcher degrees of freedom (Gelman and Loken 2013).
4.3. Sample size calculation
You need to justify your sample size to show it is well powered. This is something I did not do in my first attempt at pre-registration, and it is an important consideration as both optional stopping (Simmons et al. 2011) and low statistical power (Button et al. 2013) have been shown to be problematic.
5. Measures and materials
5.1. Measures
How are you measuring your DV? You can outline the specific task(s) you are using. Make sure you are being transparent and list all the measures to ensure you are not selectively reporting only the statistically significant ones.
5.2. Materials
Here you can outline the questionnaires that your participants have completed. Again, be transparent and list all the materials involved.
6. Procedure
How are you going to collect the data? This does not have to be extensive, but include an outline of the data collection sessions.
7. Statistical plan
7.1. Data screening procedure
When you analyse the data, how are you going to screen and clean the data? This is again to reduce the impact of researcher degrees of freedom as there are many ways of doing this. It’s important to note here that this is not a straightjacket. Unless you have a crystal ball, there may be times when something unexpected happens. You do not have to rigidly follow this, but it allows you to transparently say “this is what I planned, but because of x, y, and z I had to do this instead”.
7.2. Planned analyses
What are the planned analyses you are going to perform to answer your research question? For example, following the recommendation of Cramer et al. (2016), you can pre-register specific effects of interest in a factorial ANOVA to reduce multiplicity.
8. References
You can end on a reference list for the sources you used throughout the document.
Hopefully this post and the resources have helped if you were uncertain about how to pre-register your study. More and more journals are asking for evidence of studies being pre-registered and it is only a matter of time before it is largely compulsory for experimental research to be pre-registered. The opportunity is here to get ahead of the tide and ensure you are following open science practices. There are common concerns voiced about pre-registration with one of the most frequent being it stifles exploration. However, this is simply not true. The idea behind pre-registration is that you can show what is confirmatory research and what you have found through exploratory research. Exploration is an important creative process and the benefits can be summarised with a quote from John Tukey’s Exploratory Data Analysis (1977: page 3):
“[R]estricting one’s self to the planned analysis—failing to accompany it with exploration—loses sight of the most interesting results too frequently to be comfortable”
You just need to be open and say what you wanted to confirm and what you found through exploration, it’s all about transparency and honesty.
I am not the first to outline how you can pre-register your study, and there are other more thorough accounts such as van ‘t Veer and Giner-Sorolla (2016) which I would recommend you refer to. However, I made some mistakes during my first attempt at pre-registration. Hopefully you can learn from these mistakes if you are yet to pre-register your research or like me did not do a great job with the first attempt. I hope the resources throughout this post can be useful and if you have any further suggestions on helpful resources, please suggest them below and I can add them in.