We’ve just published a commitment to Open Science as a disclaimer about how we intend to publish our work, starting today.
It has been becoming increasingly clear that scientific practices in Psychology and Neuroscience will have to undergo important changes towards “Open Science”. Maybe inspired most by the Science publication on the difficulty to replicate many Psychology findings by the Open Science Collaboration in 2015, discussions and suggestions abound in social media and on the internet.
Some of the most salient points in current discussions are:
- the need for a general commitment of the research community to communicate and share original data, analysis methods, and programming scripts
- the need for scientists to truthfully report hypotheses, null findings, and failures
- publish in ways that make papers freely accessible
We’ve discussed Open Science practices in the lab over the last months. We’ve compared our current research and publication practices with those that might be most adequate in view of good scientific conduct, and evaluated what steps we will have to take to make the necessary changes.
The result of these discussions is our Open Science Commitment, which we have made available on our webpage today, to which I personally commit as the principal investigator of the lab, and to which the members of my group have agreed to commit to as well.
Some comments and explanations
One thing that became quite clear in our discussions is that a commitment to Open Science might seem clear and easy in theory and debate, but does not as easily translate into a realistically manageable day-to-day lab practice. As a consequence, we have formulated a commitment which we think we can actually keep. Below, I’ll give a few explaining comments.
We hope that other labs will implement their own Open Science principles, and we are open to discussion about our own!
About data sharing
Some initiatives demand that everyone share their raw data. Whereas this is relatively straightforward for many behavioral data such as reaction times and response choices, it seems much less straightforward for experiments involving motion tracking, EEG, or fMRI. Such data typically contain many segments of unusable data, complex coding of experimental conditions, and are initially stored in formats that might not be readable by every analysis software. Anyone who has ever tried to analyze such a data set that they did not acquire themselves (we have!) knows that reconstructing the coding of conditions, unusable segments of data, etc. can take weeks, even if the data are documented.
We have therefore specified to share the data necessary to replicate the analyses we report. As an example, for an EEG experiment, this might be the data of the trials we retained after semi-automatic artifact rejection.
In addition, we realized that preparing scripts for sharing can be a daunting task, especially for analysis-intensive experiments such as those involving EEG and fMRI. We’re not professional coders and handling code is a late-acquired skill for almost all of us. There’s a simple fear that others will criticize our code as inefficient, amateur-like, or even wrong.
Accordingly, the lab statement talks not only about sharing, but also commits to teaching everyone in the lab how to code cleanly, comment code, etc. We also provide written workflows that specify how a study should be conducted, so that making it open will hopefully require as little effort as possible. Ultimately, Open Science will hopefully be a part of our culture and simply “the way we do things here”.
We discussed whether future collaborations should be contingent on a commitment to Open Science also by the collaborator, or else be abandoned. We reckon that this might not be a smart move at the moment, as we don’t know what kind of principles and rules other researchers might have to comply with at their institutions. Accordingly, we have committed to actively discuss our principles at the beginning of new collaborations.
This was a tough point. In the current research world, publishing in expensive for-profit journals is usually important to advance a science career (most evidently so for untenured researchers, but also for tenured ones). To put it bluntly, everyone talks positively about the new Open Access (OA) journals, but the best research is mostly still sent to the old, closed access (or expensive OA fee) journals.
Therefore, committing to publishing exclusively Open Access seemed unrealistic in the current career climate. There are some OA journals we do not want to use as output for our research; and we object to paying the excessive OA fees many closed access journals take for making papers OA. So we will use preprint servers to publish our research OA, even if some of our publications will be submitted to closed access journals.
About peer review
There is currently a lot of chatter about the Peer Reviewer Openness Initiative. Signers of this initiative vow to review papers only if the corresponding data and scripts are publicly available, starting 2017. Though I find the aim of the initiative great, I have not signed, as I cannot justify asking of others what I have not fully implemented in my own lab. Accordingly, I will sign the initiative when we’ve fully transitioned to being an Open Science lab.
About getting better…
As is evident from the above comments, we could be more extreme in some of the stances we take. Instead, we chose to start with what we believe we can currently fulfill. But as the last point of our commitment, we have added that we will continue to strive to become better. Step by step.
Comments to this post are open — we’re curious to hear your opinions.