Reproducibility Crisis and You
There is a somewhat well-known problem in science called the Reproducibility Problem. All seasoned scientists know about it, and even some of the public knows about it. The difference is the public doesn’t know what it means, and scientists aren’t really talking about what it means about research.
What it is
Reproducibility is at the core of science. Before anything can be outrightly claimed as “fact” it must be reproduced. This could be done by the same lab or a different lab. The experiment is reproduced with the same methods and verified if the results are within an acceptable range.
What is an acceptable range? Good question. It depends on the field, but the rule of thumb I was taught is to aim for within two standard deviations (one is better) or an R2 as close to 1 as possible. There may be other standards out there, but these were the two I was taught from archaeology (standard deviations) and various physical sciences (R2).
What does all that mean?
I’m going to take a more generalised track here. The area of reproducibility in the sciences is nuanced and subject to much debate. But, at the heart of the debate is a simple idea: If you do an experiment, it needs to be doubled-checked with the methods and supplies (such as reagents) you used. The end result should be very similar to yours.
Ideally, this double-check can be done off a paper alone, without need to be in the same lab with the access to your brain. That means you should have a well written paper with descriptions of how the science was done, giving the next researcher the ability to reproduce your work as closely as possible.
This sounds really easy, right?
Science is inherently messy, but that message is lost in the PR showing how shiny and useful science can be when it goes right. There is a joke among organic chemists and molecular biology about sacrificing goats to make everything go right. Technique can get you pretty far, but there is a lot that can go awry.
We don’t tell people who in science because we’re trying to simplify the message, let alone explain why the linear progression only worked after many hours in the lab and just as many sitting in front of the data trying to make it make sense.
This applies to all research.
My dear field of geochemistry is just one area where there is a fight to make sense of the world through careful lab work and data collection. Although psychology has gotten it’s share of scrutiny, it’s all areas of research from the social sciences to physical science.
Scientists are aware of this because reproducibility is an integral part of science. And, frankly, we’re always looking for ways to make things cheaper, easier, and quicker. We like the lab, but we don’t want to sacrifice our lives to it. Each field is working out their own methods that work for our fields, such as using the Research Resource Identifier a portal for biological sciences to share information about resources.
How it affects policy making
First, we need to stop trusting journal articles blindly, and we should also look for many collaborating studies across the board. Low end journals just publish things without review or testing. High end journals, such as Science, are dedicated to “ground-breaking” work which generally hasn’t been reproduced. Mid-range journals, like Organic Syntheses, tend to be better about making sure that the data can be reproduced. Taking this into account, we’ll get a more full view of the actual research underlying any question.
When it comes to making a policy, it should be slow going in order to gather and review the most accurate data to make well-informed decisions. Sadly, the average constituent is pretty impatient for something to happen, and they also get distracted by big, shiny names on the journal. A strong communication is needed between scientists, the general public, lobbies, and policy makers to keep communication on the status of ideas open. On top of that, the conversation about reproducibility needs to be out there, and people should not be afraid to demand results before making policies.