How to fact check scientific claims
There are a lot of suspicious claims supposedly based on scientific evidence (“studies say that this pill will cure …”, “the sky is not blue according to one expert”)
In my opinion the most authoritative position of the scientific community is found in the overall consensus of the existing peer reviewed research.
But before delving deeper into the world peer reviewed research, let’s start with some fact checking basics.
When fact checking any suspicious claim, it is good to run a quick search in a fact checking tool, like Google Fact Check Explorer (https://toolbox.google.com/factcheck/explorer). This tool is useful because it requires
Google Fact Check Explorer: https://toolbox.google.com/factcheck/explorer
This tool is useful because it requires no expertise, just search for the claim and see if it has been fact checked.
The Rand Corp maintains a good list of fact checking tools at the link below, but much of those tools require some expertise that is beyond the scope of this post.
Rand Corp List of Fact Checking Tools: https://www.rand.org/research/projects/truth-decay/fighting-disinformation/search.html
Some other fact checkers to use:
- https://apnews.com/hub/ap-fact-check
- https://www.factcheck.org/
- https://www.snopes.com/
- https://www.politifact.com/
- https://www.reuters.com/fact-check
Peer Reviewed
The best step in fact checking scientific claims is to find the consensus of the existing peer reviewed research.
What does all of the research say?
Therefore when you see any questionable claims based on one study, one random “expert”, or just vaguely based on “studies”, you can avoid verifying the claim itself and instead go to the scientific community’s overarching position.
This is best explained in the following guidance from The Journalist’s Resource:
Covering scientific consensus: What to avoid and how to get it right
by Denise-Marie Ordway, The Journalist’s Resource
November 23, 2021
HOW TO GET IT RIGHT: Look to peer-reviewed research and scientific organizations for help gauging whether and how much agreement exists among scientists on a topic.
Look to these four sources for information about levels of agreement:
- Studies of scientific agreement — Academic journals occasionally publish papers analyzing existing research to establish the level of consensus on a given issue. Just last month, Environmental Research Letters published the paper, “Greater Than 99% Consensus on Human Caused Climate Change in the Peer-Reviewed Scientific Literature,” which finds “there is no significant scientific debate among experts about whether or not climate change is human-caused. This issue has been comprehensively settled, and the reality of [anthropogenic climate change] is no more in contention among scientists than is plate tectonics or evolution.”
- Surveys of subject experts— When it’s unclear whether consensus exists, scholars, health care professionals and other experts may be asked to complete a survey to share their views on an issue or research question.
A paper published in PLoS ONElast year examines the results of a series of surveys asking leading health informaticians how machine learning will influence primary care in the U.S. over the next several years. The main takeaway: The consensus is that machine learning “will engender training and primary care work force changes, improve rates of diagnostic accuracy, and increase access to primary care” within the next decade. - Consensus statements — Independent scientific organizations such as the National Academy of Sciences, American Association for the Advancement of Science and American Medical Associationissue consensus statements. These statements typically communicate either scientific consensus or a collective opinion of a convened panel of subject experts. Earlier this year, for instance, the Association of American Medical Colleges issued a consensus statement advising face masks be worn to prevent the coronavirus from spreading.
- State-of-the-Art Reviews — This type of research paper synthesizes current thinking in a field, focusing on research published within the last decade or so. A recent example: The Journal of the American College of Cardiology last month published, “Exercise Intolerance in Older Adults With Heart Failure With Preserved Ejection Fraction: JACC State-of-the-Art Review.”
Funding
Finally, who is funding the research?
See the following video and article to understand where you can find the funding for a given study
Video – https://youtube.com/watch?v=jyAjhoG7ouE&si=EnSIkaIECMiOmarE
https://media.pdx.edu/media/Locate+the+Funding+Source+of+a+Scientific+Article/1_525g4quy/29662642
Article – https://undsci.berkeley.edu/who-pays-for-science/
The main takeaway from the article is that;
- Much scientific research is funded by government grants, private companies, and non-profit organizations.
- Though funding sources may occasionally introduce bias to scientific research, science has safeguards in place to detect such biases.
- Conflicts of interest for researchers must also be listed on the article
There are several documented strategies companies use to bias scientific research:
- Fund and undertake “safe”research
- Covertly undertake or prevent “risky” industry research
- Control design & analysis of industry-funded science to ensure favourable results
- Shape and undermine external research
- Ensure favourable research is heavily represented in the evidence base
See each strategy and others laid out in detail along with real life examples in the following article:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8221522/
For now that’s it
Search-ish 