Climate Science Glossary

Term Lookup

Enter a term in the search box to find its definition.

Settings

Use the controls in the far right panel to increase or decrease the number of terms automatically displayed (or to completely turn that feature off).

Term Lookup

Settings


All IPCC definitions taken from Climate Change 2007: The Physical Science Basis. Working Group I Contribution to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, Annex I, Glossary, pp. 941-954. Cambridge University Press.

Home Arguments Software Resources Comments The Consensus Project Translations About Support

Twitter Facebook YouTube Pinterest MeWe

RSS Posts RSS Comments Email Subscribe


Climate's changed before
It's the sun
It's not bad
There is no consensus
It's cooling
Models are unreliable
Temp record is unreliable
Animals and plants can adapt
It hasn't warmed since 1998
Antarctica is gaining ice
View All Arguments...



Username
Password
New? Register here
Forgot your password?

Latest Posts

Archives

Thinking is Power: How to do your own research

Posted on 15 September 2021 by Guest Author, BaerbelW

TiP-LogoThis is a re-post from the Thinking is Power website maintained by Melanie Trecek-King where she regularly writes about many aspects of critical thinking in an effort to provide accessible and engaging critical thinking information to the general public. Note: This article is the second of a two-part series on “doing your own research.” To read the first article click here.

It seems nearly everyone is “doing their own research” these days. And to some extent it’s understandable: we want to make good decisions and there’s a seemingly endless amount of information available at our fingertips.

Unfortunately, access to information simply isn’t enough. Although it’s difficult to admit, we aren’t as knowledgeable or as unbiased as we’d like to think we are. We often resort to “doing our own research” when we want (or don’t want) something to be true…and so we set out to find “evidence” to make our case. Due to an unfortunate mixture of motivated reasoning and confirmation bias, we end up wildly misled yet even more confident we’re right.

Needless to say, this isn’t how real research works. What you’re actually doing is looking for the results of someone else’s research. The real question is, how do you decide which source to trust?

header

In a previous article I tried to point out the dangers of doing your own research, which was essentially a plea for intellectual humility and trusting experts. But if you want to “do your own research,” you basically have two options: Find and trust the expert consensus or become an expert yourself and do your own (real) research.

What is a consensus? And why is it trustworthy?

Scientific research these days is highly specialized, with each subfield having its own nomenclature, methodologies, statistical analyses, etc., required to represent and communicate its nuance and complexity. Experts have the knowledge, skills, and experience necessary to evaluate the quality of evidence provided by any particular study, and importantly, to put it into context with the larger body of literature. Scientists researching heart disease are not trained to understand the literature on black holes, for instance, or even eye diseases. (Note: This goes for you, too. The scientific literature is where specialized scientists talk to each other. It’s not for non-experts.)

Contrary to how science is often taught in school, there is no single scientific method. Instead, science is a community of experts using diverse methods to gather evidence and scrutinize claims. The social aspect of science is a major reason it’s so reliable, as there’s a systematic way to correct for the biases, errors, and even fraud of individual scientists.

At the heart of the process of science is peer-review, in which research must pass the critical scrutiny of other experts before it’s published in a scientific journal. However, a single study is never the final answer: conclusions must be replicated and fit in with the larger body of evidence before scientists will accept them.

Publication

The peer-reviewed literature is where experts share their research with other experts and can be very difficult for non-experts to read. Conversely, the popular media often sensationalizes findings when they’re “translated” for the general public.
Sources: Steinberg, et al (2017); Peter Hess, Inverse

When independent and diverse lines of evidence converge onto a conclusion, the conclusion is considered strong, and experts generally accept it. The result is an expert consensus, or the collective position of experts based on an evaluation of the body of evidence. It’s not the final word, but the starting point upon which the vast majority of experts agree. Scientists can then build upon the foundation of knowledge that was gained through the process of science to learn more about what they don’t know.

The consensus is likely incomplete and it’s open to challenge, but it’s very unlikely to be completely overturned. If there is a problem with the consensus, you’re not going to find it with your “research”…it’s going to be an expert who does. And because the incentive structure in science rewards scientists who discover new things or prove established knowledge wrong, overturning the consensus would be a career-winning strategy, the kind of which Nobel prizes are made. The likelihood of thousands of scientists giving up fame and fortune for the sake of maintaining a conspiracy, as suggested by many denialists and pseudoscience promoters, is next to none.

Most people generally trust experts and use the consensus as a short-cut for decision-making. For example, if five electricians told me the wiring in my house was in danger of starting a fire, I would almost certainly get it fixed! Industry denial campaigns, such as those by tobacco and the fossil fuel companies, know full well that the public trusts the expert consensus, which is why their strategy includes telling the public there isn’t one, using bulk fake experts to counter a real consensus, or even suggesting consensus isn’t a part of the process of science.

One reason for the confusion around the importance of consensus in science is that the word has different meanings. For many of us, it’s a popular opinion or general agreement. But expert consensus isn’t the result of group-think, and it isn’t democratic. It’s the result of highly specialized experts independently evaluating the body of evidence and arriving at a similar conclusion. No other system of acquiring knowledge is as reliable or trustworthy, and it’s due largely to the community of experts checking each other’s work.

Finally, it’s not an appeal to authority to accept the consensus of experts. It’s the prudent thing to do! What is fallacious is appealing to those who aren’t experts, are experts in another area, or who represent a minority opinion to support a claim. If you value expert opinion, the consensus position should matter more than a cherry picked “expert.”

In short, the expert consensus is the most reliable form of knowledge for non-experts. Learn to trust them…they know more than you.

NaomiOreskes JohnCook

How to find the consensus

Sometimes a consensus is measured by gauging expert opinion while other times it’s by evaluating evidence. It can take significant time and research for experts to reach a consensus, and some issues have more agreement than others. That said, some topics that are controversial from the public’s standpoint (e.g. evolution, climate change, safety of vaccines) are about as settled as science can get.

Finding the consensus, if there is one, can be challenging, but it’s still orders of magnitude easier than doing all the research yourself.

Research syntheses combine the results from studies to assess the weight of the evidence. Systematic reviews synthesize the literature and condense what’s known on a given question while meta-analyses are systematic reviews that use statistical methods to summarize the results.

Remember that individual studies aren’t the final word. If individual studies are like pieces of a puzzle, these papers help to put the puzzle together. Essentially, systematic reviews and meta-analyses help us avoid the trap of potentially being misled by a single study.

To find research syntheses, search for your keywords and the words “systematic review” or “meta-analysis.” Be sure to check the quality of the journal, and keep in mind that syntheses are only as good as the research that goes into them.

Synthesis reports are syntheses of syntheses. (Yes, it’s very meta.) Synthesis reports aren’t always available, but if they are, they provide excellent evidence of a consensus.

The best examples of synthesis reports come from climate change research. For example, Cook et al (2016) synthesized the consensus estimates from six independent studies and found a robust scientific consensus that humans are causing the climate to change.

CoCSeven independent studies into the consensus of human-caused global warming all show a strong consensus.
(Cook et al. 2016). Illustration: John Cook.

Another example from climate change research is the Intergovernmental Panel on Climate Change (IPCC). Every few years since 1988, the IPCC selects top scientists from around the world to evaluate and synthesize thousands of climate-related studies to provide the most reliable and comprehensive assessments about the causes, impacts, and future risks of a changing climate. (Their most recent assessment states, “It is unequivocal that human influence has warmed the atmosphere, ocean and land.”)

To find a synthesis report, search for your keywords and the phrase “synthesis report” or “research synthesis.”

Position statements are official points of view on scientific issues by reputable scientific organizations and/or professional societies. These reports are generally based on reviews of published literature and are typically the most common expression of a scientific consensus.

To find position statements, search for your keywords along with a relevant scientific body. If you’re unfamiliar with the scientific organizations regarding a particular issue this may take a bit of searching. (However, it’s also a good indicator that searching for the expert consensus is a better choice than trying to read the literature.) Be wary of front or astroturf organizations that create unreliable societies to promote pseudoscience or science denial.

Examples of authoritative governmental scientific organizations, non-profit organizations, or professional societies include:

  • Center for Disease Control and Prevention (CDC)
  • National Institutes of Health (NIH)
  • National Oceanic and Atmospheric Administration (NOAA)
  • National Science Foundation (NSF)
  • American Association for the Advancement of Science (AAAS)
  • American Medical Association (AMA)
  • National Academy of Sciences (NAS), especially the Based on Science section

 A very reliable indicator of consensus is if the majority of prestigious organizations have arrived at a similar conclusion.

Clinical practice guidelines are recommendations produced by various healthcare organizations and medical professional bodies to help clinicians diagnose and treat patients using the best available evidence. These statements are based on systematic reviews and generally the consensus positions of experts in their relevant medical fields.

[Finding clinical practice guidelines.]

Polls and surveys measure scientists’ opinions on a topic. While these provide useful information, especially if the results are in line with other measures of the consensus, they aren’t as reliable as literature reviews.

As always, it’s important to use credible polling organizations. In addition, pay attention to who was surveyed. Was it all scientists? Top scientists, such as those in the AAAS? Were they experts in their fields?

ExpertiseThe higher the expertise in climate science the stronger the consensus on human-caused climate change. Illustration: John Cook.

A few more tips for your search

We’re more likely to fall for misinformation when it confirms what we already believe. If you’re looking for evidence that you’re right, you will absolutely be able to find it online! So instead of asking Google a leading (or loaded) question, use neutral search terms. Or better yet, break out of confirmation bias by trying to find evidence that you’re wrong.

It’s essential to check the reliability of sources. While peer-reviewed journals are the most reliable source of scientific information, not all are created equal. One way to check the quality of a journal is to search for its Journal Impact Factor (JIF), which calculates its influence in the scientific community based on the number of times its articles are cited.

Unfortunately, the more dangerous forms of deceit online involve setting up fake science journals or organizations to promote pseudoscience or denial, and it can be easy to fall for “studies” that appear to be scientific but are of significantly lower quality. In particular, be wary of predatory journals that publish false or misleading information and lack quality publication practices and pseudoscience journals that pretend to be scientific but are created for the sole purpose of publishing fake science.

If you’re using sources of information that aren’t peer-reviewed, it’s even more important to check its reliability. Don’t waste time on an unknown site’s “About” section, instead look laterally to see what other reliable sources have to say about that.

[Learn more: Don’t be fooled…fact check!]

The take-home message

While many of us like to think of ourselves as independent thinkers, most of our beliefs come down to the sources we choose to trust. Unfortunately some have decided that “thinking for themselves” means automatically not trusting information from the government, the media, or even academia, leaving them vulnerable to misinformation that reinforces their existing beliefs or biases.

We trust experts all the time, to grow our food, to fly our planes, to wire our homes, etc. It’s only when expert opinion conflicts with our core beliefs that we resort to denial…and to “doing our own research” to find the “real truth.” Instead of cherry picking studies or experts to try and support your cherished beliefs, be honest with yourself about what you believe and why…and why you trust the sources you do.

But you can’t out-research experts. You have to be an expert to do that. If your “research” leads you to sources that tell you the experts are all mistaken or lying, you’re doing it wrong. You haven’t found trustworthy sources the experts have all missed. You’ve been misled by propaganda that appealed to your emotions or biases.

The bottom line is: If you want to believe in things that are “true,” to the best of human capabilities to know “the truth,” learn to trust experts, because they have  the knowledge and skills needed to evaluate evidence.

However, if you’re really convinced nearly all of the world’s experts are wrong, become an expert and do your own (real) research.

To learn more

Special thanks to Lynnie Bruce, Jonathan Stea, and John Cook for their feedback

0 1

Printable Version  |  Link to this page

Comments

Comments 1 to 3:

  1. I have been enjoying reading these posts. In this one, the phrase that struck me was "What you’re actually doing is looking for the results of someone else’s research."

    This is a really key factor in terms of what people think "research" is. In common usage, someone might say "I am going to the library to research this topic" (or, today, "I am going to Google to research this topic")1. This is not really "research" in terms of creating original work, though.

    In scientific terms "research" usually involves either the collection of new data, or at least a novel analysis or combination of existing data. "Research" requires that somone provide innovative thinking, analysis, and interpretation - supported by evidence, of course.

    An important stage in scientific research is to go to the library and learn what others have already learned before you ("Learn from the mistakes of others - you won't live long enough to make them all yourself"). But then you need to be able to say the following:

    • What questions remain unanswered?
    • What sort of data or evidence is needed to answer that question?
    • Can I find existing data that fits my needs?
    • How would I go about collecting new data to answer my question?
    • What does this data tell me?
    • Do I now know more about my question?

    Doing this well requires knowledge and skill.

    1. [Someone over at RC once said something along the lines of "between a library and Google, one of them wants to make you smarter, while the other wants to sell  you stuff. Choose carefully."]

    2 0
  2. Thank you for the kind and thoughtful comment.

    The thing I keep coming back to is: who do you choose to trust and why? Often it seems like those who are "doing their research" are doing so because they distrust - and therefore don't accept the conclusions of - scientists, etc. Therefore they're going to google, typing in what they want (or don't want) to believe, and going to the sources that tell them they're right. Unfortunately, those sources haven't used reliable and trustworthy methods of gaining knowledge...they have biases and existing beliefs they're attempting to justify.

    We all use trust as a heuristic for what to believe, but when an issue becomes politicized and/or part of people's identity, trusting the wrong sources can lead us astray.

    0 0
  3. Yes: who you trust is the key. In science, we often stress peer-reviewed literature, but very few people understand the review process and how it works (or not). For climate change, the anti-science crowd usually claims that the peer review system is corrupt, even though it is more often the "contrarians" that abuse it.

    Skeptical Science has a couple of rebuttals on those subjects:

    https://skepticalscience.com/Peer-review-process.htm

    https://skepticalscience.com/pal-review.htm

    and we are trying to prepare more.

    The person that cries "they are lying to you" the loudest is often the one that is doing the most lying.

     

    0 0

You need to be logged in to post a comment. Login via the left margin or if you're new, register here.



The Consensus Project Website

THE ESCALATOR

(free to republish)


© Copyright 2021 John Cook
Home | Links | Translations | About Us | Privacy | Contact Us