American endorsement of conspiracy theories is smaller than you think

If news headlines are accurate, many Americans hold deeply troubling conspiratorial beliefs about political and other issues. For example, NBC News writes, “Study finds nearly one-in-five Americans believe QAnon conspiracy theories” because 15 percent of respondents to a survey agreed with the statement “the government, media, and financial worlds in the U.S. are controlled by a group of Satan-worshipping pedophiles who run a global child sex trafficking operation.”

In that same poll, 29 percent of respondents completely or mostly agreed with the statement “The 2020 election was stolen from Donald Trump.”

But these conclusions might be overstated. Molly Roberts and I show in our paper Acquiescence Bias Inflates Estimates of Conspiratorial Beliefs and Political Misperceptions that how you ask questions in opinion surveys can bias conclusions about political beliefs.

We find that acquiescence-response bias – the tendency for survey respondents to endorse any assertion made in a survey question regardless of content – inflates the number of survey respondents who agree with political conspiracies.

That is, if you ask respondents if they agree or disagree with the statement

The 2020 election was stolen from Donald Trump

more will claim to agree than will disagree if you ask respondents if they agree or disagree with the statement

The 2020 election was not stolen from Donald Trump.

Because questions about conspiracy theories are often framed similar to the first rather than the second, acquiesence bias inflates the number of survey respondents who say they agree with the statement.

The research

We replicated work from recent published academic articles on political beliefs in the United States and China. We fielded six new surveys covering a range of topics, samples, elicitation instruments, and types of misperceptions and conspiracies.

To measure acquiescence bias, we used a simple research design. For each question asking a bout a political belief on each survey, we fielded both the original version of the question and an alternative version of the same question that flips the meaning of the agreeable response so that in the alternative version acquiescence bias works in the opposite direction.

The findings

We found that survey respondents do not return the same rate of endorsement to the two versions of each question on the majority of questions across samples, topics, instruments, and types of belief. Acquiescence bias inflates estimates by up to 50%. We found particularly large acquiescence bias on questions about democratic norms and the transition of power following the 2020 American presidential election.

Below we present Figure 1 from our paper, which compares the rate endorsing political beliefs across 55 different questions for those asked the question where agreement indicates endorsement of the conspiracy or belief (“positive-keyed” on the x-axis) to the rate endorsing for those asked the question where agreement indicates non-endorsement (“negative-keyed” on the y-axis). All but one of the points falls below the 45 degree line, which means a larger proportion of respondents endorse the conspiracy when asked the positive- than when asked the negative-keyed version. This is acquiescence bias.

Bias particularly prevalent among ideologues

In recent years, many prominent studies and news stories have reported on troubling magnitudes of conspiracy beliefs by ideological conservatives. We found that subjects who identify as very conservative exhibit larger acquiescence bias than both those who identify as very liberal and those with less ideological identification. Our results suggest that existing conclusions in part are driven by greater acquiescence bias by survey respondents with conservative leanings rather than greater endorsement of conspiracies by conservatives.

Methods to correct for acquiescence bias

We not only show how acquiescence bias inflates survey estimates of endorsement of conspiratorial beliefs. We also present some simple statistical fixes to correct for the bias. We show that if analysts field two versions of each question, one where the acquiescent response indicates endorsement of the conspiracy and one where the acquiescent response indicates non-endorsement of the conspiracy, combining responses from the two versions of the question can cancel out acquiescence bias. This yields an estimate closer to the likely true rate of endorsement in the population.

We also propose corrections for the correlation of acquiescence bias with political ideology.

Be cautious with dire news headlines

Our study does not show that conspiratorial beliefs are absent or that democracy is not in trouble. But it does show that consumers of political surveys and political news ought to be cautious about drawing conclusions from survey responses about political beliefs. And it shows that those who design surveys should account for acquiescence bias when writing questions to measure beliefs about political issues and conspiracies.