After the noisy 2020 election season in the United States, journalists wrote a lot about the inaccuracy of pre-election polls. They weren’t the only ones. According to a report by the American Association for Public Opinion Research titled 2020 Pre-Election Poll: An Assessment of the Polls for the 2020 General Election, the 2020 polls were the widest in decades at the federal and state levels. For example, a CNN poll predicted that Joe Biden would edge Donald Trump by 12 percentage points. Biden won by 4.5 points.
The report’s writers suggested a few possible reasons why the polls were so off that year. Among them:
- Trump has called many polls fake, likely discouraging his supporters from responding. Pollsters may not have sufficiently overweighted a pro-Trump response over a pro-Biden response, giving Biden a higher apparent lead.
- Democrats who responded to polls may be more supportive of Biden than those who did not. Likewise, the Republicans who responded might have been less supportive of Trump.
The report noted that polls not only overstated Biden’s support; polls have also overestimated Democratic support in the Senate. This could indicate a “shy Republican” phenomenon, which could have two causes. It could be that some Republicans don’t trust the pollsters, as Nathaniel Rakich of FiveThirtyEight told me. This lack of confidence could be partly corrected if the pollsters came from all political backgrounds. But it could also be that the people taking part are unwilling to express their true preferences to an outsider, even about an overall Democratic/Republican choice.
With the 2022 election year underway, one way to counter the problem of “shyness” is for pollsters to give the people they poll plausible deniability. In other words, a person must be able to answer a question honestly, while preventing the interviewer from knowing whether the answer is that person’s real opinion. This could be particularly useful in countries with autocratic rulers who might take revenge on dissenters.
The question is how do you achieve denial without sacrificing accuracy? Surprisingly, tossing a coin multiple times might help. Here I borrow ideas from the work of Stanley Warner in 1965 and differential privacy. Such methods have been used for surveys.
Here is an example.
A fictional country has an upcoming election in which the Tin Man is running against the Scarecrow. Shy voters won’t admit they support the Scarecrow even though many secretly like him.
So the interviewers say to each participant, “Please don’t answer yet. Instead, move to a place where you are all alone and flip a coin. If it’s heads, come back and tell me your real preference. If it’s tails, please tell me Scarecrow, whatever your true preference is. This is essentially the scheme that Warner proposed for surveys that could ask awkward questions such as whether a respondent had, for example, evaded taxes, slept with a prostitute, etc.
Let’s say two hundred people complete the survey: 140 choose Scarecrow and 60 choose Tin Man. Given what pollsters told respondents, about 100 people will answer “Scarecrow” regardless of their preference, simply because of the 50-50 chance of a draw; they flipped the tail. Of the other 100 who nodded, 40 prefer Scarecrow and 60 prefer Tin Man, so Tin Man is the favorite.
What is the advantage of confidentiality? If a person declares a preference for the scarecrow, it could be the result of a simple toss of a coin. Also note that the coin is just a way to introduce chance. Another solution would be to ask a participant to think of their best friend and answer “Scarecrow” if that best friend’s age was odd and honestly if they were even.
A few weeks later, a scandal hurts Tin Man’s reputation so badly that in some neighborhoods it’s dangerous for people to say they prefer Tin Man. So now the question is whether it is possible to conduct a new poll in such a way that any answer given by a participant enjoys plausible denial.
Suppose the pollster gives the following instructions to people answering a new poll: “Please don’t answer yet. Instead, move to a place where you are all alone and flip a coin twice. If it’s heads twice, say “Tin Man” to me. If tails both times, say “Scarecrow” to me. If you get one of each, please tell me your real preference.
There are 200 pollea. They total 122 for Tin Man and 78 for Scarecrow. But out of the 122 for Tin Man, 50 questioned for Tin Man because of the coin flips (two heads). Similarly, 50 questioned for the scarecrow because of coin flips (two sides). So the actual poll result is 72 for Tin Man and 28 for Scarecrow. Tin Man is even more the favorite.
What did this puzzle teach us? On the one hand, people in polls can enjoy plausible deniability on all sides of an issue (for both candidates in our example). On the other hand, survey results will be accurate if participants follow the rules, although more people may need to be surveyed to achieve the same statistical strengths. What remains to be seen is whether people who might take the polls will even respond to a pollster and, if they do, respond honestly when coin tossing suggests they should. The primary season has started; it’s worth trying.