Pollsters have conducted over 44,000 interviews among Iowa’s 160,000 Republicans, but they probably interviewed just 15,000 unique people. A majority of those polled took part in at least three interviews over the course of the campaign.
It seems like new Presidential polling figures are released every day. We generally talk about each new poll as a unique snapshot of the campaign with some fresh sample of a few hundred caucus-goers. That concept of polls might apply to national samples, but when polling in states as small as Iowa and New Hampshire, the number of eligible respondents is not that large compared to the number of interviews being conducted.
Making the problem worse is the falling “response rate” in polling, the proportion of eligible respondents who complete an interview. Mobile phones, caller-ID, answering machines, all have enabled potential respondents to avoid the pollster’s call. Pew reports that response rates have fallen from 21 percent to 9 percent just between 2006 and 2012. If we assume a response rate of ten percent, only some 16,000 of Iowa’s 160,000 eligible Republican caucus-goers might have agreed to take part in a poll.
Huffington Post Pollster lists a total of 94 polls of Republican caucus-goers through today, January 31, 2016, constituting a total of 44,433 interviews. As I have done before for New Hampshire, I will use this figure to see how the composition of the sample changes with different response rates.1
How large is the electorate being sampled?
Around 120,000 people participated in the Republican caucuses in 2008 and 2012. While some observers think turnout in 2016 could be higher because of the influx of voters for Donald Trump, I have stuck with the historical trend and estimate Republican turnout in 2016 at just under 124,000 citizens.
To that baseline we have to add in people who agree to complete an interview but do not actually turn out for the caucuses. In my 2012 analysis I added 20 percent to the estimated universe to account for these people, but recent findings from Pew suggest 30 percent inflation might be more accurate. With rounding, I will thus use 160,000 as my estimate for the number of Iowa Republicans who might have been eligible to be polled about the 2016 Republican caucuses.
How low is the response rate?
Most of those 160,000 people will never take part in a poll. Pew estimated 2012 “response rates,” the proportion of eligible respondents who actually complete an interview, in the neighborhood of 9 percent. To see what this means for Iowa, here is a table that presents the average number of interviews a cooperating respondent would have conducted during the 2016 campaign at different response rates. At a ten percent response rate like Pew reports, the 16,000 cooperating prospects would each need to complete an average of 2.78 interviews to reach the total of 44,433.
How many people gave how many interviews?
Finally, I’ll apply the Poisson distribution once again to estimate the number of people being interviewed once, twice, three times, etc., to see the shape of the samples at each response rate.
Even if everyone cooperates, random chance alone would result in about 13 percent of respondents being interviewed at least twice. When the response rate falls to 10 percent, most respondents are being interviewed three or four times, with fifteen percent of them being interviewed five times or more. Even with a 20 percent response rate, about double what Pew reports, a majority will have been interviewed at least twice.
Certainly someone willing to be interviewed three, four, five times or more about a campaign must have a higher level of interest in politics than the typical Iowa caucus-goer who never cooperates with pollsters. That difference could distort the figures for candidate preferences if some candidates’ supporters are more willing to take part in polls.
Basing polls on a relatively small number of cooperative respondents might also create false stability in the readings over time. Polling numbers reflect more the opinions of the “insiders” with a strong interest in the campaign and may be less sensitive to any winds of change. We might also imagine that, as the campaign winds down and most everyone eligible has been solicited by a pollster, samples become more and more limited to the most interested.
Overarching all these findings remains the sobering fact that only about one-in-ten citizens is willing to take part in polling. Pollsters can adjust after the fact for any misalignments of the sample’s demographics, but they cannot adjust for the fact that people who participate in polling may simply not represent the opinions of most Americans. We’ll see how well the opinions of those small numbers of respondents in Iowa and New Hampshire match the opinions of those states’ actual electorates on Primary Day.
