Back in 1988 I had the pleasure of conducting polls during the New Hampshire primaries on behalf of the Boston Globe. The Globe had a parochial interest in that year’s Democratic primary because the sitting Massachusetts governor, Michael Dukakis, had become a leading contender for the Presidential nomination. The Republican side pitted Vice-President George H. W. Bush against Kansas Senator Bob Dole, the upset winner of the Iowa caucuses a week before the primaries. Also in the race were well-known anti-tax crusader Jack Kemp and famous televangelist Pat Robertson. Bush had actually placed third in Iowa behind both Dole and Robertson.
We had been polling both sides of the New Hampshire primary as early as mid-December of 1987, but after the Iowa caucuses, the pace picked up enormously. Suddenly we were joined by large national polling firms like Gallup and media organizations like the Wall Street Journal and ABC News. As each day brought a new round of numbers from one or another pollster, we began to ask ourselves whether we were all just reinterviewing the same small group of people.
Pollsters conducting national surveys with samples of all adults or all registered voters never face this problem. Even with the volume of national polling conducted every day, most people report never being called by a pollster. In a population of over 240 million adults, the odds of being called to participate in a survey, even ones with a relatively large sample like 2,000 people, are miniscule. That is still true even if we account for the precipitous decline in the”response rate,” the proportion of households that yield a completed interview. A wide array of technological and cultural factors have driven survey response rates to historic lows over the past few years as this table from Pew shows clearly:
In 2012, fewer than ten percent of households were represented in a typical poll. Still, even at such a low response rate, the huge size of the United States population means that any individual has only a tiny chance of being selected from a sampling universe numbers of 24 million homes. Even for a large survey of 2,000 people, the chance of any individual household being selected is a mere 0.000008.
Those odds change drastically when we narrow the universe of eligible people to “likely” voters in an upcoming New Hampshire Republican primary. Even including people who claim they will vote but later do not, the total universe of eligible respondents in 2012 was probably just 300,000 people. To reach that figure I started with the total of 248,485 ballots cast in the Republican primary. To those voters we need to add the other people who reported that they would take part in the primary but did not actually turn out on Primary Day. For our purposes, I have used an inflation factor of 20% which brings the estimated the total number of self-reported likely Republican primary voters to 298,182 people. I rounded that figure up to 300,000 in the tables below.
Over a dozen polling organizations conducted at least one survey in New Hampshire according to the Pollster archive for the 2012 Republican primary. In all there are 55 separate polls in the archive representing a total of 36,839 interviews, or about 12% of the universe of likely voters. If all 300,000 likely Republican primary voters had been willing to cooperate with pollsters in 2012, about one in every eight of them would have been interviewed. If we choose a much more realistic response rate like ten percent, there are actually fewer cooperating likely voters than the total number of surveys collected, so some respondents must be contributing multiple interviews. Can we estimate how many there are?
It turns out the chances a person will be interviewed, once, twice, etc., or never at all can be modelled using the “Poisson distribution.” Usually a statistical distribution relies on two quantities, its average and its “variance,” but the Poisson distribution has the attractive feature that the mean and variance are identical. Thus we need only know the average number of interviews per prospect to estimate how many people completed none, one, two, or more interviews. Here are estimates of the number of interviews conducted per potential respondent at different overall cooperation rates. At a 20 percent cooperation rate, only 60,000 of the 300,000 likely voters are willing to complete an interview. Dividing the number of interviews, 36,839, by the estimated number of prospects gives us an average figure of 0.614 interviews per prospect.
Now we plug those values into the Poisson formula to see how many people are interviewed multiple times during the campaign.
In an ideal world where every one of the 300,000 likely primary voters is willing to be interviewed, 88.4% of them would never be interviewed, 10.9% would complete one interview, and 0.7% would be interviewed twice. If response rates fall to 8-10%, only 20-30% of likely voters are never interviewed.
Though only a few prospects would be interviewed more than once in the ideal, fully-cooperative world, at more realistic response rates closer to what Pew reports, many people were interviewed multiple times in the run up to the 2012 primary. If only eight percent of likely voters were willing to complete an interview, about a quarter of the prospects were interviewed twice, and one in five of them were interviewed at least three times.
We can use those estimates to see how the size and composition of the actual survey samples change as a function of response rate.
At 100% cooperation, obtaining nearly 37,000 interviews from 300,000 people means a small number, about 2,000 people, would be interviewed twice merely by random chance. So those 37,000 interviews represented the opinions of 32,000 people who were interviewed once, and another 2,000 people interviewed twice. As response rates fall, the total number of unique respondents, the height of each bar, declines, with a larger share of interviews necessarily coming from people interviewed multiple times. At a 10% response rate the proportion of people interviewed multiple times just about equals the proportion of people interviewed only once. Below that rate the proportion of people interviewed only once declines quickly.
