Technical Appendix: Comparing Trump and Sanders

trump-sanders3

The results above come from the 145 national Republican primary polls as archived by Huffington Post Pollster whose fieldwork was completed after June 30, 2015, and on or before January 6, 2016.  I started with July polling since the current frontrunner, Donald Trump, only announced his candidacy on June 16th. For Bernie Sanders I used the 155 national polls of Democrats starting after April 30th, the day Sanders made his announcement.

The models I am using are fundamentally similar to those I presented for the 2012 Presidential election polls and include these three factors:

  • a time trend variable measured as the number of days since June 30, 2015;
  • a set of “dummy” variables corresponding to the universe of people included in the sample — all adults, registered voters, and “likely” voters as determined by the polling organization using screening questions; and,
  • a set of dummy variables representing the method of polling used — “live” interviews conducted over the phone, automated interviews conducted over the phone, and Internet polling.

Trump’s support is best fit by a “fourth-order polynomial” with a quick uptick in the summer, a plateau in the fall, and a new surge starting around Thanksgiving that levelled off at the turn of the year. Support for Sanders follows a “quadratic” time trend.  His support has grown continuously over the campaign but at an ever slower rate.

Of more interest to students of polling are the effects by interviewing method and sampled universe.  Trump does over four percent worse in polls where interviews are conducted by a live human being.  Sanders does worse in polls that use automated telephone methods.  The result for Trump may reflect an unwillingness on the part of his supporters to admit to preferring the controversial mogul when talking with an actual human interviewer.

Sanders does not suffer from this problem, but polls relying on automated telephone methods show results about four percent lower than those conducted by human interviewers or over the Internet (the excluded category represented by the constant).  Since we know that Sanders draws more support from younger citizens, the result for automated polling may represent their greater reliance on cell phones which cannot by law be called by robots. This result contradicts other studies by organizations like Pew that find only limited differences between polls of cell phone users and those of landline users. Nevertheless when it comes to support for Bernie Sanders, polls that rely exclusively on landlines appear to underestimate his levels of support.

Turning to the differences in sampling frames, we find that polls that screen for “likely” voters show greater levels of support for Bernie Sanders than do polls that include all registered voters or all adults.  Trump’s support shows no relationship with the universe of voters being surveyed.  Both candidates, as “insurgents,” are thought to suffer from the problem of recruiting new, inexperienced voters who might not actually show up at the polls for primaries and caucuses.  That seems not to be an issue for either man, and in Sanders’s case it appears that the enthusiasm we have seen among his supporters may well gain him a couple of percentage points when actual voting takes place.

Finally it is clear that Trump’s polling support shows much more variability around his trend line than does Sanders’s. The trend and polling methods variables account for about 59 percent of the variation in Trump’s figures, but fully 72 percent of the variation for Sanders.

Honey, It’s the Pollster Calling Again!

Back in 1988 I had the pleasure of conducting polls during the New Hampshire primaries on behalf of the Boston Globe.  The Globe had a parochial interest in that year’s Democratic primary because the sitting Massachusetts governor, Michael Dukakis, had become a leading contender for the Presidential nomination.  The Republican side pitted Vice-President George H. W. Bush against Kansas Senator Bob Dole, the upset winner of the Iowa caucuses a week before the primaries. Also in the race were well-known anti-tax crusader Jack Kemp and famous televangelist Pat Robertson.  Bush had actually placed third in Iowa behind both Dole and Robertson.

We had been polling both sides of the New Hampshire primary as early as mid-December of 1987, but after the Iowa caucuses, the pace picked up enormously. Suddenly we were joined by large national polling firms like Gallup and media organizations like the Wall Street Journal and ABC News.  As each day brought a new round of numbers from one or another pollster, we began to ask ourselves whether we were all just reinterviewing the same small group of people.

Pollsters conducting national surveys with samples of all adults or all registered voters never face this problem.  Even with the volume of national polling conducted every day, most people report never being called by a pollster.  In a population of over 240 million adults, the odds of being called to participate in a survey, even ones with a relatively large sample like 2,000 people, are miniscule.  That is still true even if we account for the precipitous decline in the”response rate,” the proportion of households that yield a completed interview.  A wide array of technological and cultural factors have driven survey response rates to historic lows over the past few years as this table from Pew shows clearly:

In 2012, fewer than ten percent of households were represented in a typical poll.  Still, even at such a low response rate, the huge size of the United States population means that any individual has only a tiny chance of being selected from a sampling universe numbers of 24 million homes.  Even for a large survey of 2,000 people, the chance of any individual household being selected is a mere 0.000008.

Those odds change drastically when we narrow the universe of eligible people to “likely” voters in an upcoming New Hampshire Republican primary.  Even including people who claim they will vote but later do not, the total universe of eligible respondents in 2012 was probably just 300,000 people.   To reach that figure I started with the total of 248,485 ballots cast in the Republican primary.  To those voters we need to add the other people who reported that they would take part in the primary but did not actually turn out on Primary Day.  For our purposes, I have used an inflation factor of 20% which brings the estimated the total number of self-reported likely Republican primary voters to 298,182 people.  I rounded that figure up to 300,000 in the tables below.

Over a dozen polling organizations conducted at least one survey in New Hampshire according to the Pollster archive for the 2012 Republican primary.  In all there are 55 separate polls in the archive representing a total of  36,839 interviews, or about 12% of the universe of likely voters.  If all 300,000 likely Republican primary voters had been willing to cooperate with pollsters in 2012, about one in every eight of them would have been interviewed.  If we choose a much more realistic response rate like ten percent, there are actually fewer cooperating likely voters than the total number of surveys collected, so some respondents must be contributing multiple interviews.  Can we estimate how many there are?

It turns out the chances a person will be interviewed, once, twice, etc., or never at all can be modelled using the “Poisson distribution.”  Usually a statistical distribution relies on two quantities, its average and its “variance,” but the Poisson distribution has the attractive feature that the mean and variance are identical.  Thus we need only know the average number of interviews per prospect to estimate how many people completed none, one, two, or more interviews.  Here are estimates of the number of interviews conducted per potential respondent at different overall cooperation rates.  At a 20 percent cooperation rate, only 60,000 of the 300,000 likely voters are willing to complete an interview.  Dividing the number of interviews, 36,839, by the estimated number of prospects gives us an average figure of 0.614 interviews per prospect.

how-often-republicans-polled-table1

Now we plug those values into the Poisson formula to see how many people are interviewed multiple times during the campaign.

how-often-republicans-polled-table2

In an ideal world where every one of the 300,000 likely primary voters is willing to be interviewed, 88.4% of them would never be interviewed, 10.9% would complete one interview, and 0.7% would be interviewed twice.  If response rates fall to  8-10%, only 20-30% of likely voters are never interviewed.

Though only a few prospects would be interviewed more than once in the ideal, fully-cooperative world, at more realistic response rates closer to what Pew reports, many people were interviewed multiple times in the run up to the 2012 primary.  If only eight percent of likely voters were willing to complete an interview, about a quarter of the prospects were interviewed twice, and one in five of them were interviewed at least three times.

We can use those estimates to see how the size and composition of the actual survey samples change as a function of response rate.

sample-size-and-composition2

At 100% cooperation, obtaining nearly 37,000 interviews from 300,000 people means a small number, about 2,000 people, would be interviewed twice merely by random chance.  So those 37,000 interviews represented the opinions of  32,000 people who were interviewed once, and another 2,000 people interviewed twice.  As response rates fall, the total number of unique respondents, the height of each bar, declines, with a larger share of interviews necessarily coming from people interviewed multiple times.  At a 10% response rate the proportion of people interviewed multiple times just about equals the proportion of people interviewed only once.  Below that rate the proportion of people interviewed only once declines quickly.

Technical Appendix: The Model for Health Insurance Coverage

Dependent variable: Uninsured Adults 19-64 without Dependents
                    2009-2010 data for 49 states except MA

All variables are proportions and measured as "logits."
                coefficient   std. error   t-ratio   p-value 
  -----------------------------------------------------------
  const           1.22691     0.347772      3.528    0.0010   ***
  lgt_U3          0.728698    0.0918831     7.931    5.94e-10 ***
  lgt_Extr+Cons   0.280012    0.0747943     3.744    0.0005   ***
  lgt_UnPriv     −0.164442    0.0460793    −3.569    0.0009   ***
  lgt_NotEng      0.119194    0.0296539     4.020    0.0002   ***
  lgt_McCain      0.217339    0.0810080     2.683    0.0103   **

S.E. of regression   0.141875   Adjusted R-squared   0.726246

The predictors include:

U(3) – the state’s U(3) unemployment rate;
Extr+Cons – the proportion of the state’s workforce in mining, logging, and construction;
UnPriv – the proportion of the state’s private workforce that is unionized;
NotEng – the proportion of the state’s citizens who have no English language skills;
McCain – the proportion of the state’s 2008 Presidential vote won by Republican John McCain

Sources:
U(3) – Bureau of Labor Statistics, Alternative Measures of Labor Underutilization for States, 2010;
Extr+Cons – Bureau of the Census, Table 631. Employees in Nonfarm Establishments–States: 2010
UnPriv – Barry Hirsch and David Macpherson, Union Membership and Coverage Database from the CPS
NotEng – Census Bureau, Table 4A.  Population 5 Years and Over Speaking a Language Other Than English at Home by English-Speaking Ability by State: 2007; percent responding “not at all” to speaking ability;
McCain – David Leap, Atlas of U.S. Presidential Elections, Results for 2008

Technical Appendix: The Model for Voting on the Amash Amendment

I took a more conventional, less spatially-oriented approach than that used on Voteview.  I used logit analysis to estimate a model using the two DW-NOMINATE scores and a variety of dummy variables to measure other possible influences like when the Member was elected and the committees on which the Member serves.  The dataset consists of the 342 Members who served in the 112th Congress (and thus for whom the DW-NOMINATE scores are publicly available) and voted on the Amash Amendment.

It soon became clear that the relationship between support for the amendment and ideological position was more complex than a simple linear model would predict.  What piques our interest in this vote is how the two parties divided over the amendment more than the division between the parties themselves.  I thus included separate terms for each ideological dimension within each party.  After doing so, including a dummy variable for party has no independent effect.

I also examined various measures of seniority in an effort to see whether there is any truth to pundits’ observation of a generational divide between older and newer members over the issue of domestic surveillance.  It turns out that the generational divide is especially pronounced for Republicans.  Those who were first elected to Congress in 2008 or 2010 were more likely to vote for the Amash amendment regardless of ideology.  For the Democrats, the results are more muted.  Those Democrats who were voted into office alongside President Obama in 2008 were especially likely to oppose him on domestic surveillance.  However the few Democrats who were first elected in 2010 were no more likely to support the amendment than Democrats elected in 2004 or before.

Finally there is a strong effect for committee memberships.  Members who serve on the House Armed Services or Select Intelligence committees were much more likely to vote against the Amash amendment.  The effect was especially pronounced for members of the Intelligence Committee.

Favored Amendment to Cut Funds for NSA Metadata Collection
Logit, 342 observations

             coefficient   std. error      z      p-value 
  --------------------------------------------------------
  Constant    −4.11799      0.699829    −5.884    4.00e-09 ***

DW-Nominate Scores

Dimension 1 ("Liberal-Conservative")
  Dems       −12.2661       1.82358     −6.726    1.74e-11 ***
  Reps         5.13557      0.972324     5.282    1.28e-07 ***
Dimension 2 (?)
  Dems        −0.879454     0.662692    −1.327    0.1845  
  Reps        −0.302442     0.642723    −0.4706   0.6380  

First Elected to Congress
  Dems 2008     3.25482      1.07614      3.025    0.0025   ***
  Reps 2008-10  0.751204     0.334374     2.247    0.0247   **

Committee Memberships
  Armed Srvcs  −1.03730      0.453937    −2.285    0.0223   **
  Intel        −3.15582      0.992305    −3.180    0.0015   ***

Estimated R2 (McFadden) = 0.276671
Number of cases 'correctly predicted' = 258 (75.4%)

While we have come some way to understanding the factors motivating Members’ votes on the Amash amendment, the model still cannot account for the votes of about a quarter of the House.

Was Gerrymandering the Culprit? — Part I

Results updated on November 23, 2012, with final Congressional results for 434 races; NC 7th is still undecided.

It is now time to put some of the findings from earlier postings together and try to determine the extent of gerrymandering in the 2012 Congressional Elections.

Three factors should influence the number of House seats a party wins in a state Congressional election:

I have taken two separate measurements of the first item, the relationship between seats and votes.  I have calculated both a longitudinal measurement using elections from 1942 on, and a cross-sectional measurement using state results for 2012.  In both approaches I estimate the coefficients α and β of this “logit” model:

log(Democratic Seats/Republican Seats) = α + β log(Democratic Votes/Republican Votes)

The two models produce very different estimates for α, the seat “bias,” because it varies historically.  However the two estimates for β are nearly identical. The longitudinal estimate was 1.92; the cross-sectional estimate is 2.08.  For simplicity, I will just use two for the value of β.  (Mathematically, that implies that the ratio of Democratic to Republican seats varies in proportion to the square of the ratio of their votes.)

In this Technical Appendix, I explain why, if the Democrats win exactly half the vote, the only way they can win exactly half the seats is if the “bias” term α is zero. We can use this fact to create an “unbiased” distribution of seats.  I simply substitute two for β and apply it to the logit of the state-wide Democratic vote for Congress.  I will call this the “unbiased allocation.”  For each state I compare this estimate to the number of seats the Democrats actually won. Here are the results:

I have included all states where the difference between the predicted and actual number of Democratic seats was at least 0.7.  The state that gave us the word “gerrymander,” Massachusetts, shows the largest pro-Democratic deviation.  While the unbiased allocation model would award the Democrats only seven or eight of the nine seats in that state, not one Republican represents the Commonwealth of Massachusetts in Congress. The other state where Democrats did better than expected is Arizona, where they won a majority of the state’s Congressional seats with a minority of the popular vote.  Arizona had two of the closest races in the country, and they both fell to the Democrats by slim margins. All told, eight states including four New England states, have new Congressional delegations with an “extra” Democratic member in their numbers.

Many more states deviate from the unbiased allocation on the Republican side, with half-a-dozen states showing a pro-Republican bias of two, three, or, in the case of Pennsylvania, four seats. All told, sixteen states met our 0.7 criterion.  Compared to an unbiased allocation, the results in these sixteen states probably cost the Democrats 28 seats.  When we subtract out the eight extra seats the Democrats won in the pro-Democratic states, we get a net Democratic deficit in 2012 of some twenty seats compared to an “unbiased” allocation based solely on the popular vote for Congress in each state.

Before we start attributing all those seats to Republican gerrymandering, we first need to consider what other factors might influence the translation of Democratic votes to Democratic seats.  There is good reason to believe that the geographic distribution of Democratic voters by itself creates a pro-Republican bias when district lines are drawn.

 Accounting for Geography

 

Technical Appendix – Estimating the Effects of Reapportionment

Estimating the Effects of Redistricting
I took a rather simple-minded approach to the task of measuring redistricting effects.  I began by measuring the relationship between the national vote for Democratic Congressional candidates and the number of seats the Democrats won for all Congressional elections beginning in 1942.  (The 2012 data do not include four undecided seats.) I picked 1942 because it was the first election fought on seats whose boundaries were based on a New Deal Census.

I began by estimating the relationship between the proportion of seats won by the Democrats and the proportion of their vote.  I used a slightly more sophisticated statistical method here than I did when looking at the Electoral College.  I again used linear regression to estimate the relationship between seats and votes but only after first transforming each of the proportions using the “logistic” function.  In brief, I am estimating the model:

ln(Democratic Seats/Republican Seats) = α + β ln(Democratic Votes/Republican Votes)

Instead of measuring the proportion or percent of seats and votes, I transformed each variable into the (natural) logarithm of its “odds ratio.”  Suppose the Democrats hold 60% of the House of Representatives.  If I choose a seat from the House at random, the odds of my drawing a Democratic seat are 60:40.  That is the odds ratio; in this case we could also call it 3:2 after simplifying the fraction.  If I take the logarithm of this odds ratio, I get a very “well-behaved” variable.  It is no longer constrained to the range between zero and one like proportions are, and the logarithmic transformation turns the non-negative odds ratio into a continuous variable that encompasses the whole number line.  At even odds, or a odds ratio of one, the logarithm is zero.  For values below 50:50 we get negative values; for values above 50:50 the values are positive ones.  This is commonly called a “logit” transformation.

We can use this fact that the logit is zero at 50% to determine whether an electoral system is “biased.”  If  we define an unbiased system as one that awards half the seats to a party winning half the vote, that definition of unbiasedness requires that the intercept term α be zero.  (This is true no matter what value we use for β.)  If α is not zero, a vote share of 50%, whose logit is zero, will predict a share either greater or less than 50% depending on the sign of α.  A positive value means the party wins more seats than it “deserves” based on its share of the popular vote; a negative value means the party was “shortchanged.”

Using this equation as the basis, we can measure the bias associated with different apportionments by testing the “null hypothesis” that the value of α in an apportionment period is equal to zero.  I define an “apportionment period” as the five elections that begin two years after a Census and end in the next Census year.  All apportionment periods begin with an election in a year ending in two and end in the next election when the year ends in zero. As an example, the apportionment period associated with the 1960 Census begins in 1962 and ends in 1970.

This is a very crude measure of differences in apportionment to be sure.  I could create a more fine-grained measure that includes important factors like the use of nonpartisan commissions, the partisan divisions of the state legislatures in the apportionment year, and whether a party controlled all three branches of state government.  These are worthwhile tasks best engaged in after we see whether we can detect any partisan effects from apportionment simply by examining variations across the apportionment periods.

As a crude first step, then, I have created “dummy” variables for each apportionment period.  These variables have the value one for elections held in that period and zero otherwise.  For instance, the Census_40 variable in the results below has the value one for each of the five elections from 1942 to 1950 and zero after that.  Here are the results:

OLS, 36 Congressional Elections, 1942-2012
Dependent variable: log(% Dem Seats/(100 - % Dem Seats))

              coefficient   std. error   t-ratio   p-value 
  ---------------------------------------------------------
  Apportionment Biases (α)  
  Census_40    0.0673105    0.0171312     3.929    0.0005   ***
  Census_50    0.0220976    0.0185816     1.189    0.2447  
  Census_60    0.0741743    0.0191011     3.883    0.0006   ***
  Census_70    0.0645399    0.0210222     3.070    0.0048   ***
  Census_80    0.0431043    0.0204332     2.110    0.0443   **
  Census_90    0.0134422    0.0171097     0.7856   0.4389  
  Census_00   −0.0124388    0.0171423    −0.7256   0.4743  
  Census_10   −0.0881460    0.0382346    −2.305    0.0291   **

  Responsiveness (β)  
  lgt_D_Vote   1.77527      0.146841     12.09     2.09e-12 *** 

Mean dependent var   0.100578   S.D. dependent var   0.121026
Sum squared resid    0.039463   S.E. of regression   0.038231
R-squared            0.923023   Adjusted R-squared   0.900215
F(8, 27)             40.46924   P-value(F)           4.57e-13

*p<0.10; **p<0.05; ***p<0.01

This model does not have a constant term, for I have included all the dummies for the apportionment periods instead.  In this formulation each coefficient is measured as a deviation from zero, our standard for unbiasedness.  According to the statistical tests the period from 1992 until 2010 had no measurable bias.  The two tiny values we measure for those decades are not even as large as their standard errors.  The elections fought in the seats drawn after the 1950 Census also stand out as much less Democratic than in any other decade before the 1990s.

Looking more closely at the estimated coefficients, or at the graph they generate, makes it clear there are three distinct periods in these results.  From 1942 to 1990 the Democrats were the beneficiaries of a seven-seat advantage in the House of Representatives, excluding the one apportionment period beginning in 1952.  This period of Democratic dominance was followed by two decades of parity where the electoral system advantaged neither party.

One other way to view these historical patterns trends more clearly is to simplify the model above.  By removing the Census_40 variable and replacing it with a constant term, we can envision how the period of Democratic dominance gave way to parity.  Remember that statistically this model is identical to the one above, but the interpretation of the coefficients is different.

  Constant     0.0673105    0.0171312     3.929    0.0005   ***
  lgt_V_Dem    1.77527      0.146841     12.09     2.09e-12 ***
  Census_50   −0.0452129    0.0255820    −1.767    0.0885   *
  Census_60    0.00686379   0.0260132     0.2639   0.7939  
  Census_70   −0.00277057   0.0276003    −0.1004   0.9208  
  Census_80   −0.0242061    0.0271129    −0.8928   0.3799  
  Census_90   −0.0538683    0.0241829    −2.228    0.0344   **
  Census_00   −0.0797493    0.0242902    −3.283    0.0028   ***
  Census_10   −0.155457     0.0419114    −3.709    0.0010   ***

What we see now is a period of Democratic dominance that stretched from the 1940s through the 1980s with the exception of the decade following the 1952 reapportionment.  Republicans had taken control of 26 state legislatures  in 1950 compared to just 16 for the Democrats and thus controlled the redistricting process in many states.  Our data suggest they were able to reduce the Democrats’ advantage substantially in that decade, though we will see in a moment that the situation is even more complicated.

Elections over the two decades from 1962 to 1980 show the same pro-Democrat advantage that elections held in the 1940s do.  Reapportionment actions in those three decades show no significant deviation from the 1940s baseline.  After 1992, though, the situation changes.  After the 1992 elections the Democrats’ advantage fades quickly.  The effects for 1992-2010 essentially eliminate that advantage and usher in two decades where neither party was advantaged or disadvantaged by the workings of the electoral system.  However the trends in the Republicans’ favor reach historic proportions in the 2012 election.  The Republicans have now turned things in their favor beginning with the election of 2012.

I have not spoken at all about β, the slope coefficient that measures how changes in the popular vote odds translate into changes in the number of seats won. Larger values of this coefficient increase the steepness of the relationship between seats and votes. British statisticians as early as 1950 talked about  “cube law” relationship between seats and votes.  In terms of our model that translates into a value for β of three.  For Congressional elections since 1940 I estimate a value of 1.78, considerably below the “cube-law” value, but still substantially higher than one, which would indicate pure proportionality or “parity” as I call it in the graph to the right.   A party whose share of the Congressional vote rises from 40% to 60% should expect to see their share of seats increase from 33% to 67%.  A cube law system is much more ruthless giving the party at 40% a mere 23% of the seats, while one that wins 60% of the vote gets an enormous bonus winning 77 % of the seats in the legislature.

 

Final Model Estimates

Updates to the simple trends and house effects models for all 190 national likely-voter polls released as of noon on Monday, November 5th and archived at Huffington Post Pollster.

Polling Methods
If we include only interviewing methods and exclude the pollster house effect dummies, we find a significant pro-Republican effect for automated methods and an modest pro-Democratic effect for Internet polling. However the results for automated polling are really just a proxy for Rasmussen. When a dummy variable for that firm is included, the effect for automated polling becomes insignificant. Other firms like PPP that employ automated interviewing show no significant biases in either direction.

175 Likely Voter Polls from Pollster (7/1-11/4) 
Dependent variable: Obama Lead over Romney
Ordinary Least Squares

              coefficient   std. error   t-ratio   p-value 
  ----------------------------------------------------------
  const         3.66461      0.637713      5.746    4.19e-08 ***
  DaysBefore   −0.0194144    0.0102005    −1.903    0.0587   *
  Debate1      −3.40771      0.570555     −5.973    1.35e-08 ***
  Sandy         0.969663     0.727894      1.332    0.1846  
  Internet      1.17270      0.533334      2.199    0.0293   **
  Auto         −0.140361     0.662144     −0.2120   0.8324  
  Rasmussen    −2.46427      0.760485     −3.240    0.0014   ***

Mean dependent var   0.965714   S.D. dependent var   2.866567
Sum squared resid    936.3252   S.E. of regression   2.360797
R-squared            0.345133   Adjusted R-squared   0.321745
F(6, 168)            14.75677   P-value(F)           1.64e-13

The same cannot be said for Internet polling. Including or excluding the dummy variables for individual firms that use Internet polling does not eliminate the effect for that method. On the average, firms that use Internet polling report results about one percent more Democratic than firms who use human  interviewers.  The final model below keeps the Internet effect but excludes the measure for automated polling.

175 Likely Voter Polls from Pollster (7/1-11/4) 
Dependent variable: Obama Lead over Romney
Ordinary Least Squares

                  coefficient   std. error   t-ratio   p-value 
  --------------------------------------------------------------    
  const             3.77159      0.602485      6.260    3.17e-09 ***
  DaysBefore       −0.0211743    0.00981172   −2.158    0.0324   **

  Debate1          −3.21365      0.549629     −5.847    2.59e-08 ***
  Sandy             0.535302     0.701852      0.7627   0.4467  

  Internet          1.05117      0.508563      2.067    0.0403   **

  Rasmussen        −2.63924      0.467508     −5.645    6.99e-08 ***
  DemocracyCorps    2.05705      1.04816       1.963    0.0514   *
  ARG              −1.54940      0.885694     −1.749    0.0821   *
  Gallup           −4.55248      1.33905      −3.400    0.0008   ***

Mean dependent var   0.965714   S.D. dependent var   2.866567
Sum squared resid    842.2114   S.E. of regression   2.252457
R-squared            0.410956   Adjusted R-squared   0.382569
F(8, 166)            14.47659   P-value(F)           6.25e-16

House Effects
I have been much stricter in my criteria for house effects in this final model. I have only included these effects if they are significant at the 0.10 level and are based on more than one poll. Organizations that released a single poll in our timeframe like Reason/Rupe are not singled out in this model. That leaves five firms with identifiable biases. Gallup, Rasmussen and ARG all tilt Republican while DemocracyCorps and, especially, Bloomberg show Democratic biases. The Bloomberg result is based on just two polls and may not represent a systematic house effect. The Gallup result is based on the three tracking poll reports after the firm switched to using its likely-voter model. Gallup stopped interviewing after Hurricane Sandy so there are no polls from that firm more recent than October 28th.

Trends and Events
No trend changes are seen after the first Presidential debate nor after Hurricane Sandy.  Including “interaction” terms for the product of DaysBefore with these two variables produce insignificant results. The overall effect of the debate was to reduce the President’s estimated Election Day margin in polling from 3.8% to 0.6%. Polling since Hurricane Sandy shows a modest 0.5% increase in support for the President, but it fails to achieve statistical significance.  However the pace at which President Obama gains support over time falls to its lowest overall level in this model. Over the course of the campaign the President increased his margin of victory by one percentage point every 47 days (=1/0.0211743).

Polls Converge as Election Nears

The graph above plots the standard deviation of each week’s national likely-voter polls using the midpoint of the fieldwork period as the basis for classification.  Values before about mid-August are based on small numbers of polls and are thus more noisy.  Starting with the week of September 9th, there are at least 10 polls included in each week’s estimate.

As expected the variation across pollsters has diminished as we get close to Election Day.  Over the past week the standard deviation of the estimates for Obama and Romney support,  and for the margin between them, has converged to about 1.1-1.2, about a third the value seen during the month of September.

Estimates of support for President Obama have shown more variability than those for his opponent.  The average standard deviation over this period was 1.95 for the President compared to 1.56 for Governor Romney.  Naturally the lead shows much more variability than the estimates for either candidate with an average value of 2.73.

Critics of Rasmussen’s polling suggest his results converge with the results of the other pollsters as Election Day nears.  If we plot his polls against the average for all other pollsters, we find less evidence for convergence than the critics allege.

While it is obvious that Rasmussen’s polls ran more Republican than the consensus, as I have shown before, there is still a substantial gap of about two points between his results and the other pollsters.  His most recent figures put the race as tied nationally while the consensus has President Obama ahead by somewhat over two percent.

Revised Trend Models with House Effects

Includes five additional polls released between 10/25 and 10/28.

146 Likely Voter Polls from Pollster (7/1-10/28) 
Dependent variable: Obama Lead over Romney
Ordinary Least Squares

                 coefficient   std. error   t-ratio   p-value 
  ------------------------------------------------------------
  const           5.72235      0.579939      9.867    1.54e-17 ***
  Debate1        −5.47401      1.05759      −5.176    8.24e-07 ***

  Trends
  Pre-Debate     −0.0514766    0.00937824   −5.489    2.00e-07 ***
  Post-Debate    −0.0124666    0.0424105    −0.2940   0.7693

  House Effects
  Rasmussen      −2.35773      0.437616     −5.388    3.18e-07 ***
  Gallup         −4.34481      1.18503      −3.666    0.0004   ***
  YouGov/Econ     1.80215      1.02690       1.755    0.0816   *
  ARG            −1.93337      0.844044     −2.291    0.0236   **
  DemCorps        2.15593      1.03252       2.088    0.0387   **
  UTech/NJ       −3.81771      2.00889      −1.900    0.0596   *
  NPR             2.74136      1.44450       1.898    0.0599   *
  Allstate/NJ     3.74853      2.00311       1.871    0.0635   *
  Reason/Rupe     3.85149      2.00263       1.923    0.0566   *
  JZAnalytics     2.65474      0.790120      3.360    0.0010   ***

Mean dependent var   1.075342   S.D. dependent var   3.055996
Sum squared resid    519.2214   S.E. of regression   1.983304
R-squared            0.616576   Adjusted R-squared   0.578815
F(13, 132)           16.32820   P-value(F)           1.13e-21

For models like this with Rasmussen included, I created separate pre- and post-debate trends and tested whether each significantly differs from zero.  This model shows that the trend before the Denver debate was significant favorable to the President, but since the debate the race has stagnated.

 

Partisan House Effects in National Polls

Until now I have been using data from the polling archives at RealClearPolitics for this blog.  Today I began looking at the  larger archive of polls at the Huffington Post’s Pollster site.  One nice feature of this site is that they offer a copy of their data in a format (“.csv”) that can be easily imported into spreadsheets or the gretl econometrics package.

I produced the table above from a regression of the size of President Obama’s lead over Governor Romney using the 146 national likely-voter polls in the Pollster database with fieldwork starting after June 30th and ending October 28th.  Along with “dummy variables” to capture any differences among the 37 polling organizations represented in this sample I included a few other important predictors:

  • the number of days remaining between the end of fieldwork and Election Day, November 6th;
  • a dummy variable for polls whose fieldwork began after the first Presidential debate on October 3rd;
  • an “interaction” term that is the product of these last two variables to allow the estimated trend line to differ before and after the debate; and,
  • dummy variables for the method of polling using Pollster’s categories of in-person telephone interviews, automated telephone interviews, and Internet interviews.

As you might imagine, only a few of the polling organizations diverge so markedly from the consensus that we can statistically measure any effect for them.  I narrowed down the search to the ten organizations that appear in the table above which have discernible partisan house effects.

Five organizations show “statistically significant” house effects at conventional levels (p<0.05).  Three report figures with a measurable pro-Republican bias, Gallup, ARG, and Rasmussen, while two, JZAnalytics and the openly partisan DemocracyCorps, report figures favorable to the President.  Gallup’s three recent likely-voter polls diverged so substantially from the polling consensus that it tops our list with an estimated four-point tilt for Romney.  ARG and Rasmussen are often suspected of GOP leanings, and this analysis estimates that their polls lean about two percent more Republican than the model’s consensus.  Polls conducted by JZAnalytics, either alone or with co-sponsors Newsmax and the Washington Times, report results 2.6% more Democratic than the consensus.  Polls by DemocracyCorps run a bit over two percent more Democratic.

Rasmussen’s pro-Republican leaning is especially important when you consider how it dominates the polling landscape.  There are 39 Rasmussen polls in the sample, or 27% of these 146 polls from the Pollster database. I’ve looked at what the polling consensus would be like in a world without Rasmussen in this post.

Another five organizations reported results that deviated sufficiently from the consensus that they met the criterion of statistical significance at the 10% level.  Three of these come from organizations that conducted only one poll in this period, and all three had bias figures approaching four percent.  NPR’s two polls averaged 2.8% more favorable to the President, while the YouGov/Economist polls lead a bit under two points in the President’s direction.

I found no systematic differences by type of interviewing method used.  Interviewing by live telephone, automated telephone, or the Internet does not produce results that systematically favor one candidate over the other.  The fact that automated interviewing shows no consistent effects suggests that including cell phones in the sampling frame may not matter at all.  The firms like Rasmussen and PPP that use automated interviewing are banned from calling cell phones by Federal rule.  Yet there is no evidence of a bias in automated interviewing where cell phones are excluded.

The full results appear here.