How Doug Jones Won

Comparing last night’s results for the Special Election in Alabama to prior elections in that state shows the path which brought Doug Jones his unlikely victory.  Like all special elections, the Alabama election unsurprisingly failed to mobilize as many voters as last year’s Presidential contest, but the turnout last night did exceed the 2014 mid-term figure by four percentage points.  Alabamians apparently thought this race was worth the effort.

Jones gained nearly as many votes in this special election as Hillary Clinton polled in November, 2016, despite an electorate 37 percent smaller than for the Presidential election.

Doug Jones expanded the Democratic electorate in the largest Alabama counties when we compare his results to those polled by 2014 Democratic gubernatorial candidate Parker Griffith.  (There was no Democratic candidate for Senator on the ballot that year.)

Jones won 671,151 votes last night, an improvement of 57 percent statewide over Griffith’s 427,787 total three years ago.  The graph shows clearly that his advantage grew as a function of county size as measured by total turnout.  Jones’s best performances were in Madison County, where aerospace center Huntsville is located; Shelby County, which includes suburban Birmingham; and Baldwin County, just east of Mobile.  Moore carried both of the latter counties over Jones but by severely diminished margins compared to Republican performance in prior elections. Jones doubled the Democratic vote in Lee County, home to Auburn University, and made a significant gain in Tuscaloosa where the University of Alabama is located.

Moore’s support, in contrast, was strongest in the smallest counties.  Here I am comparing Moore’s performance last night to the total vote cast in the Republican primary runoff election against Luther Strange late last September.  Moore needed to mobilize sufficient numbers of Strange voters to add to his own totals going into last night’s election.  He failed to do so.

Statewide Moore received about 35 percent more votes than he and Strange together polled in September.  However Moore saw his smallest gains in the largest counties, the opposite of the pattern we saw for Jones.

Running a simple regression of Jones’s lead over Moore against demographic variables shows the dominant power of mobilized African-Americans with smaller effects for the proportions of Hispanics and people with a college degree.

In a district with no blacks, no Hispanics, and no one with a college degree, Moore would beat Jones by 79 points, e.g., 89-10.  The size of the black population played the most important role in determining support for Jones.  His lead expanded by 1.6 percent for every one percent increase in the percentage of blacks.  Hispanics played a less significant role with a coefficient about half the size of that for blacks that barely achieves statistical significance.  The proportion of a county’s residents holding a college degree mattered nearly as much as the size of its black population. These three variables alone account for about ninety-four percent of the variance in Jones’s lead over Moore.

Jones’s victory with a margin of 1.5 percent over Moore suggests that the polls that included cell phone respondents were right on track.  As I wrote yesterday, “Taken together, [my analysis of polling data] suggests that Jones has averaged a 1-2 percent lead in polls taken since the Washington Post story that included calls to cell-phone users.”  Polls limited to landline households, which predicted a Moore victory, were off the mark.

 

Some Observations on the Polling in Alabama

Alabama’s citizens will head to the polls to vote in the most competitive Senate election that state has seen for decades.  Democrat Doug Jones is trying to wrest the state into his party’s column while Republicans rally behind Roy Moore.  Most anyone reading this blog knows about the issues in this race, so I’m going to focus solely on the polls as archived at RealClearPolitics.  Since RCP does not publish data on polling methods, I examined the individual reports for each poll and, when the method used was unclear, contacted the polling agencies directly.

Polling results in this race have shown little convergence as we approach election day.  The Republican dominance in Alabama’s elections has meant that few national polling agencies have paid much attention to the state over the recent elections.  As a result, few national polling organizations have much experience surveying Alabama’s voters.  That has changed a bit as the election achieved national prominence, but still the vast majority of Alabama polls come from organizations will limited track records.  Over at FiveThirtyEight, Harry Enten observes that “Alabama polls have been volatile, this is an off-cycle special election with difficult-to-predict turnout, and there haven’t been many top-quality pollsters surveying the Alabama race.”

“Top-quality” pollsters rely on live interviewers making calls to both landline phones and cell phones.  FiveThirtyEight adds the additional criterion that the polling agency be involved with national organizations like the American Association for Public Opinion Research.  I will limit my analysis to just whether calls were made to a sample of cell phone owners.  As it turns out, this factor alone has a profound effect on a poll’s estimated margin between the candidates.

Here is a list of the available polls based on their method of interview.

Only one poll among those that included interviews with respondents via cell phone shows a lead for Moore; in contrast, only one of the landline-only polls puts Jones ahead.  The “swing” is quite substantial, about an eight-point differential based on the method used.

This difference arises from the much higher usage of cell phones by younger respondents who prefer Jones in most polls.  For instance, in today’s poll from Fox News likely voters under 45 year of age preferred Jones 59% to 28%, while voters above that age preferred Jones by only a one point margin, 45% to 44%.

I also modeled the difference in support between Jones and Moore using my standard predictors, time left before the election, and dummy variables for polling methods.  I also added a dummy which is coded one beginning on November 9th when the story about Moore’s alleged molestation of young girls was released in the Washington Post.  The variable measuring proximity to the election proved statistically insignificant, leaving just three dummies, whether the pollster made calls to cell phones, whether live interviewers were used, and whether the story had broken in the Post.

In polls taken before the publication of the molestation story, Jones trailed Moore by an average of eleven points.  Since then Jones has seen an average gain of six points, not enough on its own to return the race to even.  However polls that interviewed respondents via cell phones show a slightly larger difference of nearly seven points in Jones’s favor.  Taken together, these results suggest that Jones has averaged a 1-2 percent lead in polls taken since the Washington Post story that included calls to cell-phone users. (Update: Jones’s margin of victory over Moore was 1.5 percent statewide, right in line with this prediction.)

I also included a term for whether live interviewers were used.  Since all polls that include cell phone owners must use live interviewers by law, this remaining group represents organizations that polled only landline owners with human interviewers.  I find a small, though statistically insignificant (p<0.17) positive effect on Jones’s support from people surveyed by live interviewers.  It is hard to interpret what this effect might signify.  It could represent an unwillingness among Moore’s supporters to admit their intentions to a human interviewer but have no such hesitation when the interview is conducted by a robot.  If so, we might attribute some of the difference between cell phone and landline results to use of human interviewers in polls that include cell users.

 

A Simple Model of Senate Elections

Without a President seeking re-election at the top of the ticket, the Democrats face a substantial challenge if they wish to win back the Senate in 2016.

While the Presidential race gets all the media and pundit attention, the battle for control of the Senate also looms large in this election year.  Republicans enter the election holding 54 of the 100 Senate seats, so a net Republican loss of just five seats would put the Senate back in Democratic hands.  The Democrats have the advantage that many more Republican seats, twenty-four, are at risk in the 2016 election compared to only ten held by Democrats.  This lopsided margin reflects the result of the 2010 off-year election when Republicans picked up six seats from the Democrats.  In principle, some of those Republican senators may be more vulnerable in a Presidential year with higher turnouts and more visibility.  The Democrats certainly believe they can retake the Senate this November.  The Party aggressively recruited candidates for the Senate elections and had secured bids from all but one of its top-tier candidate selections by early October of 2015.

The Democrats also have the advantage that the party of the incumbent President usually wins a slim majority of the Senate vote in “on-year” elections when a Presidential election also takes place but loses in “off-year” elections.

on-off-year

Unfortunately for the Democrats the relationship between Senate electoral success and type of election is not so simple.  If I divide up on-year elections into ones when the President ran for re-election and ones when, like the upcoming election, he did not, a very different pattern emerges.  The President’s party fared substantially worse in the five open-seat elections since 1946 than it did in elections with the President at the top of the ticket.  While open-seat years gave the President’s party a one-percent boost compared to off-years, that difference is not statistically significant.  What matters is whether the President is running or not.

on-off-open-year2

The Democrats’ optimism is also based on the much larger number of Republican seats at risk in 2016.  I find some support for the notion that a Senate “Class” with a comparatively lopsided division of the vote in one election becomes more competitive six years later.  Statisticians call this phenomenon “regression toward the mean,” where observations that were outliers at one time show more average scores when measured again. But this effect is weak, and the division of the Senate vote in 2010, 53-47 percent Republican, was not as lopsided as the margin in terms of seats, 65-35 percent Republican.  All told the estimated “rebound” effect given the Republican 2010 landslide is just 0.6%, raising the expected Democratic vote from 47.1% to 47.7%.

Where else might the Democrats gain some relief?  Perhaps the generally positive state of the economy might provide some help.  Political scientists and economists have tested many different measures of economic conditions in models of voting for President and Congress.  One simple measure that has consistently proven significant is the change in personal income, and that proves true for Senate elections as well.

pred-econ-incum2

This chart adds the effects of the year-on-year percent change in real disposable personal income to our simple model.  While the effect of rising incomes is positive and statistically significant, it alone cannot overcome the substantial deficit facing the Democrats in an open-seat election year.  In the six years of the Obama Administration, personal income rose by at most 2.2 percent in a single year, 2012.  With a likely figure for annual income growth in 2016 at around two percent, we should expect the Democrats to win only about 48 percent of the Senate vote in 2016.

Winning a majority of the Senate vote is not a requirement for winning a majority of the contested seats.  In 2004, and most dramatically in 1982, the Republicans managed to win a majority of the seats with a minority of the votes cast.

senate-seats-votes-4

The high “swing ratio” of 2.4 means that a change of one percent in the percentage of votes won translates on average to a 2.4 percent increase in the percentage of seats.  So even fairly small changes in the division of the vote can have much larger effects on the composition of the United States Senate.

I tried some other possible influences like the approval rating of the President and the size of the President’s margin in on-year elections.  I found no “coattails” effect for the Presidential vote either in years when the President is running or years when he not.  Presidential approval does matter, but only in off-year elections, so I did not include it in this discussion about 2016.  That finding is consistent with a conventional view that off-year elections reflect public opinion about the President’s performance in office.

 

Technical Appendix: Modelling Senatorial Elections

In an effort to examine how the 2016 Senatorial elections might turn out, I have been estimating some simple models of Senate elections using aggregate data from 1946 to 2014.[1]   For my dependent variable I have chosen to use the (logit of the) total vote for Senate candidates in the President’s party. This removes party labels from the analysis and treats the two parties symmetrically. I conduct some “regression experiments” of this measure using three types of predictors.

One type represents the electoral setting.  Is this an on-year or off-year election?  And, in on-years, is the incumbent President running for re-election?  Alone these two factors account for over twenty percent of the variance in incumbent support, with President re-election bids having by far the greatest impact.  The results for this model appear in the left-hand column of the table below.  The remaining columns add other possible explanatory factors to the basic political environment.

Senate-Election-Models2

Right away we see that when a President is running for re-election, his co-partisans in the Senate have a much greater chance of winning.  Because these are measured as logits, values below zero correspond to a percentage value below fifty, while positive logits imply values above fifty percent.  Without the President running, the model has a slight negative prediction equal to the constant term.  In Presidential re-election years that negative value turns positive being the sum of the constant (-0.08) and the effect for re-election years (0.14).  By this model the Democrats in the Senate will be short that extra boost that comes from having an incumbent seeking re-election.[2]

One reason the Democrats are optimistic about their chances to retake the Senate in 2016 is that these seats were last contested in the Republican wave election of 2010.  This year those seats will be fought in the context of a Presidential election with its greater visibility and higher turnout.  I have measured this effect by including the vote from the election held six years prior.  In principle, we should expect a negative effect, as “regression toward the mean” sets in.  Republicans perhaps won by unexpectedly larger margins in 2010 so their margins should fall closer to the average this time around.

Adding the prior vote for each Senatorial “class” improves the predictive power of this simple model slightly, but the coefficient itself fails to reach significance.  It has the expected negative sign, however, and will prove much more significant in further reformulations.

The third column adds the effect of presidential approval, a common predictor in models of voting for the House.  For the Senate it turns out to have a more subtle effect.  Presidential approval has the expected positive effect on votes for Senators of the incumbent’s party, but only in off-year elections.  A long literature in political science has examined off-year elections espousing a variety of theories to explain the President’s usual losses.  I generally adhere to the “referendum” school of thought on off-years, that they give the public a chance to express their approval or disapproval of a President mid-way through his term.  That presidential approval matters not in years when a Presidential election is being held reinforces my belief in the referendum explanation for off-year voting.

The last explanatory factor is the year-on-year percent change in real disposable personal income.  Political scientists and economists have included pretty much every economic variable that might affect election outcomes in their models of presidential and congressional voting, but the one factor that often proves significant is personal income. Adding it to the model increased “explained” variance by over ten percent.

Here is a chart showing the expected national vote for the Senate Democrats as a function of the size of the increase in personal income heading into the election. They do get a small positive compensation from having lost the popular vote for these seats six years before.  However, without the re-election boost, even reaching an absurdly high four percent growth in real income would not push the expected vote for the Democrats over the 50 percent line.  In the best years of the Obama Administration, real income growth reached slightly over two percent, which would give the Democratic candidates for Senate about 48 percent of the vote.

pred-econ-incum2

I admit there are many shortcomings to this analysis.  First, I only account for 45 percent of the variance in Senatorial voting, and this only at the national level.  Senatorial campaigns are played out in states, where local forces can exert a major role.  With only 33 or 34 seats up in each election, idiosyncratic factors can swing a few decisive states.

If, as the model predicts, the Democrats should expect to win about 48 percent of the popular vote for Senate, they can nevertheless still win back the Senate.  They might follow the path taken by the Republicans in 2004 and most dramatically in the first off-year election under Ronald Reagan in 1982.  In both those years the Republicans won a majority of the contested Senate seats with a minority of the popular vote.

senate-seats-votes-4

The Rhythm of Senate Elections

From reading media reports of the 2014 election results you might believe the nation has experienced a political change of cataclysmic proportions. Republicans won 23 of the 36 states where a senatorial election was held, enough to give them control of the Senate for the next two years. Yet we need look back only to another strong Republican year, 2010, to see nearly identical results. In that year the Republicans took 24 of the 37 states where elections were held.

Historically the parties’ shares of Senate elections have swung back and forth quite substantially with the last decade appearing unusually unstable.  Here are the results for Senate elections back to 1936:

 

pctdem-trend

The Democrats reached their peak in the 1964 Johnson landslide, though this election merely confirmed the Democrats dominance of the Senate “class” elected six years during the 1958 Eisenhower recession.  Republicans have won about two-thirds of the seats in half-a-dozen elections over the same period of time.  The 2014 result is quite similar to the Republican margins in 2010, 2002, 1980, 1952, and 1946.

If you look carefully at this graph, you’ll see a certain rhythm in these results, one created by the six-year length of a Senatorial term and the power of incumbency.  In fact, if we slide the graph forward six years and superimpose the results, we get this:

senate-rhythm

Now we see how the partisan split in a Senate “class” helps explain the variation from election to election.  Because incumbents have an advantage when it comes time for re-election, the partisan composition of a Senate class tends to repeat at six-year intervals.  The Republican edge in 1946 was replicated six years later when Eisenhower won the White House. Six years after that a major political shift occurs.  The largely Republican class of 1946 and 1952 was replaced with a largely Democratic class during the Democrats sweep of the 1958 off-year elections. The Republicans’ victories in 1980 constituted a similar shift for their party.

Of course the partisanship of the class facing re-election just sets the stage on which each year’s set of electoral forces plays out.  We would expect that factors like broader trends in partisanship and the state of the economy might also influence the outcome of Senate elections.  I turn to those influences in the next article.