The Strange Case of 1976

1976 was a horrible year for Senate Republicans; adjusting for that fact makes a slight difference to my 2018 predictions.

Re-examining the results for my original model of Senate elections, it was hard to ignore how poorly the model fit the data for 1976.  Here is a graph of the model’s predicted vote for Senate and the actual vote that shows what an “outlier” 1976 is.  While Truman rallied Senate Democrats in 1948, even that event just hovers on the edge of the statistical “margin of error.”  The Republicans’ failure in the 1976 election after Richard Nixon was forced from office stands truly alone compared to the rest of the postwar elections in my dataset.

If 1976 had been a normal presidential election year, the Republicans’ Senatorial prospects would have looked fairly rosy.  Gerald Ford was running for re-election, real personal income was growing at two percent, and the Democrats were defending seats won in 1970 by a (two-party) margin of 56-44 at the height of the anti-war and anti-Nixon fervor.  That generally pro-Republican climate predicts the GOP should have won nearly 52 percent of the popular vote for Senate.

But, of course, the 1976 election was anything but normal.  It was the first presidential election after the Watergate scandals had forced Richard Nixon from office in disgrace.  Rather than winning the popular vote by the predicted four-point margin, the Republicans could muster only the same share of the vote they won back in 1970, 44 percent.  Though a number of seats changed hands, at the end of the day the Democrats held the same 61-seat Senate majority they did before the 1976 election.

I can adjust statistically for the anomalous 1976 election by adding a “dummy” variable to my model that is one in 1976 and zero otherwise.  Adjusting for 1976 radically improves all aspects of my model.  Its predictive power as measured by adjusted R-squared rises from 0.43 to 0.56, and all the coefficients are more precisely estimated.

Adding this dummy variable implicitly treats Gerald Ford as different from other Presidents running for re-election.  Ford was apparently so compromised by Watergate that his presence at the top of the ticket did not generate the kind of support his fellow Republican candidates for Senate might have expected.  With the 1976 adjustment, the overall effect of Presidential candidacies rises from 2.5 to 3.1 percent, suggesting Ford’s performance was suppressing the estimate for other Presidencies.

Adjusting for 1976 also increases the compensating effect (“regression-toward-the-mean”) of the prior vote for a Senatorial class.  With the suppressing effect of 1976 removed, I now estimate that the Democrats’ lopsided Senate victory in 2012 should be worth about 2.3 percent to the Republicans this November, compared to the 1.9 percent figure I presented earlier with no adjustment for 1976.

For comparison to the chart above, here are the predicted and actual values for the model adjusted for 1976:

Including a dummy variable for 1976 sets its residual to zero and places its predicted value precisely on the line.  The largest positive outliers are now 1978 and two Presidential years, Truman in 1948 and Barack Obama in 2016.

The effects of this modification on the predictions in my earlier article are quite modest.  Without adjusting for 1976, I predicted the Republicans will win 48.1 percent of the popular vote if Trump’s approval rating is at forty and real disposable personal income rises two percent.  With the adjustment that figure rises to 48.4 percent.

The overall conclusion remains that no likely combination of factors predicts that the Republicans will win a majority of the popular vote for Senate this fall.

 

 

Can the Republicans Hold the Senate in 2018?

Historical voting patterns and current economic and political conditions predict the Democrats can win 18 of the 33 Senate seats in contention this fall and take back control of that body.

The departure of Paul Ryan as Speaker provides yet another piece of evidence in favor of the Democrats taking back the House of Representatives this fall.  Faced with this prospect Senate Majority Leader Mitch McConnell and other Republican legislators and strategists argue that the party must now focus its efforts on maintaining its one-vote lead in the Senate.  In this essay I examine the prospects for a Democratic Senate victory as well.

At first glance, the Democrats’ prospects seem quite poor. The current “class” of Senators running for re-election last faced their voters in 2012 when Barack Obama was re-elected President. That helped the Democrats win or retain a number of seats in unlikely places like North Dakota, Missouri, and Montana. Twenty-three Democrats were elected to the Senate that year, compared to eight Republicans and two independents.  With so many more Democratic seats up in 2018, we might expect the GOP to maintain or even expand its slim Senate majority this fall.

However the Democrats themselves held a similar advantage in the 2016 election but failed to win back the Senate. The Republicans needed to defend the 24 seats they won during their “shellacking” of the Democrats in 2010, while just ten Democratic seats were at risk. Nevertheless the Democrats managed to swing only two seats into their column in 2016, suggesting that the relative number of seats at risk may not be a very powerful predictor of the eventual outcome.

One factor in the Democrats’ favor is the absence of Donald Trump at the top of the ballot come November.  Updating the chart from my previous article to include the 2016 election does not change this basic fact:

The Democrats did better than expected in the 2016 Senate elections, winning 53.7% of the popular vote.  That raised the average for the six “open seat” elections, when the President was not standing for re-election, to 48.8% from the figure of 47.9% I reported in 2016.  In general, though, having the President at the top of the ticket adds, on average, four percentage points to the vote for his co-partisans in the Senate. The Republicans are thus starting from behind despite the disparity in seats at risk.

Along with whether the President is standing for re-election, I identified three other factors that have systematically influenced the vote for Senate candidates since World War II:

  • the size of the previous popular vote for the current “class” of Senators facing re-election;
  • the President’s job approval ratings in polls near mid-term elections; and,
  • the year-on-year change in real disposable personal income per-capita.

That framework enables us to examine some possible scenarios for this fall.

There is some evidence that the size of the vote for a Senatorial class does influence how well that class fares six years later. The party advantaged in one election sees a drop in support when facing re-election.  In 2018 that factor helps the Republicans by about 1.8 percent, still not enough to overcome the deficit from not having the President at the top of the ticket.

As I find for House elections, a President’s “job-approval” rating influences the outcomes of Senatorial races.  A ten-point increase in the percent of Americans approving of the President’s performance results in a 1.2 percent increase in support for Senatorial candidates of the President’s party.

On the economic front, the Republicans have repeatedly touted their recent tax cut as providing an impetus to support for their party. As Bloomberg reported after a GOP retreat in February, “With President Donald Trump’s stubbornly low approval ratings and historical trends suggesting they’ll lose seats in the November mid-term elections, party leaders told lawmakers their salvation lies in hammering on the message that the tax cuts passed at the end of last year are putting more money in voters’ pockets.”

My earlier findings confirm that growth in real disposable personal income does have an effect on vote for the President’s co-partisans in the Senate.  In the simulations below, raising personal income growth by one percentage point yields an increase in the predicted vote for Republican candidates this fall by about 1.4 percent.

We can put these findings together and examine the model’s predictions for the Senatorial vote in 2018 given different combinations of presidential job-approval and growth in personal income.


or, visually,

In no likely scenario does my model predict a popular-vote majority for Republican Senate candidates this fall. Which of these scenarios might we see play out in November?

The job-approval figures I use in this analysis come from Gallup, since only that organization has published these ratings as far back as the 1940s.  Like most other pollsters Gallup has reported a slight rebound in Trump’s approval figures over the past couple of months, but they are still running around forty percent.

Gallup has begun reporting weekly ratings this year which explains the much lower variability of its 2018 results.  Also Gallup’s polling shows no systematic partisan bias in either direction but hews closely to the polling “consensus.”  However, Gallup may underestimate approval for President Trump by about 1.6 percentage points because it surveys all adults rather than just registered voters.

Personal income growth, despite the tax cuts, does not provide much reassurance for the Republicans either. Real disposable personal income per capita has shown only modest growth over the past few months, running at about 1.5 percent on an annualized basis.


While there has been a slight downtick since the January peak of 1.6 percent, it seems plausible that real personal income per-capita will show a gain in the neighborhood of 1.5 percent by November.

From the table of predicted vote outcomes, a job-approval rating of 40 and even two percent personal income growth corresponds to a 52-48 split in the popular vote favoring Senate Democrats this fall.  Because there is little “bias” in the translation from votes to seats in the Senate, winning 52 percent of the popular vote translates into a two or three seat margin this fall.


Using the equation shown in that chart, if the Republicans win 48 percent of the popular vote for Senate this fall, they should come away with only fifteen seats (45% of the 33 seats at risk).  A three-seat victory for the Democrats would flip control of the Senate even given the Vice President’s deciding vote.

How Doug Jones Won

Comparing last night’s results for the Special Election in Alabama to prior elections in that state shows the path which brought Doug Jones his unlikely victory.  Like all special elections, the Alabama election unsurprisingly failed to mobilize as many voters as last year’s Presidential contest, but the turnout last night did exceed the 2014 mid-term figure by four percentage points.  Alabamians apparently thought this race was worth the effort.

Jones gained nearly as many votes in this special election as Hillary Clinton polled in November, 2016, despite an electorate 37 percent smaller than for the Presidential election.

Doug Jones expanded the Democratic electorate in the largest Alabama counties when we compare his results to those polled by 2014 Democratic gubernatorial candidate Parker Griffith.  (There was no Democratic candidate for Senator on the ballot that year.)

Jones won 671,151 votes last night, an improvement of 57 percent statewide over Griffith’s 427,787 total three years ago.  The graph shows clearly that Jones’s advantage grew as a function of county size as measured by total turnout.  His best performances were in Madison County, where aerospace center Huntsville is located; Shelby County, which includes suburban Birmingham; and Baldwin County, just east of Mobile.  Moore carried both of the latter counties over Jones but by severely diminished margins compared to Republican performance in prior elections. Jones doubled the Democratic vote in Lee County, home to Auburn University, and made a significant gain in Tuscaloosa where the University of Alabama is located.

Moore’s support, in contrast, was strongest in the smallest counties.  Here I am comparing Moore’s performance last night to the total vote cast in the Republican primary runoff election against Luther Strange late last September.  Moore needed to mobilize sufficient numbers of Strange voters to add to his own totals going into last night’s election.  He failed to do so.

Statewide Moore received about 35 percent more votes than he and Strange together polled in September.  However Moore saw his smallest gains in the largest counties, the opposite of the pattern we saw for Jones.

Running a simple regression of Jones’s lead over Moore against demographic variables shows the dominant power of mobilized African-Americans with smaller effects for the proportions of Hispanics and people with a college degree.

In a county with no blacks, no Hispanics, and no one with a college degree, Moore would beat Jones by 79 points, e.g., 89-10.  The size of the black population played the most important role in determining support for Jones.  His lead expanded by 1.6 percent for every one percent increase in the percentage of blacks.  Hispanics played a less significant role with a coefficient about half the size of that for blacks that barely achieves statistical significance.  The proportion of a county’s residents holding a college degree mattered nearly as much as the size of its black population. These three variables alone account for about ninety-four percent of the variance in Jones’s lead over Moore.

Jones’s victory with a margin of 1.5 percent over Moore suggests that the polls that included cell phone respondents were right on track.  As I wrote yesterday, “Taken together, [my analysis of polling data] suggests that Jones has averaged a 1-2 percent lead in polls taken since the Washington Post story that included calls to cell-phone users.”  Polls limited to landline households, which predicted a Moore victory, were off the mark.

 

Some Observations on the Polling in Alabama

Alabama’s citizens will head to the polls to vote in the most competitive Senate election that state has seen for decades.  Democrat Doug Jones is trying to wrest the state into his party’s column while Republicans rally behind Roy Moore.  Most anyone reading this blog knows about the issues in this race, so I’m going to focus solely on the polls as archived at RealClearPolitics.  Since RCP does not publish data on polling methods, I examined the individual reports for each poll and, when the method used was unclear, contacted the polling agencies directly.

Polling results in this race have shown little convergence as we approach election day.  The Republican dominance in Alabama’s elections has meant that few national polling agencies have paid much attention to the state over the recent elections.  As a result, few national polling organizations have much experience surveying Alabama’s voters.  That has changed a bit as the election achieved national prominence, but still the vast majority of Alabama polls come from organizations will limited track records.  Over at FiveThirtyEight, Harry Enten observes that “Alabama polls have been volatile, this is an off-cycle special election with difficult-to-predict turnout, and there haven’t been many top-quality pollsters surveying the Alabama race.”

“Top-quality” pollsters rely on live interviewers making calls to both landline phones and cell phones.  FiveThirtyEight adds the additional criterion that the polling agency be involved with national organizations like the American Association for Public Opinion Research.  I will limit my analysis to just whether calls were made to a sample of cell phone owners.  As it turns out, this factor alone has a profound effect on a poll’s estimated margin between the candidates.

Here is a list of the available polls based on their method of interview.

Only one poll among those that included interviews with respondents via cell phone shows a lead for Moore; in contrast, only one of the landline-only polls puts Jones ahead.  The “swing” is quite substantial, about an eight-point differential based on the method used.

This difference arises from the much higher usage of cell phones by younger respondents who prefer Jones in most polls.  For instance, in today’s poll from Fox News likely voters under 45 year of age preferred Jones 59% to 28%, while voters above that age preferred Jones by only a one point margin, 45% to 44%.

I also modeled the difference in support between Jones and Moore using my standard predictors, time left before the election, and dummy variables for polling methods.  I also added a dummy which is coded one beginning on November 9th when the story about Moore’s alleged molestation of young girls was released in the Washington Post.  The variable measuring proximity to the election proved statistically insignificant, leaving just three dummies, whether the pollster made calls to cell phones, whether live interviewers were used, and whether the story had broken in the Post.

In polls taken before the publication of the molestation story, Jones trailed Moore by an average of eleven points.  Since then Jones has seen an average gain of six points, not enough on its own to return the race to even.  However polls that interviewed respondents via cell phones show a slightly larger difference of nearly seven points in Jones’s favor.  Taken together, these results suggest that Jones has averaged a 1-2 percent lead in polls taken since the Washington Post story that included calls to cell-phone users. (Update: Jones’s margin of victory over Moore was 1.5 percent statewide, right in line with this prediction.)

I also included a term for whether live interviewers were used.  Since all polls that include cell phone owners must use live interviewers by law, this remaining group represents organizations that polled only landline owners with human interviewers.  I find a small, though statistically insignificant (p<0.17) positive effect on Jones’s support from people surveyed by live interviewers.  It is hard to interpret what this effect might signify.  It could represent an unwillingness among Moore’s supporters to admit their intentions to a human interviewer but have no such hesitation when the interview is conducted by a robot.  If so, we might attribute some of the difference between cell phone and landline results to use of human interviewers in polls that include cell users.

 

A Simple Model of Senate Elections

Without a President seeking re-election at the top of the ticket, the Democrats face a substantial challenge if they wish to win back the Senate in 2016.

While the Presidential race gets all the media and pundit attention, the battle for control of the Senate also looms large in this election year.  Republicans enter the election holding 54 of the 100 Senate seats, so a net Republican loss of just five seats would put the Senate back in Democratic hands.  The Democrats have the advantage that many more Republican seats, twenty-four, are at risk in the 2016 election compared to only ten held by Democrats.  This lopsided margin reflects the result of the 2010 off-year election when Republicans picked up six seats from the Democrats.  In principle, some of those Republican senators may be more vulnerable in a Presidential year with higher turnouts and more visibility.  The Democrats certainly believe they can retake the Senate this November.  The Party aggressively recruited candidates for the Senate elections and had secured bids from all but one of its top-tier candidate selections by early October of 2015.

The Democrats also have the advantage that the party of the incumbent President usually wins a slim majority of the Senate vote in “on-year” elections when a Presidential election also takes place but loses in “off-year” elections.

on-off-year

Unfortunately for the Democrats the relationship between Senate electoral success and type of election is not so simple.  If I divide up on-year elections into ones when the President ran for re-election and ones when, like the upcoming election, he did not, a very different pattern emerges.  The President’s party fared substantially worse in the five open-seat elections since 1946 than it did in elections with the President at the top of the ticket.  While open-seat years gave the President’s party a one-percent boost compared to off-years, that difference is not statistically significant.  What matters is whether the President is running or not.

on-off-open-year2

The Democrats’ optimism is also based on the much larger number of Republican seats at risk in 2016.  I find some support for the notion that a Senate “Class” with a comparatively lopsided division of the vote in one election becomes more competitive six years later.  Statisticians call this phenomenon “regression toward the mean,” where observations that were outliers at one time show more average scores when measured again. But this effect is weak, and the division of the Senate vote in 2010, 53-47 percent Republican, was not as lopsided as the margin in terms of seats, 65-35 percent Republican.  All told the estimated “rebound” effect given the Republican 2010 landslide is just 0.6%, raising the expected Democratic vote from 47.1% to 47.7%.

Where else might the Democrats gain some relief?  Perhaps the generally positive state of the economy might provide some help.  Political scientists and economists have tested many different measures of economic conditions in models of voting for President and Congress.  One simple measure that has consistently proven significant is the change in personal income, and that proves true for Senate elections as well.

pred-econ-incum2

This chart adds the effects of the year-on-year percent change in real disposable personal income to our simple model.  While the effect of rising incomes is positive and statistically significant, it alone cannot overcome the substantial deficit facing the Democrats in an open-seat election year.  In the six years of the Obama Administration, personal income rose by at most 2.2 percent in a single year, 2012.  With a likely figure for annual income growth in 2016 at around two percent, we should expect the Democrats to win only about 48 percent of the Senate vote in 2016.

Winning a majority of the Senate vote is not a requirement for winning a majority of the contested seats.  In 2004, and most dramatically in 1982, the Republicans managed to win a majority of the seats with a minority of the votes cast.

senate-seats-votes-4

The high “swing ratio” of 2.4 means that a change of one percent in the percentage of votes won translates on average to a 2.4 percent increase in the percentage of seats.  So even fairly small changes in the division of the vote can have much larger effects on the composition of the United States Senate.

I tried some other possible influences like the approval rating of the President and the size of the President’s margin in on-year elections.  I found no “coattails” effect for the Presidential vote either in years when the President is running or years when he not.  Presidential approval does matter, but only in off-year elections, so I did not include it in this discussion about 2016.  That finding is consistent with a conventional view that off-year elections reflect public opinion about the President’s performance in office.

 

Technical Appendix: Modelling Senatorial Elections

In an effort to examine how the 2016 Senatorial elections might turn out, I have been estimating some simple models of Senate elections using aggregate data from 1946 to 2014.[1]   For my dependent variable I have chosen to use the (logit of the) total vote for Senate candidates in the President’s party. This removes party labels from the analysis and treats the two parties symmetrically. I conduct some “regression experiments” of this measure using three types of predictors.

One type represents the electoral setting.  Is this an on-year or off-year election?  And, in on-years, is the incumbent President running for re-election?  Alone these two factors account for over twenty percent of the variance in incumbent support, with President re-election bids having by far the greatest impact.  The results for this model appear in the left-hand column of the table below.  The remaining columns add other possible explanatory factors to the basic political environment.

Senate-Election-Models2

Right away we see that when a President is running for re-election, his co-partisans in the Senate have a much greater chance of winning.  Because these are measured as logits, values below zero correspond to a percentage value below fifty, while positive logits imply values above fifty percent.  Without the President running, the model has a slight negative prediction equal to the constant term.  In Presidential re-election years that negative value turns positive being the sum of the constant (-0.08) and the effect for re-election years (0.14).  By this model the Democrats in the Senate will be short that extra boost that comes from having an incumbent seeking re-election.[2]

One reason the Democrats are optimistic about their chances to retake the Senate in 2016 is that these seats were last contested in the Republican wave election of 2010.  This year those seats will be fought in the context of a Presidential election with its greater visibility and higher turnout.  I have measured this effect by including the vote from the election held six years prior.  In principle, we should expect a negative effect, as “regression toward the mean” sets in.  Republicans perhaps won by unexpectedly larger margins in 2010 so their margins should fall closer to the average this time around.

Adding the prior vote for each Senatorial “class” improves the predictive power of this simple model slightly, but the coefficient itself fails to reach significance.  It has the expected negative sign, however, and will prove much more significant in further reformulations.

The third column adds the effect of presidential approval, a common predictor in models of voting for the House.  For the Senate it turns out to have a more subtle effect.  Presidential approval has the expected positive effect on votes for Senators of the incumbent’s party, but only in off-year elections.  A long literature in political science has examined off-year elections espousing a variety of theories to explain the President’s usual losses.  I generally adhere to the “referendum” school of thought on off-years, that they give the public a chance to express their approval or disapproval of a President mid-way through his term.  That presidential approval matters not in years when a Presidential election is being held reinforces my belief in the referendum explanation for off-year voting.

The last explanatory factor is the year-on-year percent change in real disposable personal income.  Political scientists and economists have included pretty much every economic variable that might affect election outcomes in their models of presidential and congressional voting, but the one factor that often proves significant is personal income. Adding it to the model increased “explained” variance by over ten percent.

Here is a chart showing the expected national vote for the Senate Democrats as a function of the size of the increase in personal income heading into the election. They do get a small positive compensation from having lost the popular vote for these seats six years before.  However, without the re-election boost, even reaching an absurdly high four percent growth in real income would not push the expected vote for the Democrats over the 50 percent line.  In the best years of the Obama Administration, real income growth reached slightly over two percent, which would give the Democratic candidates for Senate about 48 percent of the vote.

pred-econ-incum2

I admit there are many shortcomings to this analysis.  First, I only account for 45 percent of the variance in Senatorial voting, and this only at the national level.  Senatorial campaigns are played out in states, where local forces can exert a major role.  With only 33 or 34 seats up in each election, idiosyncratic factors can swing a few decisive states.

If, as the model predicts, the Democrats should expect to win about 48 percent of the popular vote for Senate, they can nevertheless still win back the Senate.  They might follow the path taken by the Republicans in 2004 and most dramatically in the first off-year election under Ronald Reagan in 1982.  In both those years the Republicans won a majority of the contested Senate seats with a minority of the popular vote.

senate-seats-votes-4

The Rhythm of Senate Elections

From reading media reports of the 2014 election results you might believe the nation has experienced a political change of cataclysmic proportions. Republicans won 23 of the 36 states where a senatorial election was held, enough to give them control of the Senate for the next two years. Yet we need look back only to another strong Republican year, 2010, to see nearly identical results. In that year the Republicans took 24 of the 37 states where elections were held.

Historically the parties’ shares of Senate elections have swung back and forth quite substantially with the last decade appearing unusually unstable.  Here are the results for Senate elections back to 1936:

 

pctdem-trend

The Democrats reached their peak in the 1964 Johnson landslide, though this election merely confirmed the Democrats dominance of the Senate “class” elected six years during the 1958 Eisenhower recession.  Republicans have won about two-thirds of the seats in half-a-dozen elections over the same period of time.  The 2014 result is quite similar to the Republican margins in 2010, 2002, 1980, 1952, and 1946.

If you look carefully at this graph, you’ll see a certain rhythm in these results, one created by the six-year length of a Senatorial term and the power of incumbency.  In fact, if we slide the graph forward six years and superimpose the results, we get this:

senate-rhythm

Now we see how the partisan split in a Senate “class” helps explain the variation from election to election.  Because incumbents have an advantage when it comes time for re-election, the partisan composition of a Senate class tends to repeat at six-year intervals.  The Republican edge in 1946 was replicated six years later when Eisenhower won the White House. Six years after that a major political shift occurs.  The largely Republican class of 1946 and 1952 was replaced with a largely Democratic class during the Democrats sweep of the 1958 off-year elections. The Republicans’ victories in 1980 constituted a similar shift for their party.

Of course the partisanship of the class facing re-election just sets the stage on which each year’s set of electoral forces plays out.  We would expect that factors like broader trends in partisanship and the state of the economy might also influence the outcome of Senate elections.  I turn to those influences in the next article.