Meet Leo, Our Senate Model

How does Leo interpret a poll?#

Leo says We focus on the margin between two major candidates, taking steps to make different polls directly comparable. We tweak polls that count registered voters instead of likely ones. We make further adjustments depending on who conducted the poll.

Here’s why Polls of registered voters tend to be more favorable to Democrats than polls of likely voters, especially in midterm elections. But different pollsters make different decisions about how to define likely voters and how to reach them. These decisions tilt their polls toward Democrats or Republicans in systematic, and predictable, ways. Systematic differences among pollsters are sometimes called “house effects.”

What it means in practice Public Policy Polling, a Democratic-leaning firm, and Harper Polling, a Republican-leaning firm, are among the most frequent pollsters. If they were to conduct polls of the same race on the same days in 2014, we would expect PPP’s result to be about 2 points more favorable to the Democrat. To make apples-to-apples comparisons between polls, we adjust the margins accordingly, using information from all of the polls we’ve observed so far this cycle.

How does Leo combine many polls?#

Leo says After adjusting the polls, we take a weighted average for each race, giving more weight to polls with a larger sample size and more recent polls (with a poll’s date being especially important the closer we get to Election Day). We also give more weight to a poll when we are more certain about its pollster’s house effect.

Here’s why Many polls, like those by PPP and Harper, are based on automated telephone surveys, which are legally prohibited from calling mobile phones and thus fail to capture some voters’ views. They nonetheless contain meaningful information about public opinion, especially when multiple polls are combined.

We’ll let you in on a secret about Senate forecasting. By November, if you simply averaged the polls — no adjustments, no weights, just a simple average — your model would do pretty well. You could have correctly predicted 138 out of the 149 Senate races with polls over the last five election cycles. That success rate — 93 percent — is nearly as good as the best public models. (Thanks, polling community!)

Weighting the polls over time allows the dynamics of a race to change more quickly than a simple average.

What it means in practice By giving greater weight to recent polls, Leo has downgraded Senator Kay Hagan’s chance of winning re-election in North Carolina. The race appears to be more competitive than polls suggested last summer, when she had a clear lead.

What does Leo count besides head-to-head polls?#

Leo says For incumbents running for re-election, we consider their approval ratings. We also consider each candidate’s political experience; money raised; the state’s most recent presidential result; national polls on the public’s mood; and whether the election happens in a midterm or presidential year. The weight for each variable depends on its record in predicting the past 20 years of Senate elections. We update the weights every day. (We do this using a linear regression, allowing larger variances for states with a history of voting for different parties in Senate and presidential elections.)

Here’s why In 90 percent of races, these six variables — listed above roughly in descending order of importance — let you predict a candidate’s share of the vote within 10 points. Ten points is still a lot — expert observers could do that without any data at all — but it’s a good place to start.

What it means in practice We measure political experience based on a candidate’s highest elected office, on a scale that ranges from city council to the Senate itself. That means we won’t capture the experience of a candidate like Rick Weiland, a former aide to Tom Daschle, who is running in South Dakota. Still, 75 percent of open races — those without an incumbent running — are won by the candidate who has been elected to a higher office. (Incumbents win 88 percent of the races they enter.)

We use generic-ballot polls — questions like “If the 2014 election for U.S. House of Representatives were being held today, would you vote for the Republican candidate or the Democratic candidate in your district?” — to capture the national mood. We do not use presidential approval ratings, which have a slightly weaker predictive record than generic-ballot questions, after accounting for the other variables.

How does it combine polls with the other variables?#

Leo says The more polls there are, the more weight we put on the polling average. But when polls are sparse or when the election is still months away, we stick closer to the background information, like political experience. (Formally speaking, we use the background model as a Bayesian prior.)

Here’s why History guides us here. The results of previous elections dictate how much weight to give background information and polls at different points in the cycle. At the beginning of an election year, when the polls can be unstable or misleading, roughly two-thirds of the forecast comes from the background model. By Election Day, this declines to less than 15 percent, depending on how many polls have been conducted.

What it means in practice Including the background information currently pulls the model’s estimates toward the Democrats in North Carolina, where the Republicans running are relatively inexperienced, and toward the Republicans in Louisiana, which voted for Mitt Romney in 2012.

Hold on. We don’t even know who the candidates will end up being in some of these races.#

Leo says Every day, Leo assigns each possible general-election matchup a probability based on primary polling and fund-raising data.

Here’s why In several states the general election forecast depends on which two candidates meet in the general election. If we want to make a good forecast, we shouldn’t assume the strongest candidate will always win. This fact holds for incumbents as well: while nearly all incumbents win their primary races, it’s not guaranteed. (For example, Thad Cochran, Republican of Mississippi, came within a few thousand votes of losing his primary in June.) Data on polling and fund-raising provides us with a rough idea on the likelihood of different matchups.

What it means in practice One of the primary elections most likely to affect our general-election predictions is in Alaska (on Aug. 19). The Republicans’ chances will increase if the strongest general-election candidate wins this race. Based on polls, fund-raising and experience, our model judges the strongest potential nominee to be Daniel Sullivan or Mead Treadwell.

How does Leo combine its predictions to get an overall answer?#

Leo says We don’t think the races are independent. If the economy starts booming, it will probably help Democrats everywhere. If President Obama bungles an international crisis, Republicans everywhere could benefit. Even on Election Day, our model assumes the races will be correlated to some extent: The pollsters will tend to miss consistently in one direction or the other across the different races.

Here’s why In historical data, specifically Senate elections since 2004, the races do not appear to move in independent ways. As late as August 2012, for example, most forecasters were projecting that Republicans would win around 13 races. They won eight.

What it means in practice Letting the errors be correlated is crucial for making a good projection of the total number of seats a party will win. Because misses on the state level will tend to be in one direction or the other, we should be more uncertain about how many seats each party will control. (If the races were independent, our errors would be more likely to cancel each other out.)

How well does Leo do?#

Leo says The best way to judge Leo is to test whether it is well calibrated. That is, when our forecast says a Republican has an 80 percent chance of winning, as the forecast currently does in Colorado, do Republicans actually win four out of every five such races? We can’t run a laboratory experiment that repeats the Colorado Senate race five times, of course. But we can look back at past races in which Leo would have assigned odds of 80 percent and see how often the favorite actually won. Sure enough, the track record is solid.

We know, however, that most readers will not use calibration to judge Leo. People will instead say it was “right” when it gave the eventual winner at least a 50 percent chance of victory. So here’s a less-sophisticated answer to the question: In November 2012, using only information known then, Leo would have “missed” one Senate race — in North Dakota. Our model would have given Rick Berg, the Republican candidate, a 61 percent chance of winning. Its best estimate was that Mr. Berg would beat Heidi Heitkamp by one percentage point. The reverse was true. In all, Leo would have “missed” – we can’t resist the quotation marks – on four of its 172 Election Day predictions over the past five election cycles:

But we want to emphasize: A model in which every 51-percent favorite wins is not much better than a coin that always comes up heads.

How well calibrated is Leo earlier in a campaign?#

Leo says Nice use of “calibrated.” Of course, it’s much easier to predict the winner of an election on Election Day than it is in April. Though Leo’s predictions are just as well calibrated early in a campaign — with history suggesting that 60 percent, for instance, really means 60 percent — our forecasts are far more precise by November. Late in an election cycle, you should expect to see fewer tossups. That’s because the model incorporates the uncertainty inherent in political campaigns, recognizing that many things could happen between now and the election.

Since 2004, Leo’s biggest shift between April and November would have come in the 2006 race in which Senator George Allen, Republican of Virginia, lost to Jim Webb. (In August of that year, Mr. Allen used the word “macaca” to describe a Democractic campaign volunteer.) In April, Leo’s odds would have suggested it was unlikely that Kay Hagan would defeat incumbent Senator Elizabeth Dole in North Carolina in 2008, or that Senator Russ Feingold would be defeated in Wisconsin in 2010. In April 2010, Leo would have given Mr. Feingold a 92 percent chance of winning his race.

Do you want even more detail? Do you disagree with some of Leo’s choices?#

If you have any questions or comments, please contact Leo on Twitter: Tweet to @nytleo

You can also find all of the data and code behind Leo on GitHub.

Credits#

Historical results from the Open Elections project, CQ Press Voting and Elections Collection, Dave Leip’s Atlas of U.S. Presidential Elections, and the Federal Election Commission. Fund-raising data from the Federal Election Commission. Biographical data from Project Vote Smart and candidate websites. Polls aggregated by Pollster, the Roper Center for Public Opinion Research, the U.S. Officials’ Job Approval Ratings project, Polling Report, Gallup, Talking Points Memo, The Argo Journal, Real Clear Politics and FiveThirtyEight.

Leo owes an intellectual debt to earlier models, including those created by political scientists and especially the FiveThirtyEight model, which popularized ideas about adjusting polls, combining polls with other information and national swings.