Bloodless Coup

The Supreme Court Forecasting Project

The latest issue of Perspectives on Politics (sorry, I don’t know how to link to it) features a symposium on the Supreme Court Forecasting Project. This project is built with the goal of creating a statistical model that can predict the voting behavior of the Supreme Court. This is (according to the goals of political science) a worthy endeavor. But the model itself is yet another example of the atheoretical numbers fetish that marks political science at its worst.

The symposium article by Andrew Martin, Kevin Quinn, Theodore Ruger and Pauline Kim lays out the decision tree that is the core of their model. I suppose one can praise their forthrightness in acknowledging the lack of theory behind their work. They explicitly note “variables were not chosen for explicitly theoretical reasons. Rather, they were chosen based on their availability and plausible relationship to Supreme Court decision making.” But that hardly excuses the fact that they are clearly trying to build a major endeavor upon an unstable foundation. Once you get past that, all you have to do is look at the initial steps in the decision tree to note more problems. The first questions is – was the lower court decision liberal? That opens the door to all sorts of coding problems. Many decisions aren’t obviously “liberal” or “conservative” – far from it. It is no surprise that there are serious reliability problems with that measure. Then the next step in the decision tree deals with which circuit it comes from – but there is no theoretical reason behind the groupings of the circuits. Even the goal – predicting whether the Supremes will affirm or reverse – is simplistic. There is no discussion of the grounds upon or scope of the ruling.

Suzanna Sherry of Vanderbilt University Law School notes these problems, and others, in her scathing response to this article. Much of her response deals with the methodological problems in this study. But Sherry finds some interesting results here that withstand these criticisms. Most interestingly, the model is best at predicting the conservative justices, and does a poor job of predicting the votes of the four justices often referred to as the liberals. Since this is a heavily attitudinal model this suggests that justices like Rehnquist and Scalia make their decisions on the basis of their politics, but the other justices are different. Sherry constructs a strong case in arguing that the “liberal” justices vote on the basis of the law and precedent. Sherry further notes that while the “conservatives” vote their politics and the “liberals” vote the law it appears that Justice O’Connor is doing something else. O’Connor takes a “rather cavalier attitude toward precedent” – even precedents she has joined. She appears to basically take a pragmatic approach to legal questions. Sherry further notes that Justice Kennedy behaves this way on occasion too, but that he does it more rarely and more explicitly.

There are serious problems with this project. But even with its flaws it does strengthen the “attitudinalists” case that some justices on the Supreme Court, perhaps particularly those often labeled “conservative”, decide cases on the basis of their political preferneces.