January 04, 2005

The Supreme Court Forecasting Project

The latest issue of Perspectives on Politics (sorry, I don’t know how to link to it) features a symposium on the Supreme Court Forecasting Project. This project is built with the goal of creating a statistical model that can predict the voting behavior of the Supreme Court. This is (according to the goals of political science) a worthy endeavor. But the model itself is yet another example of the atheoretical numbers fetish that marks political science at its worst.

The symposium article by Andrew Martin, Kevin Quinn, Theodore Ruger and Pauline Kim lays out the decision tree that is the core of their model. I suppose one can praise their forthrightness in acknowledging the lack of theory behind their work. They explicitly note “variables were not chosen for explicitly theoretical reasons. Rather, they were chosen based on their availability and plausible relationship to Supreme Court decision making.” But that hardly excuses the fact that they are clearly trying to build a major endeavor upon an unstable foundation. Once you get past that, all you have to do is look at the initial steps in the decision tree to note more problems. The first questions is – was the lower court decision liberal? That opens the door to all sorts of coding problems. Many decisions aren’t obviously “liberal” or “conservative” – far from it. It is no surprise that there are serious reliability problems with that measure. Then the next step in the decision tree deals with which circuit it comes from – but there is no theoretical reason behind the groupings of the circuits. Even the goal – predicting whether the Supremes will affirm or reverse – is simplistic. There is no discussion of the grounds upon or scope of the ruling.

Suzanna Sherry of Vanderbilt University Law School notes these problems, and others, in her scathing response to this article. Much of her response deals with the methodological problems in this study. But Sherry finds some interesting results here that withstand these criticisms. Most interestingly, the model is best at predicting the conservative justices, and does a poor job of predicting the votes of the four justices often referred to as the liberals. Since this is a heavily attitudinal model this suggests that justices like Rehnquist and Scalia make their decisions on the basis of their politics, but the other justices are different. Sherry constructs a strong case in arguing that the “liberal” justices vote on the basis of the law and precedent. Sherry further notes that while the “conservatives” vote their politics and the “liberals” vote the law it appears that Justice O’Connor is doing something else. O’Connor takes a “rather cavalier attitude toward precedent” – even precedents she has joined. She appears to basically take a pragmatic approach to legal questions. Sherry further notes that Justice Kennedy behaves this way on occasion too, but that he does it more rarely and more explicitly.

There are serious problems with this project. But even with its flaws it does strengthen the “attitudinalists” case that some justices on the Supreme Court, perhaps particularly those often labeled “conservative”, decide cases on the basis of their political preferneces.

Posted by armand at January 4, 2005 03:23 PM | TrackBack | Posted to Law and the Courts


which, perhaps one need not add, makes the conservative justices the activists of the current Court. of course, we all knew that already, given that they have overturned more legislation than any other modern Court.

Posted by: joshua at January 5, 2005 11:05 AM | PERMALINK

This is the part of political science that I just don't understand. What is the point of such a model? Just it's existence seems to argue that SC decisions are not based on anything like truth, justice or "doing the right thing", but instead are predictable based on partisan issues. That's certainly a slipperly slope to start down. In any event, as Armand notes, what is the point? Why build a model, if not as part of a theoretical understanding of some event or action? I continue to be utterly unimpressed by what political scientists who study American politics do. This is interesting? This is fascinating?

Posted by: baltar at January 5, 2005 01:40 PM | PERMALINK

The point, as I take it, is two-fold: first, to test the case that the attitudinalists make that justices vote their own politics and ideology (this is an on-going debate in the poli sci side of literature on the Court, but support for the attitudinalists has been increasing), and, secondly, to allow for better projections of court decisions. Lots of political scientists don't give a hoot about predictions, but these people do. In fact part of the study is a comparison of their model's predictions versus those of "experts" (the Linda Greenhouses of the world).

And I should note that it's not ENTIRELY atheoretical. It's just that important sections of it are (these folks are obviously grasping at available data) and the rest includes sloppy measures.

Posted by: Armand at January 5, 2005 01:52 PM | PERMALINK
Post a comment

Remember personal info?