Big Ten Geeks: No more BPI, please

If you’ve been watching or reading any of ESPN’s college basketball coverage over the past couple of months, you know what BPI is. That’s because ESPN is shoving this proprietary metric down its audience’s throat. Who should be #1? Let’s check BPI. Is your team on the bubble? Let BPI decide. When SportsCenter loads up a resume of an at-large hopeful, it’s BPI that’s splashed on the screen.

At this point, it’s clear that the Worldwide Leader is going to make us chug BPI until we’re saying it in our sleep. ESPN claims it’s great rating system, better than RPI. It claims that BPI exposes pretenders, and helps you fill out a better bracket.

But I’m calling this what it is—a propaganda campaign.

At face value, BPI sounds like a good system. For one, it was invented by Dean Oliver, the guy who literally wrote the book on advanced basketball statistics. This guy is basketball’s answer to Bill James. If anyone can come up with a great metric, it’s him. What does Oliver have to say about BPI? Let’s read:

There are a number of small details that we have in our methodology to make it reflective of a résumé for a tournament team — these are pretty technical and many people won’t be interested, so we won’t go into detail, but we think they improve how the tool works.

This is the first, and biggest, problem with BPI. Much of BPI remains a mystery. How much? That’s a mystery, too. This alone might be enough to discredit the entire ranking system. Advanced statistics should follow a similar philosophy to all scientific work, which is to say that it must be available for analysis and critical review. I can’t explain all of the reasons why I might like or dislike various parts of BPI, because I don’t know what those parts are.

I’m not entirely sure what the purpose of keeping so much of BPI a secret is, but I think there are two possibilities. The first is we take Oliver at his word: BPI is complicated, and no one will understand it anyways. This is a self-defeating rationale, because ultimately no one should be using a metric they cannot understand. If no one can understand BPI, then no one should use BPI.

The second reason I can come up with is far more likely—ESPN doesn’t want anyone ripping off its metric and putting the numbers somewhere else. It wants BPI available to only its viewers, or its Insider subscribers. And that’s not unreasonable. ESPN is in the business of sports, and it delivers high-quality programming and content precisely because it is a business. We shouldn’t ignore that. But there is a fine line between protecting intellectual property, and credibility. Take Pomeroy’s rankings, for example. Much of the content is only available with an annual subscription, but that doesn’t stop Pomeroy from fully explaining his methodology. It would take a ton of work, and considerable expense, but one could fully recreate Pomeroy’s ratings if one were so inclined.

And yet, no one has done so. I’ll leave the discussion as to why that hasn’t happened for another time, but it’s clear that the risk that someone will rip off BPI is minimal. In short, there’s no good reason for ESPN to hide parts of the BPI calculation.

On to the next problem:

If a team or its opponent is missing one of its most important players (determined by minutes per game) for a contest, that game is less important for ranking the teams compared to games in which both teams are at full strength.

Again, this sounds good, but it’s easy to see why this is flawed. Let’s consider the case of Indiana. Cody Zeller, Victor Oladipo, and Kevin Ferrell have all played about the same number of minutes this season. But it’s clear that one of these things is not like the others. Zeller and Oladipo are on the shortlist for the Naismith Award. Ferrell, while having a fine season for a freshman, is not going to make the Big Ten Third Team. Yet if Ferrell were to miss a game for injury, BPI would weight that same the same as if Oladipo or Zeller went down. We know that’s the wrong approach.

Now, a fair question is whether the wrong approach is nonetheless better than no approach. Kenpom does not adjust games for injury, after all. And frankly, I don’t know the answer to that question, because BPI does not explain itself. All we get from BPI is one anecdotal example from the 2012 Syracuse team, which certainly is not enough to justify this adjustment.

Indeed, this is symptomatic of BPI. The devil is always in the details, and we have none. For example:

Another way that BPI can rank teams differently than Sagarin or Kenpom is counting close games at home versus on the road. In BPI, a close win at home is better than a close loss on the road against the same opponent.

What does Oliver mean by “close?” I think we can all agree that a 5-point win at home is better than a 5-point loss on the road. Home court advantage hovers around four points, after all. But this seems to indicate that a 1-point win at home should be viewed as better than a 1-point loss on the road. But again, we know that’s wrong. So why is BPI placing so much emphasis on the win? Again, we don’t know.

And then there’s the final problem I have with BPI: it’s not clear what it wants to be. By including margin-aware metrics, it looks similar in overall form as Pomeroy. But then there’s this nonsensical wrinkle:

BPI gives marginally decreasing credit for bigger wins, with a 30-point win being only about 20 percent better than a 15-point win, not twice as good, which can happen in other methods.

By capturing blowouts, but not overweighting them, BPI credits the ability of good teams to easily beat poor teams without providing incentive to win by 30 when 20 is a safe margin. By capturing both blowouts and close games in this way, BPI summarizes a team’s résumé for the NCAA tournament well.

“Incentive?” What a bizarre concern for a rating system that has yet to be given any legitimacy by anyone with any power over the fates of college teams. As much as ESPN would like to change this fact, BPI is not the RPI. There’s no indication that the Selection Committee uses it. And thus, there’s no need to provide any incentives. The last thing that Bo Ryan has on his mind when his Badgers are beating Presbyterian by 40 is what this will do to his team’s BPI, Pomeroy Ranking, or any other metric. Now, if the Selection Committee came out and said that it’s going to rely heavily on Pomeroy (which does reward bigger blowouts over smaller ones) in selecting and seeding teams, then maybe Ryan and other coaches might care, and Club Trillion will reduce its membership.

But in the second place, there’s no reason to believe that’s happening. Even if it were, there’s the problem that predictive power exists in those bigger blowouts in the first place.

And this is another reason why BPI doesn’t make sense. If the goal is to encourage the sportsmanship by inventing a new metric that stops rewarding teams after they win by more than 20 points, fine (although I doubt Division I players, as a group, have a self-esteem problem). But if the goal is to come up with a rating system that appropriately ranks teams by how good they are and how likely they are to win more games in the future, this is precisely the wrong approach. It doesn’t matter if it’s unsportsmanlike—if big blowouts predict more future wins than smaller blowouts, then we shouldn’t ignore the former.

All in all, BPI is a metric shrouded in mystery. Of what little we know about it, much of it is wrong. It’s furthermore unclear as to what exactly it’s trying to measure. Yes, it’s better than RPI, but what isn’t? While I’m all for innovation in this space, BPI is a step in the wrong direction.

6 Comments

Your Opinion?
Show Comments (6 Comments)
James B Jr. on 3/5/2013 @ 3:35pm EST Said:

I might take BPI seriously if they actually used it when they are predicting their brackets. MN is a 10 seed in the latest ESPN bracket, and their BPI is 16. HUH!?

Ryan G on 3/5/2013 @ 4:29pm EST Said:

Pretty dumb metric. My biggest thing is ‘pace’. What does that entail? I know kenpom uses it, but if pace is used, a team that plays slow, waiting for a good shot, limiting possessions may be ranked higher/lower (i don’t know which) than a run and gun team. There is no better way to play, maybe one is more exciting, but that shouldn’t affect a teams ranking in terms of which team is better. Also, Illinois is 70th in the BPI and ‘out’ of the tourney, but they are on the verge of being ranked in both the AP and Coaches poll, their RPI is 34 (on espn.com), and they get a 7 or 8 seed in all of the bracket predictions.

aroznowski on 3/5/2013 @ 6:07pm EST Said:

BPI is probably ESPN’s college basketball version of QBR.

Josh Reed, Big Ten Geeks on 3/6/2013 @ 10:19am EST Said:

Ryan,

Pace is just a normalizer. Think of it like you would “at-bats” for baseball. To compute batting average, you divide hits by at-bats. To compute points per possession, you take points and divide it by possessions. Teams are then ranked by how many points they score per possession relative to the points they allow per possession. Thus, the pace aspect is simply meant to account for scoring margin *without* penalizing teams that play faster or slower because, as you rightly point out, there is no “better” way to play.

Aroznowski–couldn’t agree more.

Chris on 3/9/2013 @ 12:23pm EST Said:

I think they’re simply digging too deep and connecting dots that really have no meaning. Your “best player” example was perfect. I also believe there’s a lot of predetermined subject ratings on how good teams are to begin the season, which leads to losing teams remaining high in the rankings since teams are rewarded for not having “bad losses” . But if nearly your whole conference is ranked in the top 60, then repeatedly losing isn’t represented accurately vs other teams in other conference s.
I swear not losing to bad teams is more important than beating good ones according the BPI, why else would a team like Illinois be so low?

RPI May not be perfect but at least it makes sense most of the time

Josh on 3/11/2013 @ 11:31pm EST Said:

While BPI has it problems as well, it’s nowhere near as flawed as the RPI system we still use today, for what reason I can’t figure out.