Is RPI on the way out?

[gogophers]

This weekend's edition of the Wall Street Journal reports that the NCAA is considering getting rid of RPI for seeding the men's basketball team, perhaps as early as next year. It's too soon to say RPI, RIP, but NCAA does seem to be wondering aloud, at least in a context that it actually cares about, March Madness, whether there's a better measure. Most posters on this board think they know the answer. Some quotes from the article, with some familiar refrains:


"As another edition of March Madness looms, the NCAA is accelerating the search for new cutting-edge data tools to better measure which teams should be invited to the annual college-basketball extravaganza and which ones should stay home.

When the selection committee convenes next month to pick teams for this year’s tournament, it will again lean heavily on a 36-year-old formula called Rating Percentage Index. Used since 1981, it attempts to take 351 Division I teams that play vastly different schedules and rank them based on their performance and the quality of their opponents.

Despite numerous tweaks over time, it has faced frequent criticism for being outdated in today’s world of rapidly advancing sports analysis. Unlike more sophisticated metrics, RPI only accounts for whether a team wins, not the margin of victory, and is widely considered to use an overly simplistic method for determining schedule strength.

Changes to this archaic system could be in place as early as next March. To devise alternatives, the NCAA last month brought together some of the most prominent analysts and mathematicians in the college basketball world.

....

There’s no doubt that the different ranking systems can have varied results with real-world implications. The RPI, for instance, ranks West Virginia as the 30th-best team in the country, while Pomeroy has the Mountaineers at No. 3. A composite metric would compensate for such a wide discrepancy.


Before that can happen, the NCAA first must answer a basic question: What is the purpose of this new metric?

“They have to understand what they’re trying to pick for the tournament,” Pomeroy said. “Are you trying to pick the teams that are most accomplished? Or the best teams the moment the tournament starts?”

This is a subtle but significant distinction. The RPI and KPI are results-oriented metrics, which means they reward and penalize teams based on the outcomes of their games, the quality of their opponents and where the games are played. KenPom and Sagarin are predictive metrics, which means they also factor in the scores of games.

The difference explains the disagreement over West Virginia: Results-oriented metrics see a team with six losses, while predictive metrics see that the Mountaineers lost those six games by an average of 3.7 points.

“Having multiple data points in a conversation allows you to minimize and identify outliers,” Pauga said. “The RPI is quantifying results, whereas a metric like KenPom is ranking the quality of teams, and those two don’t necessarily coincide.”

. . . .

In the meeting, the attendees suggested the possibility of devising two composites, one based on results-oriented metrics for picking teams and the other on predictive metrics for seeding them within the bracket. That idea was met with concern that it would only add to the confusion. But it wasn’t ruled out.

. . . .

When it deliberates, the selection committee currently refers to team sheets that show, among other things, a team’s record against opponents ranked in the top 50 and bottom 200 of the RPI. That figure has historically been important in determining who winds up reaching the tournament, which the statisticians say is a problem: RPI is relatively easy for teams to manipulate with strategic scheduling.

Moving forward, these team sheets could be built not around RPI, but a composite metric, which would go a long way toward improving the tournament selection process by more precisely revealing “good wins” and “bad losses.”

[Phaedrus]

Getting a metric is very complex but relatively simple to implement. The trick is getting people to agree on which variables are important and then putting them with proper weighting. THAT is next to impossible.

[Millennium]

Can't happen soon enough. 

There is no need to find the perfect solution straightaway, just something better than what we've had.  I think as soon as people decide that RPI no longer works, then it will be much easier to make tweaks going forward.

Up until now, it seems, the NCAA has been afraid of change, and THAT was the biggest impediment to improving the selection and seeding.

[mikegarrison]

Feb 26, 2017 20:51:32 GMT -5 Phaedrus said:

Getting a metric is very complex but relatively simple to implement. The truck is getting people to agree on which variables are important and then putting them with proper weighting. THAT is next to impossible.

I spent three years working with a group on coming up with a metric for airplane fuel efficiency. The most important things we did were:

1) establish an agreed-upon set of key criteria that we could use to judge any proposed metrics
2) create a set of data we agreed to use to test the metrics with
3) test and test and test

This is why Bofa, for example, always talks about how Pablo is intended to predict the outcome of future matches rather than to reflect the result of past matches. It's because that was his key criterion, and if you choose different key criteria you are probably going to end up with different metrics.

What criteria might be considered?

a) if used for seeding, will lead to fewest upsets
b) selects the "bubble teams" most likely to get to sweet sixteen
c) selects the teams that had the best performance outside of their conference
d) measures relative strength of conferences
e) rewards teams for best past performance over the year
f) selects most consistent performing teams, least likely to have a bad match
g) selects teams most likely to pull off an upset by playing over their usual level of performance

These are just some of the few I can imagine. Many of them are contradictory. In order to find "the right metric" you need to agree in advance what you are hoping it will show you.

[mongoliavolleyball]

The formula should have three parts:

1)(45%) Wins (in season), counting in Losses of 1-3 points as Draws (conference ONLY), counting in Losses as losses (in season greater than 3 points)

2)(45%) Statistics of conference matches, Consistent scoring conference matches, Come back winning conference matches

3)(10%) Opponent played and their rank (pre and season), Number of matches opponent won (pre and season), Points by which opponent won (pre and season)

Just my opinion...

[ay2013]

In terms of volleyball, I don't object to the current iteration of the RPI being an important factor in the committee's decision regarding the bracket. Sure I'd make a couple changes regarding the RPI formula (for example, road wins should count for more than home wins, weighting sets won like they do in the international game, and common opponents records shouldn't be counted more than once), but, even without doing that, in general, the issues with the criteria and the bracket is rarely a product of the raw RPI. I think the committee's over reliance of the RPI in MULTIPLE aspects of the selection criteria is far more of an issue than the RPI itself. Much of the criteria is double counting the RPI, which places a bit too much reliance on the RPI in the overall process. However, in the end, we probably get one or two seeds every year and three or four bids ever year that we disagree with. That's not some huge issue, in the grand scheme of things. IMO, in regards to volleyball, the NCAA could do a couple of things such as placing less emphasis on double counting the RPI in the selection process and seeding 32 rather than 16, to make the bracket a bit more fair rather than ditching the RPI all together.

And to clarify what I mean regarding not counting common opponents records more than once, I'll give a scenario. Say in the preseason UCLA plays Saint Mary's and Stanford play's Saint Mary's. Under the current formula, for UCLA's RPI, Saint Mary's overall record not only counts for UCLA's opponent record (50% of UCLA's RPI), but also it's opponent of opponents record since UCLA will eventually play Stanford in conference. What I'm suggesting is that Saint Mary's overall record only gets counted ONCE for UCLA, as apart of UCLA's opponent record. Essentially, when UCLA plays Stanford, the only teams that get added to UCLA's opponent of opponent record (25% of its RPI) are those teams for which UCLA has NOT played.

[vbman100]

Repeal!

Replace!

[Phaedrus]

Feb 26, 2017 21:42:46 GMT -5 Millennium said:

Can't happen soon enough. 

There is no need to find the perfect solution straightaway, just something better than what we've had.  I think as soon as people decide that RPI no longer works, then it will be much easier to make tweaks going forward.

Up until now, it seems, the NCAA has been afraid of change, and THAT was the biggest impediment to improving the selection and seeding.

Actually, that Is the wrong approach because you need to replace a flawed but workable system. Now you want to replace it with a work in progress. What needs to happen is to create a parallel system to compare against the RPI. Then people can compare. But that still ignores that everyone has different criteria for being eligible for the big dance.

[Not Me]

People will like any system, until their team is left out, or incorrectly ranked.

Well, not everyone. There are people who will simply argue against everything.

[tomclen]

I realize that whatever system is used, smart coaches are going to do their best to 'game' the system.

But, it seems like the current RPI leads to some absurdities in scheduling. Instead of coaches trying to put together a schedule that will best develop their team and players and create exciting matches for fans, too often coaches seem to focus on only getting wins against certain types of teams.

[redbeard2008]

The bigger issue is the NCAA "Travel Ban" - RPI just provides the rationale for implementing it. Get rid of RPI and they'll just use something else to the same effect.

[n00b]

Feb 27, 2017 12:18:24 GMT -5 tomclen said:

I realize that whatever system is used, smart coaches are going to do their best to 'game' the system.

But, it seems like the current RPI leads to some absurdities in scheduling. Instead of coaches trying to put together a schedule that will best develop their team and players and create exciting matches for fans, too often coaches seem to focus on only getting wins against certain types of teams.

I disagree. I think the matches that happen because of RPI are actually very good for NCAA Volleyball.

Why would Pac-12 and Big Ten teams ever schedule Western Kentucky, Lipscomb, Wyoming, etc if there weren't an advantage to it? You can shut those teams out of getting quality matches and they'll never be able to build an at large resume if they don't win their conference. I'm not necessarily saying that it's the NCAA's job to do that, but I think it is a very positive side effect.

[redbeard2008]

Feb 27, 2017 12:44:07 GMT -5 n00b said:

Why would Pac-12 and Big Ten teams ever schedule Western Kentucky, Lipscomb, Wyoming, etc if there weren't an advantage to it? You can shut those teams out of getting quality matches and they'll never be able to build an at large resume if they don't win their conference. I'm not necessarily saying that it's the NCAA's job to do that, but I think it is a very positive side effect.

Have no problem, as long as it doesn't discourage Wisconsin from scheduling Washington...

[n00b]

Feb 27, 2017 12:53:21 GMT -5 redbeard2008 said:

Feb 27, 2017 12:44:07 GMT -5 n00b said:

Why would Pac-12 and Big Ten teams ever schedule Western Kentucky, Lipscomb, Wyoming, etc if there weren't an advantage to it? You can shut those teams out of getting quality matches and they'll never be able to build an at large resume if they don't win their conference. I'm not necessarily saying that it's the NCAA's job to do that, but I think it is a very positive side effect.

Have no problem, as long as it doesn't discourage Wisconsin from scheduling Washington...

And it doesn't or shouldn't.

If a coach simply wants to lose fewer matches, he'll avoid top teams regardless of what ranking system is in place.

[c4ndlelight]

Feb 27, 2017 12:44:07 GMT -5 n00b said:

Feb 27, 2017 12:18:24 GMT -5 tomclen said:

I realize that whatever system is used, smart coaches are going to do their best to 'game' the system.

But, it seems like the current RPI leads to some absurdities in scheduling. Instead of coaches trying to put together a schedule that will best develop their team and players and create exciting matches for fans, too often coaches seem to focus on only getting wins against certain types of teams.

I disagree. I think the matches that happen because of RPI are actually very good for NCAA Volleyball.

Why would Pac-12 and Big Ten teams ever schedule Western Kentucky, Lipscomb, Wyoming, etc if there weren't an advantage to it? You can shut those teams out of getting quality matches and they'll never be able to build an at large resume if they don't win their conference. I'm not necessarily saying that it's the NCAA's job to do that, but I think it is a very positive side effect.

PAC-12 and Big Ten teams would schedule WKU and Lipscomb if a non-RPI SOS metric were implemented because they are good teams. What the RPI does is make it credit a team more for playing Hofstra than LMU and New Mexico St. than Colorado. That is not good for volleyball.