|
Post by n00b on Nov 21, 2014 13:38:59 GMT -5
If you flip Washington to #2 and Wisconsin to #3, that would give an additional correct result. Surprised the algorithm wouldn't do that automatically, although maybe it's still running!? Home court advantage is 161.
|
|
|
Post by vbwheatley on Nov 21, 2014 13:46:21 GMT -5
If you flip Washington to #2 and Wisconsin to #3, that would give an additional correct result. Surprised the algorithm wouldn't do that automatically, although maybe it's still running!? Home court advantage is 161. Got ya - I thought the objective was solely to have the highest # of W/L correct. Taking HCA into account can change that. Thanks.
|
|
|
Post by The Bofa on the Sofa on Nov 21, 2014 14:28:07 GMT -5
If you flip Washington to #2 and Wisconsin to #3, that would give an additional correct result. Surprised the algorithm wouldn't do that automatically, although maybe it's still running!? Washington over Wisconsin is already correct because of the HCA
|
|
|
Post by The Bofa on the Sofa on Nov 21, 2014 14:31:50 GMT -5
i know this is data, but there is something about this that doesn't meet the 'eye' test when used as a ranking (vs. predictor) - like UNLV could it be the variation/consistancy - what I mean is throwing out the best/worst performance (outliers), so that some good teams who had one or two brain fart games don't get penalized - or throw out one best and one worst result If you "ignore" UNLV's win against CSU, then what's the point of this exercise? I've said all along doing this type of approach gives results that look odd. However, I'd actually like to hear the discussion of the actual methodology and foundations outside of "it don't like right"
|
|
|
Post by The Bofa on the Sofa on Nov 21, 2014 14:32:21 GMT -5
Home court advantage is 161. Got ya - I thought the objective was solely to have the highest # of W/L correct. Taking HCA into account can change that. Thanks. I can do that, too. There are a few variations on the theme.
|
|
|
Post by vbwheatley on Nov 21, 2014 14:45:21 GMT -5
Got ya - I thought the objective was solely to have the highest # of W/L correct. Taking HCA into account can change that. Thanks. I can do that, too. There are a few variations on the theme. Understood. This is really good work. I'm sure you've dumped plenty of time into this already, but it would be interesting to see what URS calculated for 2013 after the end of the regular season, and what its predictive value was for the NCAA tournament that followed. Did it predict the Final Four? Would be fascinating.
|
|
|
Post by The Bofa on the Sofa on Nov 21, 2014 14:48:25 GMT -5
I can do that, too. There are a few variations on the theme. Understood. This is really good work. I'm sure you've dumped plenty of time into this already, but it would be interesting to see what URS calculated for 2013 after the end of the regular season, and what its predictive value was for the NCAA tournament that followed. Did it predict the Final Four? Would be fascinating. Unfortunately, the results of a single NCAA tournament are not nearly enough to test the quality of any predictions, even if I were to do it. There are better ways to test the accuracy of predictions than looking at the NCAA tournament. Then again, I wouldn't expect this to do as well as regular Pablo in predicting outcomes in the first place (if it did, I'd already be using something more like it) The reason these things don't pass the eyetest is because it doesn't properly account for upsets. It treats upsets as real events that need to described. Even if you are going to base rankings solely on win/loss, the ELO approach is much more realistic, where it not only allows for upsets, it requires them. ccmanlb sees UNLV up there (btw, congrats to Cindy for her 500th win) and says, no way UNLV could beat those teams around them. But this system doesn't care, all it cares about is who they did beat and who they lost to. So unlike the way we normally think of rankings, this one does not deal with hypothetical matchups. It's a different thought process altogether.
|
|
|
Post by badgerbreath on Nov 21, 2014 16:51:08 GMT -5
Basically, the fact that there is 87% fit, means that only 13% of the results in volleyball are intransitive (A beats B beats C beats A), or run counter to the home-away dichotomy. I guess we tend to pay attention to such events when they happen so they seem more common than that. It is still surprising it is so rare.
|
|
bluepenquin
Hall of Fame
4-Time VolleyTalk Poster of the Year (2019, 2018, 2017, 2016), All-VolleyTalk 1st Team (2021, 2020, 2019, 2018, 2017, 2016)
Posts: 12,440
|
Post by bluepenquin on Nov 21, 2014 17:52:38 GMT -5
i know this is data, but there is something about this that doesn't meet the 'eye' test when used as a ranking (vs. predictor) - like UNLV could it be the variation/consistancy - what I mean is throwing out the best/worst performance (outliers), so that some good teams who had one or two brain fart games don't get penalized - or throw out one best and one worst result If you "ignore" UNLV's win against CSU, then what's the point of this exercise? I've said all along doing this type of approach gives results that look odd. However, I'd actually like to hear the discussion of the actual methodology and foundations outside of "it don't like right"
Unfortunately I wouldn't have clue how to properly discuss your methodology on this than I could on the Pablo Rating (this is way over my head). I can only trust the results - this system is (much) better at 'predicting' who won matches already played and Pablo is very successful at predicting who would win matches going forward. This is based only on wins and losses (like RPI), while Pablo uses points in its calculation. So maybe through examples I could better understand. A team like Oklahoma is significantly worse here than RPI. A thought would be that the win over Texas would do some considerable good in a system like this - where RPI doesn't care at all where the wins come from. Colorado comes in much better vs. RPI - and more improvement than other PAC 12 teams. LSU is a little worse. i bring those 3 teams up to better understand - Oklahoma and Colorado has what seems like a huge upset on their resume, yet both teams go in different directions from RPI. LSU has a huge upset loss - but it doesn't appear to have much if any negative impact compared to RPI - especially when considering that just about every SEC team does worse in this System. So curious - does this better reward the High and Low teams (Oklahoma, LSU, Colorado) or does it reward the more consistent team (very few upset wins and losses). I am guessing that the answer is 'it depends'? Also - at this point in the season, there is just very little movement in RPI and Pablo - especially at the top. Would this system act similarly?
|
|
|
Post by BeachbytheBay on Nov 21, 2014 17:55:45 GMT -5
Understood. This is really good work. I'm sure you've dumped plenty of time into this already, but it would be interesting to see what URS calculated for 2013 after the end of the regular season, and what its predictive value was for the NCAA tournament that followed. Did it predict the Final Four? Would be fascinating. Unfortunately, the results of a single NCAA tournament are not nearly enough to test the quality of any predictions, even if I were to do it. There are better ways to test the accuracy of predictions than looking at the NCAA tournament. Then again, I wouldn't expect this to do as well as regular Pablo in predicting outcomes in the first place (if it did, I'd already be using something more like it) The reason these things don't pass the eyetest is because it doesn't properly account for upsets. It treats upsets as real events that need to described. Even if you are going to base rankings solely on win/loss, the ELO approach is much more realistic, where it not only allows for upsets, it requires them. ccmanlb sees UNLV up there (btw, congrats to Cindy for her 500th win) and says, no way UNLV could beat those teams around them. But this system doesn't care, all it cares about is who they did beat and who they lost to. So unlike the way we normally think of rankings, this one does not deal with hypothetical matchups. It's a different thought process altogether. my point is, a team's best performance and it's worst performance are less likely to be representative of the team - hence if you were to 'throw out' those performances, sure UNLV wouldn't be predicted to beat CSU (but it wouldn't have anyway), so throwing out the outlier results would make the model more accurately conceivably - I'm not a statistician. the point is, the model already has enough samples so that it can afford to throw out a couple for each team to make it more accurate unfortunately we can't have UNLV play all those teams to prove the theory I guess because UNLV lost twice to North at home, that result is one that just really stood out
|
|
stc23
Sophomore
Posts: 195
|
Post by stc23 on Nov 21, 2014 20:10:15 GMT -5
So maybe through examples I could better understand. A team like Oklahoma is significantly worse here than RPI. A thought would be that the win over Texas would do some considerable good in a system like this - where RPI doesn't care at all where the wins come from. Colorado comes in much better vs. RPI - and more improvement than other PAC 12 teams. LSU is a little worse. i bring those 3 teams up to better understand - Oklahoma and Colorado has what seems like a huge upset on their resume, yet both teams go in different directions from RPI. LSU has a huge upset loss - but it doesn't appear to have much if any negative impact compared to RPI - especially when considering that just about every SEC team does worse in this System. So curious - does this better reward the High and Low teams (Oklahoma, LSU, Colorado) or does it reward the more consistent team (very few upset wins and losses). I am guessing that the answer is 'it depends'? I suspect that really big upsets may not have much (or any?) impact in URS -- they're such outliers compared to the rest of a team's results that it's impossible to correctly account for them -- and the reason these three teams are treated differently is that one of the upsets you mentioned wasn't nearly as big as the others...
Oklahoma's 2nd-best win is at home vs Kansas. The range of 5910-5930 places OU just high enough to correctly predict that match (KU 6070 - 161 HCA), and just low enough to also predict their loss @ SMU (5770 + 161). The only way to correctly predict the Texas upset would be to rate OU well above the Longhorns, which would cause misses on several other matches for one or both teams, so URS basically ignores that outlier and tries to fit the rest of their results. At first glance, the only other miss I noticed for OU was the loss to TCU (which is similar to the Texas upset in the opposite direction), while Texas is rated far enough above Florida to have no other misses on their record.
Similarly for LSU, any system is likely to miss on the wins @ UK and @ A&M, as well as the losses to Miss St and Central Arkansas, so their rating is based on the rest of their results. Their range appears to be bounded by the highest/lowest ratings that correctly account for their home loss to UK and win @ UTSA. (Allowing for a small amount of approximation/rounding in the range endpoints.)
Colorado is a different story because the win over UW isn't that big an outlier compared to other wins over Pac-12 opponents. Their rating (and very limited range) is exactly where it needs to be to correctly predict the wins over UW, @ Arizona, and @ UCLA, as well as their losses vs Illinois and @ Oregon.
In other words, OU over Texas and Miss St over LSU were too big of upsets for URS to account for them, but Colorado over UW was not.
Illinois was also mentioned above. Despite having a couple questionable losses that most any system is going to miss (@ NW, VT), their high rating makes sense considering that wins over UNC, Colorado, @ PSU, and @ Nebraska are more than enough to offset the resulting misses against UCLA and Wisconsin (@ Ill).
If the data was updated to account for Illinois' recent loss @ Wisconsin, I think their range would now be capped at 6985.
(Edited to remove OSU from Illinois' list of "questionable" losses. I originally included it based on my expectation that most systems would miss that outcome as well, but it calling it "questionable" is unfair to OSU.)
|
|
|
Post by The Bofa on the Sofa on Nov 21, 2014 20:33:57 GMT -5
Unfortunately, the results of a single NCAA tournament are not nearly enough to test the quality of any predictions, even if I were to do it. There are better ways to test the accuracy of predictions than looking at the NCAA tournament. Then again, I wouldn't expect this to do as well as regular Pablo in predicting outcomes in the first place (if it did, I'd already be using something more like it) The reason these things don't pass the eyetest is because it doesn't properly account for upsets. It treats upsets as real events that need to described. Even if you are going to base rankings solely on win/loss, the ELO approach is much more realistic, where it not only allows for upsets, it requires them. ccmanlb sees UNLV up there (btw, congrats to Cindy for her 500th win) and says, no way UNLV could beat those teams around them. But this system doesn't care, all it cares about is who they did beat and who they lost to. So unlike the way we normally think of rankings, this one does not deal with hypothetical matchups. It's a different thought process altogether. my point is, a team's best performance and it's worst performance are less likely to be representative of the team But you are still viewing this from a probability/hypothetical standpoint. There is no question about "representative of the team" matches. ALL matches are representative of the team. The team is everything that it has done. Nothing gets thrown out. The question is, how can they be adjusted and still account for as much as possible? So if a team loses to teams ranked 100 and 110, but beats teams that are ranked 50, 40 and 30, instead of ranking them somewhere in the middle, the system will rank them so they are better than the team at 30. That is what must happen when you try to get as many wins as possible.
|
|
|
Post by BeachbytheBay on Nov 21, 2014 20:43:27 GMT -5
my point is, a team's best performance and it's worst performance are less likely to be representative of the team But you are still viewing this from a probability/hypothetical standpoint. There is no question about "representative of the team" matches. ALL matches are representative of the team. The team is everything that it has done. Nothing gets thrown out. The question is, how can they be adjusted and still account for as much as possible? So if a team loses to teams ranked 100 and 110, but beats teams that are ranked 50, 40 and 30, instead of ranking them somewhere in the middle, the system will rank them so they are better than the team at 30. That is what must happen when you try to get as many wins as possible. well, yes all matches are representative of a team - but I guess the question is better phrased - do all matches represent the team the most accurately?, 'best win' and 'worst loss' could very possibly (and I'd surmise very likely) be due to an extreme (I'll use food poisoning of a team as an example), so that a ranking system is better (and possibly a prediction system) by discounting extreme cases - and in the situation where a team doesn't have such a wild fluctuation, then throwing out the two samples wouldn't matter much anyway it's my intuitive based theory, on the other hand, it gets rid of information used to also determine 'how good' and 'how bad' a team's floor and ceiling is - I like the idea of throwing out the outliers for ranking, guess I don't like it as much for predicting
|
|
bluepenquin
Hall of Fame
4-Time VolleyTalk Poster of the Year (2019, 2018, 2017, 2016), All-VolleyTalk 1st Team (2021, 2020, 2019, 2018, 2017, 2016)
Posts: 12,440
|
Post by bluepenquin on Nov 21, 2014 22:19:01 GMT -5
So maybe through examples I could better understand. A team like Oklahoma is significantly worse here than RPI. A thought would be that the win over Texas would do some considerable good in a system like this - where RPI doesn't care at all where the wins come from. Colorado comes in much better vs. RPI - and more improvement than other PAC 12 teams. LSU is a little worse. i bring those 3 teams up to better understand - Oklahoma and Colorado has what seems like a huge upset on their resume, yet both teams go in different directions from RPI. LSU has a huge upset loss - but it doesn't appear to have much if any negative impact compared to RPI - especially when considering that just about every SEC team does worse in this System. So curious - does this better reward the High and Low teams (Oklahoma, LSU, Colorado) or does it reward the more consistent team (very few upset wins and losses). I am guessing that the answer is 'it depends'? I suspect that really big upsets may not have much (or any?) impact in URS -- they're such outliers compared to the rest of a team's results that it's impossible to correctly account for them -- and the reason these three teams are treated differently is that one of the upsets you mentioned wasn't nearly as big as the others...
Oklahoma's 2nd-best win is at home vs Kansas. The range of 5910-5930 places OU just high enough to correctly predict that match (KU 6070 - 161 HCA), and just low enough to also predict their loss @ SMU (5770 + 161). The only way to correctly predict the Texas upset would be to rate OU well above the Longhorns, which would cause misses on several other matches for one or both teams, so URS basically ignores that outlier and tries to fit the rest of their results. At first glance, the only other miss I noticed for OU was the loss to TCU (which is similar to the Texas upset in the opposite direction), while Texas is rated far enough above Florida to have no other misses on their record.
Similarly for LSU, any system is likely to miss on the wins @ UK and @ A&M, as well as the losses to Miss St and Central Arkansas, so their rating is based on the rest of their results. Their range appears to be bounded by the highest/lowest ratings that correctly account for their home loss to UK and win @ UTSA. (Allowing for a small amount of approximation/rounding in the range endpoints.)
Colorado is a different story because the win over UW isn't that big an outlier compared to other wins over Pac-12 opponents. Their rating (and very limited range) is exactly where it needs to be to correctly predict the wins over UW, @ Arizona, and @ UCLA, as well as their losses vs Illinois and @ Oregon.
In other words, OU over Texas and Miss St over LSU were too big of upsets for URS to account for them, but Colorado over UW was not.
Illinois was also mentioned above. Despite having a couple questionable losses that most any system is going to miss (@ NW, VT), their high rating makes sense considering that wins over UNC, Colorado, @ PSU, and @ Nebraska are more than enough to offset the resulting misses against UCLA and Wisconsin (@ Ill).
If the data was updated to account for Illinois' recent loss @ Wisconsin, I think their range would now be capped at 6985.
(Edited to remove OSU from Illinois' list of "questionable" losses. I originally included it based on my expectation that most systems would miss that outcome as well, but it calling it "questionable" is unfair to OSU.) This makes sense - thanks for the explanation. I am sure Oklahoma wouldn't appreciates a system that considers their win against Texas as such an outlier that it is treated as if the win never existed. Or LSU's loss to Mississippi State was also such an outlier that the loss is stripped from their record. I realize that these matches are open for consideration at a later date, but as of now...
|
|
|
Post by nuclearbdgr on Nov 21, 2014 23:20:51 GMT -5
This is fantastic. Was the HCA empirically determined? What are the biggest outliers from standard Pablo? How hi would Wisconsin rank if Coach Sheffield remembered his belt?
|
|