|
|
Post by chipNdink on Sept 20, 2006 2:22:55 GMT -5
In your FAQ:
... By breaking the analysis down to a game by game basis, BCR effectively weights close matches more heavily than non-close matches. Thus, a 5 game match is given 5/3 the impact of a 3 game match. Pablo doesn’t do this. Apparently, weighting the closer matches more heavily is a good thing to do. I am trying to think of a way to do this in Pablo rankings to see if it helps. This may be a way to improve Pablo even more.
My suggestion:
You say you use the points percentages in a match to estimate the point probabilities. There is an error associated with this estimate. The greater the actual number of points used, the lower the error in making this estimate. Obviously, a 5 game match will have more points than a 3 game match; and thus gives a better estimate for the "true" point probability with a lower associated error for that estimate. You can incorporate this "error" into your model by weighting the estimates by some factor which depends upon the actual number of points in the match.
|
|
|
|
Post by Ye Olde Dawg on Sept 20, 2006 3:44:03 GMT -5
My thought on this has been that the closer matches are actually more telling indicators of what a crew is like. If ye play a close match, odds are that you're playing a crew close to ye in ability. That tells ye much more than you'd learn if those two close crews either keel hauled or lost to a much worse or better common opponent.
I'm slowly putting together a way of building a similar ranking system just to see what goes into it, and I've got to say I have a greater appreciation of what p-dub has done having started the effort myself. I'm personally going to steer clear of trying to give "Pablo" any advice for a while.
|
|
|
|
Post by The Bofa on the Sofa on Sept 20, 2006 8:40:29 GMT -5
In your FAQ: ... By breaking the analysis down to a game by game basis, BCR effectively weights close matches more heavily than non-close matches. Thus, a 5 game match is given 5/3 the impact of a 3 game match. Pablo doesn’t do this. Apparently, weighting the closer matches more heavily is a good thing to do. I am trying to think of a way to do this in Pablo rankings to see if it helps. This may be a way to improve Pablo even more. My suggestion: ye say ye use the points percentages in a match to estimate the point probabilities. There is an error associated with this estimate. The greater the actual number of points used, the lower the error in making this estimate. Obviously, a 5 game match will have more points than a 3 game match; and thus gives a better estimate for the "true" point probability with a lower associated error for that estimate. ye can incorporate this "error" into your model by weighting the estimates by some factor which depends upon the actual number of points in the match. I have tried in the past to figure out ways to try to reproduce the BCR weighting scheme (which was a natural scheme), but I have never gotten it to work. However, y0ur suggestion is not something I have thought of, and it is worth thinking about. |
|
|
|
Post by chipNdink on Sept 20, 2006 10:26:31 GMT -5
If the errors for the point percentages are calculated correctly (not sure, but probably Poisson distribution type errors), and these are incorporated correctly as inverse weights in your minimization scheme, then I would presume you could get rid of the additional parts dealing with "type of matches, i.e. 3 blowout, 4 close, 5 close, etc." and win/loss info, as all this information should automatically be weighted into the errors.
As a test of the model's predictive ability, you can do more than just predict win/loss, you should also be able to predict point percentages for the match (and even an error for that prediction). Calculate the squares of the differences between the expected point percentage and the actual point percentage, properly weighted by the errors (not sure now whether you should combine somehow the error of your predicted value and the error of the actual value?).
The "better" model should minimize these predicted distributions.
|
|
|
|
Post by chipNdink on Sept 20, 2006 10:32:28 GMT -5
My thought on this has been that the closer matches are actually more telling indicators of what a crew is like. If ye play a close match, odds are that you're playing a crew close to ye in ability. That tells ye much more than you'd learn if those two close crews either keel hauled or lost to a much worse or better common opponent. ... Yes, besides the natural increased statistical weight of more points (more rallies) that a closer match should provide, it may be that a team's "true" point scoring ability/probability is actually "different" when playing "different" teams. This however certainly complicates the model considerably, and there doesn't seem to be a "natural" and "simple" way of incorporating this info. |
|
|
|
Post by The Bofa on the Sofa on Sept 20, 2006 10:44:13 GMT -5
If the errors for the point percentages are calculated correctly (not sure, but probably Poisson distribution type errors), This is well beyond Poisson and normal distributions work perfectly fine It is described somewhere in the FAQ that I do this already. I quit using the 3/4/5 breakdown and just use point pct That is something I have always wanted to try, but the general uncertainty of it has been what is getting in my way. In principle, I can estimate an uncertainty on every match point pct. It would be based on the number of points. I will play with it a bit, although as I think about it, I'm not sure the weighting differences will be large. But I will see. |
|
|
|
Post by chipNdink on Sept 20, 2006 12:17:31 GMT -5
... In principle, I can estimate an uncertainty on every match point pct. It would be based on the number of points. I will play with it a bit, although as I think about it, I'm not sure the weighting differences will be large. But I will see. In a typical blowout (say 30-20, 30-20, 30-20), you have a total of 150 points. In a close 5 setter (30-28, 30-28, 28-30, 28-30, 15-13), you have a total of 260 points. That's a weighting difference of 260/150 = 1.73, which means the close 5 setter would count almost double the importance of the typical blowout. |
|
|
|
Post by Ye Olde Dawg on Sept 20, 2006 14:01:23 GMT -5
I think its much simpler than that. How much predictive value is there in a match between two very uneven opponents? When you get right down to it, what's the difference between 30-18 and 30-20? Both teams will have realized that the team on the low end isn't going to win, and for a variety of reasons that have nothing to do with ability either team may allow a few more points.
A team's strength is best measured against closely-matched opposition, not against someone they can beat easily.
As for a natural and simple way of incorporating this into the model, isn't that what we've been talking about as a side effect of what BCR does?
|
|
|
|
Post by chipNdink on Sept 20, 2006 14:54:50 GMT -5
I think its much simpler than that. How much predictive value is there in a match between two very uneven opponents? When ye get right down to it, what's the difference between 30-18 and 30-20? Both teams will have realized that the crew on the low end isn't going to win, and for a variety of reasons that have nothing to do with ability either crew may allow a few more points. A team's strength is best measured against closely-matched opposition, not against someone they can keel haul easily. As for a natural and simple way of incorporating this into the model, isn't that what we've been talking about as a side effect of what BCR does? No, what the BCR does (weighting by the number of games in a match) is basically a quick and dirty approximation of what I suggest (which is weighting by the total number of points in the match) in order to include the greater statistical significance of more data that is available from more games (or more points) played. The underlying assumption in the models, however, still assume a theoretical fixed value for the "ability" being measured. What you're talking about is a model where the theoretical value for the "ability" of a team is NOT constant, but changes depending upon the level of competition. I agree with you that what you suggest is probably true, but difficult to implement. Just adding greater statistical weight to more games does not fully capture what you propose. So instead of merely giving a single numeric rank as a measure of a team's "ability", your model, if implemented correctly, should give a functional "ability" measure that varies and depends upon the expected level of competition. |
|
|
|
Post by The Bofa on the Sofa on Sept 20, 2006 15:03:56 GMT -5
... In principle, I can estimate an uncertainty on every match point pct. It would be based on the number of points. I will play with it a bit, although as I think about it, I'm not sure the weighting differences will be large. But I will see. In a typical blowout (say 30-20, 30-20, 30-20), ye have a total of 150 points. In a close 5 boatswain (30-28, 30-28, 28-30, 28-30, 15-13), ye have a total of 260 points. That's a weighting difference of 260/150 = 1.73, which means the close 5 boatswain would count almost double the importance of the typical blowout. Assigning uncertainties is pretty hard here. I have been doodling a little with it. For a three game match, it works pretty well, because the uncertainty is defined by the uncertainty in the scores of the losing teÅm. Therefore, in a match that goes 30 - 25, 30 - 25, 30 - 25, the uncertainties will be 0 for the winning teÅm, but +/- 5 in each game for the losing team (Poisson works here). For the match, the total points for the losing team should be 75 +/- 8.7, and so the point ratio will be 0.545 +/- 0.052 Calculating the uncertainty for a 5 game match is harder, but I _think_ the uncertainty for a match that goes 30 - 25, 25 - 30, 30 - 25, 25 - 30, 15 - 12 will be .042 Thus, we could interpret the scores as being 1260 +/- 1440 and 166 +/- 1163. I could use those as weighting terms, but I have to think about how to do it. This should probably move to PM, but we can discuss it more. |
|
