Post
by Chris Whittington » Sun Jun 20, 2010 10:35 am
Rating lists as they are currenlty compiled are worse than useless and perpetuate by positive feedback the cul-de-sac that is chess program development.
Fundamentally a rating list should be independent of the development process. In other words it should measure something that can't be easily predicted by developers.
Think about a parallel system for a minute. Education and exams. If the exam is known to teachers beforehand, and teachers are rewarded by class exam results, then they will teach "to the exam", even, at worst, teaching known answers to the known questions. This is no education that anyone could want, but it leads to successful exam grades (teachers like it, their pay depends), continual use of the same exam boards that set the questions (exam boards like that) and collusion between teachers and exam boards which can result in payoffs, manipulations, fraud and worse.
Now, translate this back to computer chess. Developers work, basically all in the same way at present, they make a tweak or write some new code and then autoplay test several tens of thousands of games to detect if there's a better win/draw/loss statistic. If there is they keep the version, else they tweak again. The education process for a chess engine.
Testers then autoplay several tens of thousands or more games, and present a list of win/draw/loss statistics (the exam). These statistics tell the developers nothing they didn't already know, instead they provide positive feedback to developers to continue doing what they do. The exam is known and developers work to the exam.
We get continued use of favoured exam boards, continued use of a flawed(I think so) development process that emphasises playing in the comp-comp pool only (which favours tactical developments over strategic developments) and opens up all possibilities for collusion, manipulation and corruption between examiners (rating lists) and teachers (programmers) and others too. Meanwhile the feedback loop between rating lists and developments pushes computer chess into a particular direction and pushes the rating lists to ever new and unrealistic highs as they get more and more out of tune with reality.
Solution is to dump rating lists and promote a different way to measure excellence. Even find a new measure of excellence.