Worlds top chess engines needs lots of comparing
Posted: Fri Sep 16, 2011 8:22 am
Robert Houdart advise at least about 500 games for his older chess engine (Houdini 1.5a), versus his newest chess engine (Houdini Pro 2.0)
to really compare the two engines.
I have run multiple tournament runs between each and are busy with a 500 run, which will take more than a week to finish.
With that in mind, I have ran two sets of 10 games each (30 minute tournaments) between Houdini 1.5a and Houdini 2.0 (32 bit).
1. Houdini 1.5 won the first tournament set with 83% versus Houdini 2.0's 17%
2. Houdini 2.0 won the next set of 10 games with 75% versus Houdini 1.5a's 25%
Thus one can conclude that it is very important to play a LOT of games between the worlds top chess engines, before one can say with
confidence that one chess engine is indeed stronger than the other and by how much. The closer the comparison, the more games one needs
to play.
I have seen a lot of runs in the 10's, 20 and even 30's, where one chess engine beats the other, but it still could be that the loser is actually
the stronger of the two!
My 500 run test is at this stage at 60 games (30 minute tournament). Houdin 2.0 leads with 51% versus 49%. Only after game 33 did
Houdini 2.0 pass Houdini 1.5a on the stats. Thus if I had stopped after 32 games, it would have appeared that Houdin 1.5a is the better engine...
to really compare the two engines.
I have run multiple tournament runs between each and are busy with a 500 run, which will take more than a week to finish.
With that in mind, I have ran two sets of 10 games each (30 minute tournaments) between Houdini 1.5a and Houdini 2.0 (32 bit).
1. Houdini 1.5 won the first tournament set with 83% versus Houdini 2.0's 17%
2. Houdini 2.0 won the next set of 10 games with 75% versus Houdini 1.5a's 25%
Thus one can conclude that it is very important to play a LOT of games between the worlds top chess engines, before one can say with
confidence that one chess engine is indeed stronger than the other and by how much. The closer the comparison, the more games one needs
to play.
I have seen a lot of runs in the 10's, 20 and even 30's, where one chess engine beats the other, but it still could be that the loser is actually
the stronger of the two!
My 500 run test is at this stage at 60 games (30 minute tournament). Houdin 2.0 leads with 51% versus 49%. Only after game 33 did
Houdini 2.0 pass Houdini 1.5a on the stats. Thus if I had stopped after 32 games, it would have appeared that Houdin 1.5a is the better engine...