Houdini Versions

[Taner Altinsoy]

I would like to know the elo difference between different versions of Houdini. I suspect 1.02 might not be better than 1.01 elowise so I want to run a tournament among these 3 versions and expect your suggestions. Is running a few thousand 30 sec games sufficient? Or do you think 1 min or 2 min games would be better? Mr Houdart's opinion would be much appreciated.

Taner

[Robert Houdart]

Houdini 1.02 was mainly a bug fix release, there shouldn't be any real difference in strength with Houdini 1.01.
Houdini 1.0 is probably about 15 to 20 points weaker.
As always, your mileage may vary depending on opponents, hardware and time control.

Robert

[Vael Jean-Paul]

Hi Robert,

I will soon send you my results!

Everybody use his own system in testings..i always look how fast is my hardware..when i build a new system i run test positions,look to his nodes/sec.
I check when he get his stable high nodes/sec. because your computer need a little time to get his 100% cpu usage..checking Temps..so i know exactly when
he is working under full load.
Then i choose my Time for a game needed go get these stable high nodes/sec. with max. load for one move!
Long before i had a AMD 3400+ with that system i play 30min. games
Now with my Core i7 920 @3.80Ghz i running Blitz 5min. and i go even deeper in depth then before!
Many play 1min. games..these i find to fast for my system to get nice good quality games..results change to much..but of course it can be fun!
Everybody is free to do what he likes!

JP.

[Taner Altinsoy]

Whatever time control you use you have to play at least 500 games in order to get a reliable elo estimation. That's why I'm planning to play a few thousand 30 sec games and since it's diff version of the same engine fast time controls does not matter that much. As of now I've completed 952of 300 games and it's 1.01= 330/634, 1.02 = 316/634 and 1.00 = 304/634. Elo difference is approximately 10 between them. So yes there seems to be a 20 elo gap between 1.00 and 1.01 but 1.02 does look like a -10 elo. Although I have to say it's still early for any conclusion, I will play at least 3000 games or more to come up with a reliable elo estimation.

Taner

[Charles]

Add 80 more games suggesting 1.01 slightly better than 1.02 -- 4 cpu 32 bit.

40 games 1.02
Houdini_w32_4CPU_1.02 - Rybka 4 w32 : 19.0/40 8-10-22 (0==011=0=11====0=1=1======01=10000====0=) 48%

Nunn 20 position suite tc=+3s
50 games 1.01
Houdini_w32_4CPU - Rybka 4 w32 : 21.0/40 13-11-16 (011=11110=0==10=100===111==0==0==00=011=) 53%



By itself it means nothing -- but add it to the ongoing tests/comparisons and it does seem to suggest 1.01 is better than 1.02

[Gerold]

Houdini 1.02 was mainly a bug fix release, there shouldn't be any real difference in strength with Houdini 1.01.
Houdini 1.0 is probably about 15 to 20 points weaker.
As always, your mileage may vary depending on opponents, hardware and time control.

Robert

Hi Robert.
In game in 5minutes i had Houdini 1.01 a bit stronger than 1.02 by 15 elo

Just started another match to double check will run 500 games.

Best,
Gerold.

[Robert Houdart]

Gerold, Charles, Taner, Jean-Paul,

In truth, I wouldn't know what could cause these differences other than random behaviour. The evaluation function in both versions is identical, and the changes in the search algorithm are minor.

If all your results confirm a difference of 10 or 15 Elo, I'll happily revert the minor changes in the search algorithm back to the Houdini 1.01 state.
The standard deviation of a 500 game match is about 2% or 15 Elo, so clearly we need a lot more games. For a 15 Elo strength difference I usually run about 2000 games, for 10 Elo I require about 4000 games.

Looking forward to your results,
Robert

[Taner Altinsoy]

Hello,

After 1484/3000 30 sec games the elo of engines looks like this. I will complete the tournament and play more games if needed.

Houdini W32 1cpu(1.01) 2809
Houdini W32 1cpu(1.02) 2802
Houdini W32 1cpu(1.00) 2789

Please note the 20 elo difference of 1.00 and 1.01 which has been confirmed and established before by many others.

regards,
Taner

[Vael Jean-Paul]

A easy and very good way to see if the new version plays better is to run STS tests from Swaminathan,now 1000 positions and when see higher scores, most of the time he will have better results when he play against other engines as the version before!

We know all how more games how better,but to play thousand and thousand games against 1,2 or 3 engines tell you not much,because there is always a engine he don't like to play against it!

Let a engine play against as many possible different engines and you will see faster if this engine is stronger or not!

Just to tell a little example..look this engine is stronger then Rybka 4 i run thousand games and get 54% ,and if you let it play against Rybka 3 ,this engine looses! And you will see that with every engine version!
I look always to the total,all engines have to play against all same engines,then i get very fast (after every whole day and some weeks testing)
a idea how strong a engine is,or know fast if the new version is stronger or not!

Not only games,alo using STS ,other test positions,before a engine play against all other engines...

JP.

[Robert Houdart]

Taner, your results are still pretty much consistent with the initial hypothesis that 1.01 and 1.02 are about equally strong, and 1.0 is about 15 to 20 Elo weaker. More games needed .