value of LMR and null-move

[hyatt]

Thought it best to move this to a new topic since the old subject had nothing to do with this part of the discussion.

Here's current data from the tests I have running:

Code: Select all

Name                  Elo    +    - games score oppo. draws
Crafty-23.3-1        2704    3    3 30000   64%  2592   22%  
Crafty-23.3-2        2703    3    3 30000   64%  2592   22%  
Crafty-23.2-1        2656    3    3 30000   58%  2592   22%  
Crafty-23.2-2        2652    3    3 30000   58%  2592   22%  
Crafty-23.3R01-1     2643    3    3 30000   57%  2592   23%  
Crafty-23.3R01-2     2641    3    3 30000   56%  2592   23%  
Crafty-23.1-2        2637    3    3 30000   56%  2592   23%  
Crafty-23.1-1        2632    3    3 30000   55%  2592   23%  
Crafty-23.3R02-1     2587    3    3 30000   49%  2592   23%  
Crafty-23.3R02-2     2586    3    3 30000   49%  2592   23%  
Crafty-23.0-2        2543    3    3 30000   44%  2592   20%  
Crafty-23.0-1        2543    3    3 30000   44%  2592   20%  
Crafty-23.3R03-2     2457    4    4 30000   33%  2592   20%  
Crafty-23.3R03-1     2457    4    4 30000   33%  2592   20%  

Now, what is what. First, I ran every test twice as I had been asked to "stress-test the A/C" since it has been having major problems. As a result, I ran each version twice, which is the -1 and -2 you see on the end of each name.

Crafty-23.0, Crafty-23.1 and Crafty-23.2 are the released versions to date. Crafty-23.3 is the current version that will be released in a couple of weeks. 23.3R01 has null-move disabled. 23.3R02 has LMR disabled. 23.3R03 has both null-move and LMR disabled.

Summary:

R03 is obviously the worst version, with an Elo of 2457 after 30,000 games. R02 has null-move on, LMR off, and shows a gain of +130 over the base version, which is the gain by adding null-move. R01 has LMR on, null-move off, and adds +185 Elo. You might expect the total gain to be the sum of those two, or +315. But they are, as I mentioned, complementary in how they operate, so the total gain is only +246.

This is larger than the last time I ran this. Not sure why. For LMR, I am not surprised that it has gone up some since this has been changed quite a bit since the last time I ran this test. I can't recall any changes to the null-move search to make it more effective. But apparently something has changed in the search or the eval that helps it. Or which has introduced a side-effect that null-move covers up effectively and removing it exposes the issue.

I am now starting a run with stockfish 1.6, which is what I have been using in all my tests. I have modified it to eliminate LMR while leaving everything else intact. I'll report the results when this finishes...

[mcostalba]

I am now starting a run with stockfish 1.6, which is what I have been using in all my tests. I have modified it to eliminate LMR while leaving everything else intact. I'll report the results when this finishes...

Bob, could you pelase hold on a minute and use Stockfish 1.8 instead of 1.6 ?

You are doing this test to verify latest LMR trends (because you have already did this last year) so, given this goal, I _strongly_ suggest to use 1.8 because LMR is greatly different from 1.6

If you test 1.6 you get obsoletes results even before to start the test.

So don't waste your time (and mess the discussion) but please use the _proper_ reference that is Stockfish 1.8

Thanks

[hyatt]

I am now starting a run with stockfish 1.6, which is what I have been using in all my tests. I have modified it to eliminate LMR while leaving everything else intact. I'll report the results when this finishes...

Bob, could you pelase hold on a minute and use Stockfish 1.8 instead of 1.6 ?

You are doing this test to verify latest LMR trends (because you have already did this last year) so, given this goal, I _strongly_ suggest to use 1.8 because LMR is greatly different from 1.6

If you test 1.6 you get obsoletes results even before to start the test.

So don't waste your time (and mess the discussion) but please use the _proper_ reference that is Stockfish 1.8

Thanks

I can do that. The reason I don't try to keep the most current versions is that I have a ton of data, all based on 1.6 (which is my normal test opponent). And I am only interested in measuring the improvements made in Crafty. If I change opponents, I have to rerun all the old results to make the comparisons valid. In this case, that is not important, so I'll run this with 1.8...

[mcostalba]

I can do that. The reason I don't try to keep the most current versions is that I have a ton of data, all based on 1.6 (which is my normal test opponent). And I am only interested in measuring the improvements made in Crafty. If I change opponents, I have to rerun all the old results to make the comparisons valid. In this case, that is not important, so I'll run this with 1.8...

Thanks, I would have not asked if I was not sure 1.6 and 1.8 are very different in LMR implementation.

[hyatt]

I can do that. The reason I don't try to keep the most current versions is that I have a ton of data, all based on 1.6 (which is my normal test opponent). And I am only interested in measuring the improvements made in Crafty. If I change opponents, I have to rerun all the old results to make the comparisons valid. In this case, that is not important, so I'll run this with 1.8...

Thanks, I would have not asked if I was not sure 1.6 and 1.8 are very different in LMR implementation.

For the record, you have a _TON_ of warnings that should be cleaned up. I much prefer to eliminate all warnings by fixing the source rather than telling the compiler to just not display a set of warnings...

starting test using 1.8 now...

[mcostalba]

For the record, you have a _TON_ of warnings that should be cleaned up.

Hmmm this is _not_ expected, actually compile output is with very very few warnings on MSVC and Intel and with no warnings on gcc.

Could you please post compile output ?

Thanks, this is something important for me to fix

[Rebel]

For the record, you have a _TON_ of warnings that should be cleaned up.

Hmmm this is _not_ expected, actually compile output is with very very few warnings on MSVC and Intel and with no warnings on gcc.

Could you please post compile output ?

Thanks, this is something important for me to fix

I got only 1, something about cmd.exe (the kind of DOS box thing)

Nothing to worry about IMO.

Ed

[hyatt]

For the record, you have a _TON_ of warnings that should be cleaned up.

Hmmm this is _not_ expected, actually compile output is with very very few warnings on MSVC and Intel and with no warnings on gcc.

Could you please post compile output ?

Thanks, this is something important for me to fix

I got only 1, something about cmd.exe (the kind of DOS box thing)

Nothing to worry about IMO.

Ed

I had to remove the warning message deletions, as our cluster software is never completely up to date, to avoid changes that alter program behaviour in unexpected ways. When I removed the warning message deletions for intel's compiler, I got lots of warnings about the order of operations being non-specified... plus a few others. I'll try to re-run the make and copy the messages here later...

[hyatt]

OK, test starts off weird. I first started a test between two versions of stockfish 1.8. I am calling them 1.8 and 1.8a. In 1.8a, at every "if (cond)" that causes the LMR reduction to be done, I just added a simple "0 &&" at the front which causes that code to not execute and actually be optimized out (I looked at the .s file to verify). Before starting a big round robin with crafty + 5 normal opponents, + new (no-LMR) stockfish, I thought I would just run 1.8 vs 1.8a to make sure that things look sane. Here's the results so far:

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws
   1 Stockfish 1.8 64bit   2625    6    6  2379   58%  2575   47% 
   2 Stockfish 1.8a 64bit  2575    6    6  2379   42%  2625   47% 

So far, in testing LMR on and off, it looks like +50 for stockfish 1.8a. Not sure why it is showing up as a far smaller improvement than what is in Crafty, unless there are other things in the new stockfish search that are complementary and taking up part of the slack when LMR is disabled. I'm going to let this run just a bit longer to see if anything changes, then I will run the round-robin with 7 opponents including both SF versions to see how it does overall...

[Tord]

For the record, you have a _TON_ of warnings that should be cleaned up. I much prefer to eliminate all warnings by fixing the source rather than telling the compiler to just not display a set of warnings...

So do we.

What compiler are you using? I don't get a single compiler warning (when compiling with -Wall) with GCC 4.2 in Mac OS X 10.6.