value of LMR and null-move

[Sentinel]

Removing LMR increases the brenching factor. Hence TM & IID must also be tuned for the testing to be accurate .

Not necessarily. Plus you can always test with time per move TC.

[Sentinel]

You may want to try with longer TC, LMR really kicks in at deep searches.

It would be interesting how much it gains going from 10"+0.1 to some thing like 30"+0.1, you just need to test 1.8 vs 1.8a because we have already seen that more or less is comparable to gauntlet result.

Bob doesn't see the difference with increasing TC in Crafty. I would say that's really strange.
For example for Ippo in my tests it's the following for default vs. no null move/LMR version (A vs A' testing):
6''+0.1'' 130elo
15''+0.25'' 146elo
1'+1'' 200elo
4'+4'' 242elo

Now it's starting to really take a lot of time to test, but I wonder when does it saturate...

[ThinkingALot]

Plus you can always test with time per move TC.

Such TC is apparently better for engines with low branching factor...

[mcostalba]

BTW, I made the changes as you suggested. Reading Bob I get the impression he made different ones?

I have to thank Bob for what he has done so far for testing 1.8 on his cluster, really nice thing.

But to have serious tests I would ask Bob to following this guidelines:

1) Post the exactly applied patch that removed LMR so that I can verify is correct

2) Post used TC. I cannot speak for Crafty, but for SF almost surely long TC helps LMR. So I suggest 30" per game and even 1' per game to see how LMR scales with TC.

3) Limit the test to 1.8 vs 1.8-no-LMR, gauntlet in this case could be thrown in a second step if needed (and I think is not for this particualr LMR scalability test).

Anhyhow thanks for testing !
Marco

[Sentinel]

Plus you can always test with time per move TC.

Such TC is apparently better for engines with low branching factor...

Removing LMR effectively increases the branching factor, that's correct. However, the only impact branching factor has on TC is in determining the optimal ratio between desired time per move and maximum time per move since this depends on time to complete iteration.
When TC is fixed time per move, the whole TM is practically off, so I really don't see what influence branching factor has there when you don't control anything...

[Bo Persson]

For the record, you have a _TON_ of warnings that should be cleaned up. I much prefer to eliminate all warnings by fixing the source rather than telling the compiler to just not display a set of warnings...

So do we.

What compiler are you using? I don't get a single compiler warning (when compiling with -Wall) with GCC 4.2 in Mac OS X 10.6.

I use Intel's compiler almost exclusively. Here's a couple for reference:


add<EvaluationFunction<KRKP> >("KRKP");
^
material.cpp(366): remark #383: value copied to temporary, reference to temporary used

material.cpp(421): remark #383: value copied to temporary, reference to temporary used
get<F>().insert(pair<Key, F*>(buildKey(keyCode), new T(WHITE)));
^
detected during instantiation of "void EndgameFunctions::add<T>(const std::string &) [with
T=EvaluationFunction<KNNK>]"

return rank_distance(square_rank(s1), square_rank(s2));
^
square.h(159): remark #981: operands are evaluated in unspecified order

Those aren't even warnings, but "remarks" issued by the compiler.

#383 says that the string literal "KRKP" is implicitly converted to a std::string parameter. That is quite intentional!

#981 tells us that the two calls to square_rank() can be made in any order, because there is no defined order for evaluating parameters. So what?!

I usually have these setting for the Intel 11.1 compiler:

Code: Select all

   // Turn off stupid diagnostics

   #pragma diag_suppress 304
   #pragma diag_suppress 383
   #pragma diag_suppress 810
   #pragma diag_suppress 981
   #pragma diag_suppress 1418
   #pragma diag_suppress 1419
   #pragma diag_suppress 1572

[ThinkingALot]

When TC is fixed time per move, the whole TM is practically off, so I really don't see what influence branching factor has there when you don't control anything...

Suppose an engine has T seconds per move. Each move it spends X seconds, there X < T is some random variable. X is at least not less than exp{log(T)/(bf)}, where (bf) is the branching factor (for simplicity assume it to be constant). If (bf) tends to 1 this lower bound converges to T, so the engine actually utilizes almost all availible time.

[nepossiver]

OK, clusters are idle so I can run over 500 games at a time. Results so far:

Code: Select all

Rank Name                  Elo    +    - games score oppo. draws
   1 Stockfish 1.8 64bit   2848   15   15  1763   83%  2552   19% 
   2 Stockfish 1.8a 64bit  2810   14   14  1754   78%  2561   22% 
   3 Glaurung 2.2          2556   13   13  1638   43%  2619   21% 
   4 Toga2                 2550   14   14  1595   42%  2618   19% 
   5 Fruit 2.1             2448   14   14  1625   28%  2639   20% 
   6 Glaurung 1.1 SMP      2387   15   15  1649   21%  2653   16% 

1.8a stockfish has LMR completely removed. Note that 1.8 and 1.8a will have 6,000 games between them by the time this is finished, along with 6000 games against each of the other opponents (a real RR that will waste some time since the others are also playing each other in addition to the two stockfish versions. So far, only +38 against a group, vs +50 head-to-head.

I got a bigger decrease:

Code: Select all

   1 Stockfish 1.8           84   12   12  1250   70%   -62   25% 
   2 Stockfish-no-LMR 1.8     0   12   12  1188   59%   -62   27% 
   3 Komodo32 1.2 JA        -62    8    9  2438   35%    43   26% 

I applied Marco's patch and compiled both with the standard Makefile, no pgo nor additional fast switches. Time control 40 / 10"+0.1", but on an old laptop, Pentium M 1.73GHz. Initial positions were from Dann Corbit 625_neutral.epd

[hyatt]

You may want to try with longer TC, LMR really kicks in at deep searches.

Agree.

The deeper the better recursive-null-move rocks also

BB also made a valid point regarding futility pruning, on a very short TC it favors the no-LMR version. And that should be avoided.

So a reasonable time-control is needed.

I chose for Blitz 5m. Still debatable of course.

My first try to run 100 blitz 5m games failed due to a wrong parameter (early draws) in the Fritz GUI and I had to abort the match. As a result many games were terminated after 15-20 moves with a draw result. However not counting the draws (35) SF 1.8 had 17 wins and SF 1.8 (no LMR) only 3. Telling IMO.

I am running it again now (draw=never) and with Blitz 5m SF 1.8 typically hits 20-22 plies in the middle-game, SF 1.8 (no LMR) typically 15-16. I am using 4 cores for each version, no PB. This in order to get a reasonable depth.

Bob of course will start to hammer on the issue of 100 games but if the results turns to be 60/30/10 it's clear we are not talking about a 50 elo improvement

BTW, I made the changes as you suggested. Reading Bob I get the impression he made different ones?

Ed

I simply added "0 &&" to the three if statements that cause the LMR code to be executed, which effectively disables the code and optimizes it away so that it is not even present in the binary.

My time control was 1+1 on 3.2ghz 64-bit intel hardware. If you are getting more than 50, with just LMR removed, I'd look at the methodology you are using first. I _know_ that my changes to search.cpp did nothing but disable LMR, and that it did disable it at all 3 points where it is done. Why you would use 4 cores is beyond me. Nothing like adding another level of uncertainty to the testing, since parallel search is so non-deterministic. BTW 17:3 doesn't tell "anything".

[hyatt]

You may want to try with longer TC, LMR really kicks in at deep searches.

It would be interesting how much it gains going from 10"+0.1 to some thing like 30"+0.1, you just need to test 1.8 vs 1.8a because we have already seen that more or less is comparable to gauntlet result.

Bob doesn't see the difference with increasing TC in Crafty. I would say that's really strange.
For example for Ippo in my tests it's the following for default vs. no null move/LMR version (A vs A' testing):
6''+0.1'' 130elo
15''+0.25'' 146elo
1'+1'' 200elo
4'+4'' 242elo

Now it's starting to really take a lot of time to test, but I wonder when does it saturate...

The first question is "how many games"? As far as t/c goes, mine was 1min+1sec.