Not necessarily. Plus you can always test with time per move TC.ThinkingALot wrote:Removing LMR increases the brenching factor. Hence TM & IID must also be tuned for the testing to be accurate .
value of LMR and null-move
Re: value of LMR and null-move
Re: value of LMR and null-move
Bob doesn't see the difference with increasing TC in Crafty. I would say that's really strange.mcostalba wrote:You may want to try with longer TC, LMR really kicks in at deep searches.
It would be interesting how much it gains going from 10"+0.1 to some thing like 30"+0.1, you just need to test 1.8 vs 1.8a because we have already seen that more or less is comparable to gauntlet result.
For example for Ippo in my tests it's the following for default vs. no null move/LMR version (A vs A' testing):
6''+0.1'' 130elo
15''+0.25'' 146elo
1'+1'' 200elo
4'+4'' 242elo
Now it's starting to really take a lot of time to test, but I wonder when does it saturate...
-
- Posts: 144
- Joined: Sun Jun 13, 2010 7:32 am
- Contact:
Re: value of LMR and null-move
Such TC is apparently better for engines with low branching factor...Sentinel wrote:Plus you can always test with time per move TC.
Re: value of LMR and null-move
I have to thank Bob for what he has done so far for testing 1.8 on his cluster, really nice thing.Rebel wrote: BTW, I made the changes as you suggested. Reading Bob I get the impression he made different ones?
But to have serious tests I would ask Bob to following this guidelines:
1) Post the exactly applied patch that removed LMR so that I can verify is correct
2) Post used TC. I cannot speak for Crafty, but for SF almost surely long TC helps LMR. So I suggest 30" per game and even 1' per game to see how LMR scales with TC.
3) Limit the test to 1.8 vs 1.8-no-LMR, gauntlet in this case could be thrown in a second step if needed (and I think is not for this particualr LMR scalability test).
Anhyhow thanks for testing !
Marco
Re: value of LMR and null-move
Removing LMR effectively increases the branching factor, that's correct. However, the only impact branching factor has on TC is in determining the optimal ratio between desired time per move and maximum time per move since this depends on time to complete iteration.ThinkingALot wrote:Such TC is apparently better for engines with low branching factor...Sentinel wrote:Plus you can always test with time per move TC.
When TC is fixed time per move, the whole TM is practically off, so I really don't see what influence branching factor has there when you don't control anything...
- Bo Persson
- Posts: 14
- Joined: Thu Jun 10, 2010 10:34 am
- Real Name: Bo Persson
- Location: Malmö, Sweden
Re: value of LMR and null-move
Those aren't even warnings, but "remarks" issued by the compiler.hyatt wrote:I use Intel's compiler almost exclusively. Here's a couple for reference:Tord wrote:So do we.hyatt wrote:For the record, you have a _TON_ of warnings that should be cleaned up. I much prefer to eliminate all warnings by fixing the source rather than telling the compiler to just not display a set of warnings...
What compiler are you using? I don't get a single compiler warning (when compiling with -Wall) with GCC 4.2 in Mac OS X 10.6.
add<EvaluationFunction<KRKP> >("KRKP");
^
material.cpp(366): remark #383: value copied to temporary, reference to temporary used
material.cpp(421): remark #383: value copied to temporary, reference to temporary used
get<F>().insert(pair<Key, F*>(buildKey(keyCode), new T(WHITE)));
^
detected during instantiation of "void EndgameFunctions::add<T>(const std::string &) [with
T=EvaluationFunction<KNNK>]"
return rank_distance(square_rank(s1), square_rank(s2));
^
square.h(159): remark #981: operands are evaluated in unspecified order
#383 says that the string literal "KRKP" is implicitly converted to a std::string parameter. That is quite intentional!
#981 tells us that the two calls to square_rank() can be made in any order, because there is no defined order for evaluating parameters. So what?!
I usually have these setting for the Intel 11.1 compiler:
Code: Select all
// Turn off stupid diagnostics
#pragma diag_suppress 304
#pragma diag_suppress 383
#pragma diag_suppress 810
#pragma diag_suppress 981
#pragma diag_suppress 1418
#pragma diag_suppress 1419
#pragma diag_suppress 1572
-
- Posts: 144
- Joined: Sun Jun 13, 2010 7:32 am
- Contact:
Re: value of LMR and null-move
Suppose an engine has T seconds per move. Each move it spends X seconds, there X < T is some random variable. X is at least not less than exp{log(T)/(bf)}, where (bf) is the branching factor (for simplicity assume it to be constant). If (bf) tends to 1 this lower bound converges to T, so the engine actually utilizes almost all availible time.Sentinel wrote:When TC is fixed time per move, the whole TM is practically off, so I really don't see what influence branching factor has there when you don't control anything...
-
- Posts: 3
- Joined: Tue Jul 13, 2010 3:31 am
Re: value of LMR and null-move
I got a bigger decrease:hyatt wrote:OK, clusters are idle so I can run over 500 games at a time. Results so far:
1.8a stockfish has LMR completely removed. Note that 1.8 and 1.8a will have 6,000 games between them by the time this is finished, along with 6000 games against each of the other opponents (a real RR that will waste some time since the others are also playing each other in addition to the two stockfish versions. So far, only +38 against a group, vs +50 head-to-head.Code: Select all
Rank Name Elo + - games score oppo. draws 1 Stockfish 1.8 64bit 2848 15 15 1763 83% 2552 19% 2 Stockfish 1.8a 64bit 2810 14 14 1754 78% 2561 22% 3 Glaurung 2.2 2556 13 13 1638 43% 2619 21% 4 Toga2 2550 14 14 1595 42% 2618 19% 5 Fruit 2.1 2448 14 14 1625 28% 2639 20% 6 Glaurung 1.1 SMP 2387 15 15 1649 21% 2653 16%
Code: Select all
1 Stockfish 1.8 84 12 12 1250 70% -62 25%
2 Stockfish-no-LMR 1.8 0 12 12 1188 59% -62 27%
3 Komodo32 1.2 JA -62 8 9 2438 35% 43 26%
-
- Posts: 1242
- Joined: Thu Jun 10, 2010 2:13 am
- Real Name: Bob Hyatt (Robert M. Hyatt)
- Location: University of Alabama at Birmingham
- Contact:
Re: value of LMR and null-move
I simply added "0 &&" to the three if statements that cause the LMR code to be executed, which effectively disables the code and optimizes it away so that it is not even present in the binary.Rebel wrote:Agree.mcostalba wrote:You may want to try with longer TC, LMR really kicks in at deep searches.
The deeper the better recursive-null-move rocks also
BB also made a valid point regarding futility pruning, on a very short TC it favors the no-LMR version. And that should be avoided.
So a reasonable time-control is needed.
I chose for Blitz 5m. Still debatable of course.
My first try to run 100 blitz 5m games failed due to a wrong parameter (early draws) in the Fritz GUI and I had to abort the match. As a result many games were terminated after 15-20 moves with a draw result. However not counting the draws (35) SF 1.8 had 17 wins and SF 1.8 (no LMR) only 3. Telling IMO.
I am running it again now (draw=never) and with Blitz 5m SF 1.8 typically hits 20-22 plies in the middle-game, SF 1.8 (no LMR) typically 15-16. I am using 4 cores for each version, no PB. This in order to get a reasonable depth.
Bob of course will start to hammer on the issue of 100 games but if the results turns to be 60/30/10 it's clear we are not talking about a 50 elo improvement
BTW, I made the changes as you suggested. Reading Bob I get the impression he made different ones?
Ed
My time control was 1+1 on 3.2ghz 64-bit intel hardware. If you are getting more than 50, with just LMR removed, I'd look at the methodology you are using first. I _know_ that my changes to search.cpp did nothing but disable LMR, and that it did disable it at all 3 points where it is done. Why you would use 4 cores is beyond me. Nothing like adding another level of uncertainty to the testing, since parallel search is so non-deterministic. BTW 17:3 doesn't tell "anything".
-
- Posts: 1242
- Joined: Thu Jun 10, 2010 2:13 am
- Real Name: Bob Hyatt (Robert M. Hyatt)
- Location: University of Alabama at Birmingham
- Contact:
Re: value of LMR and null-move
The first question is "how many games"? As far as t/c goes, mine was 1min+1sec.Sentinel wrote:Bob doesn't see the difference with increasing TC in Crafty. I would say that's really strange.mcostalba wrote:You may want to try with longer TC, LMR really kicks in at deep searches.
It would be interesting how much it gains going from 10"+0.1 to some thing like 30"+0.1, you just need to test 1.8 vs 1.8a because we have already seen that more or less is comparable to gauntlet result.
For example for Ippo in my tests it's the following for default vs. no null move/LMR version (A vs A' testing):
6''+0.1'' 130elo
15''+0.25'' 146elo
1'+1'' 200elo
4'+4'' 242elo
Now it's starting to really take a lot of time to test, but I wonder when does it saturate...