Stockfish question

[hyatt]

BTW you'd certanly know that you can download and setup MSVC 2010 Express version for free and in less then half an hour, do you ?

Well, as nobody volunteered I had too I have an eng-eng match running now, SF 1.8 vs SF 1.8 (no LMR). Long time ago I did the stuff.

The normal SF typically hits 5-6 more plies than the no-LMR version on a time-control of 40/10. Nevertheless the first game looks bad for SF 1.8

We will see...

Ed

I am running 4 versions of Crafty at the moment. normal, no-null-move, no-LMR and no-null-move-or-LMR. Once that finishes, assuming our A/C doesn't die in the cluster room, I will repeat with Stockfish, except I will only remove LMR to save some testing time.

For the record, I don't like eng-eng tests when they are the same version except for one feature turned off. Tends to exaggerate the gain or loss of any change.

[kingliveson]

BTW you'd certanly know that you can download and setup MSVC 2010 Express version for free and in less then half an hour, do you ?

Well, as nobody volunteered I had too I have an eng-eng match running now, SF 1.8 vs SF 1.8 (no LMR). Long time ago I did the stuff.

The normal SF typically hits 5-6 more plies than the no-LMR version on a time-control of 40/10. Nevertheless the first game looks bad for SF 1.8

We will see...

Ed

I am running 4 versions of Crafty at the moment. normal, no-null-move, no-LMR and no-null-move-or-LMR. Once that finishes, assuming our A/C doesn't die in the cluster room, I will repeat with Stockfish, except I will only remove LMR to save some testing time.

For the record, I don't like eng-eng tests when they are the same version except for one feature turned off. Tends to exaggerate the gain or loss of any change.

Excellent idea...surprising though that we do not already have handy such data.

[hyatt]

BTW you'd certanly know that you can download and setup MSVC 2010 Express version for free and in less then half an hour, do you ?

Well, as nobody volunteered I had too I have an eng-eng match running now, SF 1.8 vs SF 1.8 (no LMR). Long time ago I did the stuff.

The normal SF typically hits 5-6 more plies than the no-LMR version on a time-control of 40/10. Nevertheless the first game looks bad for SF 1.8

We will see...

Ed

I am running 4 versions of Crafty at the moment. normal, no-null-move, no-LMR and no-null-move-or-LMR. Once that finishes, assuming our A/C doesn't die in the cluster room, I will repeat with Stockfish, except I will only remove LMR to save some testing time.

For the record, I don't like eng-eng tests when they are the same version except for one feature turned off. Tends to exaggerate the gain or loss of any change.

Excellent idea...surprising though that we do not already have handy such data.

We do. I ran this for Crafty, stockfish and fruit last year. But Crafty and Stockfish have changed significantly since then, so a re-test will bring the data current. There are way too many wild-a** guesses about what this stuff does. It is not a miracle cure-all for tree searching. For Crafty, about 18 months ago or so, either LMR or null-move added +80. Both together only added +120 since when you think about it, they are complementary in how they function, both reducing the search depth to get out more quickly...

Should have complete results (posted in other thread where we were discussing this) tomorrow. I just posted some partial data there, showing null-move is adds 60 by itself, and LMR about 115. This latter number is about 10,000 games (out of 30,000) shy of being complete, then I have a run with both null and LMR off to get a base-line. I already have results for normal crafty with both enabled.

[kingliveson]

BTW you'd certanly know that you can download and setup MSVC 2010 Express version for free and in less then half an hour, do you ?

Well, as nobody volunteered I had too I have an eng-eng match running now, SF 1.8 vs SF 1.8 (no LMR). Long time ago I did the stuff.

The normal SF typically hits 5-6 more plies than the no-LMR version on a time-control of 40/10. Nevertheless the first game looks bad for SF 1.8

We will see...

Ed

I am running 4 versions of Crafty at the moment. normal, no-null-move, no-LMR and no-null-move-or-LMR. Once that finishes, assuming our A/C doesn't die in the cluster room, I will repeat with Stockfish, except I will only remove LMR to save some testing time.

For the record, I don't like eng-eng tests when they are the same version except for one feature turned off. Tends to exaggerate the gain or loss of any change.

Excellent idea...surprising though that we do not already have handy such data.

We do. I ran this for Crafty, stockfish and fruit last year. But Crafty and Stockfish have changed significantly since then, so a re-test will bring the data current. There are way too many wild-a** guesses about what this stuff does. It is not a miracle cure-all for tree searching. For Crafty, about 18 months ago or so, either LMR or null-move added +80. Both together only added +120 since when you think about it, they are complementary in how they function, both reducing the search depth to get out more quickly...

Should have complete results (posted in other thread where we were discussing this) tomorrow. I just posted some partial data there, showing null-move is adds 60 by itself, and LMR about 115. This latter number is about 10,000 games (out of 30,000) shy of being complete, then I have a run with both null and LMR off to get a base-line. I already have results for normal crafty with both enabled.

Comparing previous numbers you mentioned to the partial data, it's over 40% shift with LMR should the results hold. Quite a change. Well, lets see the final results with both LMR and null-move removed to put the wild-a** guesses behind.

[BB+]

It happens to be easy with Stockfish, but disabling LMR (and related ideas) need not be so easy.
For instance, in IvanHoe, not only is there LMR, but also various "ignore this move" futility conditions. I don't how much these "extended futility checks" interact with LMR.

Here are some relevant parts (I chose all_node.c):

Code: Select all

      if (cnt > 5 && NextMove->phase == ORDINARY_MOVES
          && (move & 0xe000) == 0 && SqSet[fr] & ~MyXRAY && depth < 20)
        {
          if ((5 << (depth - 6)) + MAX_POSITIONAL (move) +
              (POS0->Value) < VALUE + 35 + 2 * cnt)
            {
              cnt++;
              continue;
            }
        }
      if (depth < 20 && (2 << (depth - 6)) + (POS0->Value) < VALUE + 125
          && NextMove->phase == ORDINARY_MOVES && MyKingSq != fr
          && SqSet[fr] & ~MyXRAY && (move & 0x8000) == 0
          && !MySEE (POSITION, move))
        {
          cnt++;
          continue;
        }

Those are "pre-make" futile conditions. There is also one after the make_move:

Code: Select all

          if (cnt > 5 && depth < 20 && POS1->cp == 0
              && (2 << (depth - 6)) - POS1->Value < VALUE + cnt - 15)
            {
              UNDO (POSITION, move);
              cnt++;
              continue;
            }

Finally there is the LMR proper:

Code: Select all

          if (NextMove->phase == ORDINARY_MOVES && cnt >= 3)
            {
              new_depth = depth - 2 + EXTEND - BSR (1 + cnt);
              /* usa BSR32 con 32-bits ? */
              if (QSEARCH_CONDITION)
                v = -OppQsearch (POSITION, 1 - VALUE, 0);
              else if (LOW_DEPTH_CONDITION)
                v = -OppLowDepth (POSITION, 1 - VALUE, new_depth);
              else
                v = -OppCut (POSITION, 1 - VALUE, new_depth);
              if (v < VALUE)
                goto DONE;
            }

Given that "cnt > 5" is a condition with the first/third futility checks, there might be some overlap with LMR. To further complicate matters, low_depth search has no explicit LMR (to my eye), and varies these futility checks (also, when depth is more than maybe 12 half-ply in the above, I think the conditions become almost always false).

[Sentinel]

Should have complete results (posted in other thread where we were discussing this) tomorrow. I just posted some partial data there, showing null-move is adds 60 by itself, and LMR about 115. This latter number is about 10,000 games (out of 30,000) shy of being complete, then I have a run with both null and LMR off to get a base-line. I already have results for normal crafty with both enabled.

An important question here, what's the TC?
From my tests doubling the time adds more than 1 percent in default vs. disabled null move/LMR.
So if you test with 16 times longer TC, the difference increases for roughly 40 elo.

[Sentinel]

Given that "cnt > 5" is a condition with the first/third futility checks, there might be some overlap with LMR. To further complicate matters, low_depth search has no explicit LMR (to my eye), and varies these futility checks (also, when depth is more than maybe 12 half-ply in the above, I think the conditions become almost always false).

The point is all reductions should be disabled, and there is no point in trying to separate LMR from futility pruning.
To cut the story short, just commenting out part of each line which has new_depth in it, works the job:

Code: Select all

new_depth = depth - 2 + EXTEND; // - BSR (1 + cnt);

[Rebel]

For the record, I don't like eng-eng tests when they are the same version except for one feature turned off. Tends to exaggerate the gain or loss of any change.

My experience is different. As a rule of fist, test positional changes against a wide range of engines, test search related changes against your own.

Ed

[hyatt]

Should have complete results (posted in other thread where we were discussing this) tomorrow. I just posted some partial data there, showing null-move is adds 60 by itself, and LMR about 115. This latter number is about 10,000 games (out of 30,000) shy of being complete, then I have a run with both null and LMR off to get a base-line. I already have results for normal crafty with both enabled.

An important question here, what's the TC?
From my tests doubling the time adds more than 1 percent in default vs. disabled null move/LMR.
So if you test with 16 times longer TC, the difference increases for roughly 40 elo.

I am running 1+1. My testing for LMR does not mirror yours, however. I've tested two versions, A and B where B has no LMR, and found no difference in A vs B Elo from 10s+0.1s to 10m+10s which is as far as I have gone. The relative ratings of A and B may well shift some depending on the time control, but the gap between them remains quite constant in my testing.

[hyatt]

For the record, I don't like eng-eng tests when they are the same version except for one feature turned off. Tends to exaggerate the gain or loss of any change.

My experience is different. As a rule of fist, test positional changes against a wide range of engines, test search related changes against your own.

Ed

The problem is that if you test A vs A', you might get +30 with a search change. Then if you run A and A' against a range of engines, you may only get +5. It is easier, and more accurate, to use the same test setup each time, then there's no accidental changes to skew the results unknowingly.