Its not your Ratio - Lc0 has not caught up to Stockfish NNUE

[Chess_Dragon]

Subtitle: Happy Belated 2nd Birthday to the T60 net!

Lc0 net history:
https://github.com/LeelaChessZero/lc0/w ... ts-for-Lc0
2019-07-26 : 42850 : last T40 net 24x256 just before 1st T60 net started the same day
2019-07.26 ( 600001 : 1st T60 30x320 using random moves, est. -1200 Elo; same day as 42850)
2020-06-02 : LS15 : from jjosh 24x256 net while T60 was getting stronger
2021-08-26 : 609996 : recent T60 30x384 net (previously enlarged from the original)

Match conditions:
- G/3'+1" (~12 hours each match)
- Ryzen 7 2700X 16SMTthreads 2GB hash | RTX 2060 , Syzygy-6 (Sf:Sy50off) , all m.o.=50
- 100 games: DOTS2 50 openings colors reversed each match

Engines:
- Stockfish-21082721 recent net & patch (avx2: GCC 10.3, Ubuntu 21.04).
- Lc0-0.28.0 latest official release 2021-08-25 (Ubuntu 21.04), multi_stream=true


The last T40 net vs. the Stockfish commit (was before NNUE) for the same day, 2 years 1 month ago:

Code: Select all

Match #1                  Elo   +  -  Games  Score      Data
  1 Stockfish_19072603  :  34  54 54   100   55.0%   +36 -26 =38
  2 Lc0-0.28.0_42850    :   0  54 54   100   45.0%   Draws 38.0%

One year and 3 months ago the LeelaStein 15 was released. This net vs. the last 'classical' before NNUE Stockfish, 1 year 1 month ago:

Code: Select all

Match #2                        Elo   +  -  Games  Score      Data  
  1 Lc0-0.28.0_LS15           :  24  45 45   100   53.5%   +25 -18 =57
  2 SfLastClassical_20073110  :   0  45 45   100   46.5%   Draws 57.0%

A very recent Stockfish vs. a very recent T60 net:

Code: Select all

Match #3                  Elo   +  -  Games  Score      Data
  1 Stockfish-21082721  :  86  46 44   100   62.0%   +34 -10 =56
  2 Lc0-0.28.0_609996   :   0  44 46   100   38.0%   Draws 56.0%

Under the above conditions:
2+ years ago until now: Lc0 is 52 Elo further behind Stockfish than it was at the start of T60, comparing Match #1 with Match #3.
1+ years ago: Lc0 is 110 Elo further behind Stockfish than it was at the time of Stockfish's last 'classical', comparing Match #2 with Match #3.
Currently: Stockfish is 86 Elo ahead of Lc0.


Cross-checking:

Code: Select all

Match #4                   Elo   +  -  Games  Score      Data
  1 Stockfish-21082721  :  190  53 49   100   75.0%   +53 -3 =44
  2 Lc0-0.28.0_42850    :    0  49 53   100   25.0%   Draws 44%
  
Match #5                   Elo   +  -  Games  Score      Data
  1 Lc0-0.28.0_609996   :  108  50 49   100   65.0%   +41 -11 =48
  2 Stockfish_19072603  :    0  49 50   100   35.0%   Draws 48.0%

Under the above conditions:
Recent Stockfish is 190 Elo stronger than the last T40 net, Match #4.
Recent Lc0 is 108 Elo stronger than Stockfish on the day of the last T40 net/start of T60, Match #5.
Difference is 82 Elo, but not to double count Match #1: 82 - 34* = 48 Elo, consistent with the above 2+ years 52 Elo gain in Stockfish vs. Lc0.
(* Match #4 relates to Match #1 with Stockfish 'then' version +34, but Match #5 relates to Match #1 with Lc0 at 0.)

[dorsz]

Interesting. I have found this thread just before a moment.

[Chess_Dragon]

A study in Lc0 nets progress vs. Stockfsih.
All nets using Lc0 0.28.0, cuda-fp16, defaults.
Thanks to DOT3, this forum viewtopic.php?f=7&t=3303 , low draw rates.

R7 2700X 16threads 2Gh | RTX 2060
G/45 secs. + 0.5 secs. | Syzygy-6
DOTS3 book each engine: all 100 openings colors reversed, 200 games each.

Last Stockfish commit just before NNUE (200731) vs. Lc0 with nets available at that time:

Code: Select all

1. Gauntlet - Sf Last Classical vs SV 30x384 (200726) & T60 (200731, 24x320) & LS 15 (200612, 24x256)
   # PLAYER                     :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 Stockfish_20073110_LC      :     0.0   ----   303.5     600    51      54  188  231  181    38
   2 Lc0-0.28.0--atLC-604434    :    -1.8   37.4    99.5     200    50      50   60   79   61    40
   3 Lc0-0.28.0__LS-15.0        :    -1.8   35.3    99.5     200    50      60   61   77   62    38
   4 Lc0-0.28.0_SV-t60-4585     :    -8.8   39.7    97.5     200    49     ---   60   75   65    38

Pre-NNUE Stockfish conistantly was only slightly ahead of the Lc0 nets at that time.

T60 increased its size, the 1st large net vs. the Stockfish available at that time:

Code: Select all

2. Match - T60 1st 30x384 (201207) vs Sf (201205)
   # PLAYER                    :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 Stockfish_20120516        :     0.0   ----   121.5     200    61     100   78   87   35    44
   2 Lc0-0.28.0--1st-606512    :   -76.9   35.4    78.5     200    39     ---   35   87   78    44
White advantage = 23.93 +/- 20.41 | Draw rate (equal opponents) = 45.63 % +/- 3.31

5 months after 1. Stockfish (with NNUE) is definitely ahead of Lc0's net.

Current Stockfish (211009) vs. the 1st large T60, and current T60 net:

Code: Select all

3. Gauntlet - Sf-New vs T60 1st 30x384 (201207) & T60-New (211010)
   # PLAYER                    :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 Stockfish_21100909        :     0.0   ----   271.0     400    68     100  188  166   46    42
   2 Lc0-0.28.0--1st-606512    :  -126.2   39.5    65.5     200    33      62   24   83   93    42
   3 Lc0-0.28.0_610207         :  -134.3   38.4    63.5     200    32     ---   22   83   95    42
White advantage = 12.04 +/- 13.60 | Draw rate (equal opponents) = 47.22 % +/- 2.67

Stockfish now is even further ahead. The Lc0 10 month old net did better than the current!?

Current Lc0 net vs. the first large T60, the same Lc0 nets as in 3.:

Code: Select all

4. Match - T60-New (211010) vs T60 1st 30x384 (201207)
   # PLAYER                    :  RATING  ERROR  POINTS  PLAYED   (%)  CFS(%)    W    D    L  D(%)
   1 Lc0-0.28.0_610207         :    45.8   32.7   113.0     200    56     100   58  110   32    55
   2 Lc0-0.28.0--1st-606512    :     0.0   ----    87.0     200    44     ---   32  110   58    55
White advantage = 7.13 +/- 18.07 | Draw rate (equal opponents) = 56.10 % +/- 3.11

As expected for Lc0 nets vs. each other: the newest is ahead - adding mystery to 3.

Four .pgn's in 1 folder: https://www.mediafire.com/folder/e8cy6m ... -T60_range

[Chess_Dragon]

Just a game: Stockfish @ G/5 secs + .1 sec vs. Lc0 G/15 secs + .25 sec .

Black looks strong on the long diagonal but white's pawn sac. 19.e4-e5 is winning, 19...dxe5 did not work.

The game is playable here, with evals.:
http://chess.tuxtown.net/game-replayer. ... 374ea697db