ChessBase: A Gross Miscarriage of Justice in Computer Chess

[BB+]

Here is (a copy of) the first Riis image:

riis02.gif (13.6 KiB) Viewed 1858 times

Already one can critique that "year-based" methodology (rounding to the nearest year) can be a bit distortive for some -- however, I have a specific question.

He makes a big deal out of Junior's post-Fruit jump here. Can anyone confirm his Junior numbers of ~2800 in 2005 and ~3000 in 2006? Maybe the latter is multicore? As noted above, I find Junior 9 (late 2004) and Junior 10 (mid 2006) to differ by 70-80 Elo, from CCRL/CEGT data. Also, Junior 11 was not released until 2009, so does Riis simply interpolate the 2007/8 numbers (and similarly with other engines)? Given that this graph forms a significant kernel of his argument, its genesis should be more adequately described. He claims the data are derived from "historical SSDF and CCRL data" (obviously erroneous for the 2004 Rybka datapoint, and others) -- maybe he splices one into the other at the juncture of interest (note that CCRL was founded about this time), and this causes the blip?

Incidentally, his claim of a "new paradigm" of "ultra-fast climbers" for the later years (2008-11) is a bit dubious, as many of the engines that he highlights under this moniker are flat-lining by now. The big jump seems to be in 2004-8, though even here his data rendition visually exaggerates the slope via the inclusion of early Rybka/Naum/Fruit versions (he should exclude them for 2004, at the least, particularly if there is the above SSDF-CCRL conjunction). Furthermore, given that the data do not seem to be hardware-uniformised (from what I can tell), for all I know, the "jumps" in the years of interest (2004-8) simply coincide with the mass-marketing of multicore machines. (E.g., he lists Rybka 4.1 x64 as 3128 in his [CEGT] table, and it is over 3200, nearly 3300, in the graph, so maybe the latter is multicore, whereas early Rybka datapoints would be single core)?

As I say, as his argument (its relevance aside) is largely based on these graphs, less sloppiness would be (much) appreciated. The graph above could almost be a caricature of how not to present data, given that it appears to: splice multiple datasets (two named, though at least one more must exist for Rybka/Naum 2004), round off release dates to the nearest year, linearly interpolate years for which there is no release, mix data across hardware differences, and include somewhat extraneous data from early years of some engines (but not others). This in addition to what I think are simply errors (e.g. the Rybka 2005 datapoint of ~2675), and the "casually sourced" (look it up yourself!) nature of it all. [Incidentally, it seems the earliest SSDF Rybka appearance is Rybka 1.0 Beta at 2773 on a slow 450Mhz computer, not much comparable to the CCRL normalisation. The earliest Naum is 3.1 (CCRL has 1.91 from early 2006), so the 2004/5 Naum datapoints do indeed appear contrived].

[Uly]

And that argument is supposed to make sense to someone?

The argument is that game playing code is much more CRUCIAL than time management code.

Give Rybka 1.0 Beta a subpar time management, and it's still going to beat the competition of the time, because its game playing code was much better (and thus, not just a copy).

Subpar doesn't mean non-existent, you can always claim that engines at Depth 1 play very weak chess, and that wouldn't prove anything.

Try Critter against critter 1 second to see what I was actually talking about. Timing decisions are CRUCIAL.

But Critter and critter have the same game playing code.

[BB+]

After perusing a bit further, all the 2004/5 datapoints for Rybka/Naum/Fruit given by Riis look contrived. His statement: The following two charts are derived from historical SSDF and CCRL data is simply fallacious for Naum in 2004/5 and Rybka/Fruit in 2004, as data does not exist for these in either SSDF or CCRL. For Rybka in 2005, his datapoint is ~2675, while CCRL lists a much larger number (~2900), and SSDF on a much slower machine is already ~2773.

For Fruit in 2005, he appears to have a datapoint a bit above 2700. CCRL 40/4 lists Fruit 2.2.1 (Oct 05) at 2851 and CCRL 40/40 has Fruit 2.1 at 2793. The datapoint of Riis looks off by almost 100 (if from CCRL). The principal Fruit listed in SSDF is 2.2.1 at 2831, again nowhere the ~2700 of Riis. So again I conclude his number comes from a unspecified source ((CEGT does have 2713, so maybe he borrowed that?). Not exactly Garbage In, but close. Once one plots his graph in a remotely sensible/correct manner, his "paradigmatic" argument seems essentially to evaporate.

All this being said, the sustained 85 Elo/year for 2.67 years by Rajilch (R1-R3) does seem w/o peer historically among engines at the top. This would have been a more direct argument to advance in behalf of VR's acumen (though wouldn't say anything about R1's originality, of course).

[BB+]

Another miss for the eagle-eyes behind Riis, failing to note he got a timeline wrong -- perhaps it ruined his narrative tale, or perhaps they simply don't care (emphasis added):

The next step on this path came in January 2008 when Strelka 2.0 was released, [...] Sometime thereafter someone with larcenous intent approached Rajlich via email posing as GM Larry Kaufman.

How is it that I, with just a bit of search, can find that the date of the latter event is more likely to be "early 2007"? http://rybkaforum.net/cgi-bin/rybkaforu ... 5#pid74545

Aside from the Strelkadata, there was an unintended release of the Rybka eval in early '07 due to someone fraudently pretending to be me.

Of course, this is marginally relevant (though it does pre-condition the Strelka story somewhat), but then I'd say his whole spiel is.

[BTW, "larceny" is not the proper word here, as VR would not be deprived of the use of his property -- this is analogously why copyright infringement is not "theft"].

[hyatt]

And that argument is supposed to make sense to someone?

The argument is that game playing code is much more CRUCIAL than time management code.

Give Rybka 1.0 Beta a subpar time management, and it's still going to beat the competition of the time, because its game playing code was much better (and thus, not just a copy).

Subpar doesn't mean non-existent, you can always claim that engines at Depth 1 play very weak chess, and that wouldn't prove anything.

Try Critter against critter 1 second to see what I was actually talking about. Timing decisions are CRUCIAL.

But Critter and critter have the same game playing code.

NOT if you break part of the "game playing code" by making one do a 1 second search regardless of the clock. The code that sets the time for the next search, decides when to stop early, and decides when to search longer IS a part of "the game-playing code". Last time I looked, the chess clock was an actual part of the game and is defined in the various rule-books that have been used since I first directed a tournament...

You want to exclude each piece of code Vas copied as "not part of the game-playing code." The ENTIRE ENGINE is "the game playing code" from opening book moves, endgame tablebase moves, and everything in between, including the UCI code since if you remove that, it will not be able to play...

[Uly]

The ENTIRE ENGINE is "the game playing code"

If this was true the ICGA wouldn't have needed to specify "game playing code" in their rule 2.

[hyatt]

Sure they did. They wanted to exclude the GUI, because those are being shared. Xboard for example...

Again, why are you trying to argue without having any information or understanding about what has transpired in the past?

Some of us did not like the "game playing code" exception, because GUIs have taken over more and more of the game. UCI as an example, where the GUI sets the time limit and tells the engine how long to search, when to stop, when to ponder, what to ponder, etc. Excepting xboard, many GUIs handle the book as well, including book learning. And endgame tables at the root. I think that for such GUIs, only ONE copy should be allowed. xboard does NOTHING about playing the game, it is a true "interface" simply passing moves and clock times back and forth as needed, making zero decisions about game-playing.

[Rebel]

He is repeating past false statements, and referring to the "king of false statements" by referencing Ed's web page which is intentionally misleading and unsound.

Can you elaborate? I didn't find anything false in Ed's pages.

One simple one.

Take one PST from Fruit. The initialization code can be modified by changing four constants, and EXACTLY produces the PST values for the same Rybka PST. Here is where Ed's page is blatantly wrong..

The PST's were generated with the provided code by Zach and Mark, confirm the values as found in Zach's document. And the Rybka PST's don't match Fruit, just one, the rook. Like the Fruit bishop PST found in Crafty, just one. Not 12 as you claimed and the documents imply.

Now that they are visualized it is clear for everybody to see.

Blame Zach, Mark, not me.

And I am waiting for an admission from Zach and Mark since August / September when the issue was debunked by Chris and Miguel.

[Rebel]

The ENTIRE ENGINE is "the game playing code"

If this was true the ICGA wouldn't have needed to specify "game playing code" in their rule 2.

On top of that it's widely known Rybka's time comtrol is Rybka's Achilles' heel, 'perhaps Vas can't even copy & paste properly

[hyatt]

He is repeating past false statements, and referring to the "king of false statements" by referencing Ed's web page which is intentionally misleading and unsound.

Can you elaborate? I didn't find anything false in Ed's pages.

One simple one.

Take one PST from Fruit. The initialization code can be modified by changing four constants, and EXACTLY produces the PST values for the same Rybka PST. Here is where Ed's page is blatantly wrong..

The PST's were generated with the provided code by Zach and Mark, confirm the values as found in Zach's document. And the Rybka PST's don't match Fruit, just one, the rook. Like the Fruit bishop PST found in Crafty, just one. Not 12 as you claimed and the documents imply.

Now that they are visualized it is clear for everybody to see.

Blame Zach, Mark, not me.

And I am waiting for an admission from Zach and Mark since August / September when the issue was debunked by Chris and Miguel.

Can you stop with the dishonesty. I will make this statement crystal clear so that you will not be able to twist it any longer:

If you take the PST initialization code for the knight, which I JUST published here, and you run it "as is", it produces the exact numbers that are used in Fruit's evaluation. If you change the 4 constants, as indicated in Zach's report, and run it again, it produces the EXACT NUMBERS that are in Rybka's hard-coded PST values. EXACT matches.

Of course the two sets of 64 numbers do not match value by value. The two programs use different material values for a pawn. So Vas tweaked those 4 constants to produce something that worked, probably by some tuning runs. But the FRUIT code, with FOUR changes, produces the EXACT Rybka numbers. Ditto for bishops. And rooks. And even pawns if you ignore the extra bonus for the center squares that are a single value added to the EXACT values produced by the Fruit PST initialization code with the constant changes given by Zach.

The numbers DO MATCH PERFECTLY. You sound like a complete idiot when you keep saying "you change those constants, the numbers don't match." Of course the numbers don't match 1 to 1 since the constants WERE changed. But the Fruit code produces the RYBKA numbers PERFECTLY with just the constant changes Zach gave. Time to stop twisting the truth and repeating a statement that is clearly false and intentionally misleading in the way it is presented.