Programmers Open Letter to ICGA on Rybka/Fruit

[Rebel]

Not one of my proudest moments. I was simply dead wrong. Happens to most of us sometimes.

Same here.

By the way, this thread proves that you need to block the possibility to use colored text on this message board.

LOL

[BB+]

Generally speaking, an engine is probably not very good if a trivial change results in a large improvement. There is certainly no reason to think such a change would put it ahead of the rest of the field. But Rybka did achieve this, against Toga, commercial Fruit and all others, for a few years.

For the 2nd point, Rybka 1.0 Beta and commercial Fruit are about the same Elo level in their 32-bit versions. For the first point, the history of the development of Fruit seems to imply that Letouzey made no particular effort to optimise strength in Fruit 2.1 [for one (as he told me himself), he left in various "development-oriented" rather than "performance-oriented" code -- there are other comments about the Fruit history that I made here]. Particularly from the Fruit 2.1 readme:

Although I believe I could keep on increasing strength by adding more and more eval terms, I have little interest in doing so. I would not learn anything in the process, unless I develop new tuning/testing techniques. Ideally I would like to spend more time in alternative software, like my own GUI perhaps (specific to engine testing/matches).

So if you conclude from this that Fruit was "not very good" and thus rather suspectible to improvement, I guess I would agree. As I noted in the post linked above, Thomas Gaksch gained ~125 Elo over Fruit 2.1 while working essentially on a "hobby" basis.

In software engineering, lines of code is considered a poor measure of programmer performance.

The question at hand is not about programmer performance. I should think that "lines of code" would be one measure of originality, which is more relevant here.

At any rate, I hope you realize that you must subject other ICGA contestants to the same BB+ -style derivation analysis in order to obtain a prior for the judgment concerning Rybka...

I can't agree with the "must" here, as it will depend on the direction the ICGA process takes. If an issue is raised as to whether the Rybka-Fruit overlap could fall under "accepted practises" in the field (and who knows -- Rajlich might get a number of programmers to sign a letter stating this, for all I know), then I agree that this could be useful.

But outside of something like this, and without a specific complaint against any other contestants, I don't see any pressing reason to undertake such an analysis. I would expect that most CC programmers are already capable of stating whether they think the current information on the Rybka/Fruit overlap is sufficient to render Rybka 1.0 Beta non-original, and I don't think a Fruit/XYZ analysis would shed any additional light on this originality question [whether or not it would help the public understand the issues better is a different matter -- also, it seems to me that having XYZ be Crafty (or Stockfish, though it's never competed in an ICGA event) would suffice].

I think that using HIARCS or Shredder for this purpose would be an excellent idea. Of course, it's not my time...

Doing a complete ASM-based analysis (with no "Osipov code" to follow) would be a major pain. But I suspect that either Uniacke or Meyer-Kahlen would agree to allow some independent ICGA inspector to see their source code if it came to that [whether or not the "source code" would be the prime item of study, or merely a guide to a ASM-based study (so as to try to replicate the conditions with Rybka) does not seem to me to matter much, but outsiders seem to perceive there is some distinction between the two]. It would also likely be necessary to redact most of the specifics of such an analysis, which would mean that the "public" would be reliant on the expert opinion of the investigator [this last point is again not crucial to me, though I can't imagine the nitpickers on various fora would fail to jump on it].

[BB+]

There are many reasons for that, but it is important to realize that there is a world of difference between studying a program, finding a few tricks you can use, and implementing these techniques in your own program (which everybody agrees is OK) on the one hand, and using somebody else's complete program as the basis for your own work on the other hand. We're talking about the latter.

To further Tord's comment: I suspect that most computer chess programmers have from their own experiences a fairly clear view of matters such as "originality" with code/ideas, or (say) the extent to which a decompilation can be used as evidence, et cetera, and find the typical arguments raised by "outsiders" often to be at best tangential. This being said, whether or not said programmers agree with the conclusions expressed in the Open Letter is another matter [and there are some notable names missing from it].

There are many reasons for that, but it is important to realize that there is a world of difference [...]

If there's a world of difference it should be easy to prove. But there isn't, and it won't be.

I'm not quite sure what is meant here, but I would say that "yes" there is a world of difference between the use of code and the use of ideas, and "yes" one can detect the difference between the two in many cases. For instance, Rajlich was quite clear that code from Rybka 3 was taken in the IPPOLIT incident.

[clumma]

Rybka 1.0 Beta and commercial Fruit are about the same Elo level in their 32-bit versions.

True, but ICGA aside, I'll point out that Rajlich didn't attempt to profit from Rybka until it had a significant Elo lead.

For the first point, the history of the development of Fruit seems to imply that Letouzey made no particular effort to optimise strength in Fruit 2.1

Strange behavior for someone entering their engine in high-level tournaments.

At any rate, I hope you realize that you must subject other ICGA contestants to the same BB+ -style derivation analysis in order to obtain a prior for the judgment concerning Rybka...

I can't agree with the "must" here

Not a Bayesian I take it... -Carl

[BB+]

True, but ICGA aside, I'll point out that Rajlich didn't attempt to profit from Rybka until it had a significant Elo lead.

I don't have the exact date he got the sales details sorted out and was open for business, but already in early December 2005 he was announcing prices:

However, I am now working on this project full time, and much as I would like to just concentrate on the technical aspect, the truth is that there is now a business to run. The first commercial release will be as plain as can be - Rybka 1.0 standalone UCI engine, no GUI, no book, no copy protection, no engine capability not currently specified in the UCI protocol. The price will be 34 Euro. The original target date was Dec 16, but thanks to the incredible speed of the CEGT team this has been moved up to ASAP Additional announcements are forthcoming.

On Dec 13 2005, there was already someone complaining about the shareit! exchange rates. Do you consider the Elo lead Rybka had (over commercial Fruit) at this date to be significant?

http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Rybka 1.0 Beta 32-bit	2893	+14	−14	65.2%	−108.5	30.8%	2061
Fruit 05/11/03 	        2888	+12	−12	49.8%	+1.3	34.5%	2637

http://www.husvankempen.de/nunn/40_40%2 ... liste.html

340 	Fruit 05/11/03       	2819 	13 	13 	1724 	50.5% 	2815 	40.3%
351 	Rybka 1.0 Beta w32 	2815 	6 	6 	7357 	59.3% 	2750 	34.7%

http://iggor.110mb.com/ratings/NIL42_alph.txt

0    12 Fruit 05/11/03 HD1-2 UCI       : 2803   21  21   558    52.0 %   2789   47.7 %
0    21 Rybka 1.0 Beta UCI             : 2757   20  20   743    54.2 %   2728   35.4 %

[kingliveson]

[...] Do you consider the Elo lead Rybka had (over commercial Fruit) at this date to be significant?

http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Rybka 1.0 Beta 32-bit	2893	+14	−14	65.2%	−108.5	30.8%	2061
Fruit 05/11/03 	        2888	+12	−12	49.8%	+1.3	34.5%	2637

http://www.husvankempen.de/nunn/40_40%2 ... liste.html

340 	Fruit 05/11/03       	2819 	13 	13 	1724 	50.5% 	2815 	40.3%
351 	Rybka 1.0 Beta w32 	2815 	6 	6 	7357 	59.3% 	2750 	34.7%

http://iggor.110mb.com/ratings/NIL42_alph.txt

0    12 Fruit 05/11/03 HD1-2 UCI       : 2803   21  21   558    52.0 %   2789   47.7 %
0    21 Rybka 1.0 Beta UCI             : 2757   20  20   743    54.2 %   2728   35.4 %

To my surprise, when I brought up the Elo difference last year, no one picked up on it.

On a different subject: Serious concerns have been raised regarding Bob being a judge on the ICGA panel that will render final opinion on the matter. These are legitimate complaints given the fact that he has already publicly expressed his opinion, and therefore must recuse himself.

Franklin

[Uly]

I think the Fruit 05/11/03 version was released at much later date.

[BB+]

You may be correct that the public release was later. This version seems simply to be the last beta from Fabien. It does date from Nov 2005, however. I don't know if the testing groups might have got this "beta" version earlier than public, or not. See http://www.fruitchess.com/purchase.htm

Fruit 2.2, 2.2.1, and the last beta from Fabien Fruit 05/11/03 (November 2005) are available for Windows, Linux and Mac OS X.

[BB+]

For the first point, the history of the development of Fruit seems to imply that Letouzey made no particular effort to optimise strength in Fruit 2.1

Strange behavior for someone entering their engine in high-level tournaments.

Note that Fabien did more optimisation with the version playing in Reykjavik. Fruit 2.1 was a development version.

Not a Bayesian I take it... -Carl

I can already see there is little Fruit/XYZ overlap for a number of XYZ's (such as Crafty and Stockfish). I don't see any compelling reason to undertake a similar comparison with Fruit/HIARCS or Fruit/Shredder.

[hyatt]

[...] Do you consider the Elo lead Rybka had (over commercial Fruit) at this date to be significant?

http://www.computerchess.org.uk/ccrl/40 ... t_all.html

Rybka 1.0 Beta 32-bit	2893	+14	−14	65.2%	−108.5	30.8%	2061
Fruit 05/11/03 	        2888	+12	−12	49.8%	+1.3	34.5%	2637

http://www.husvankempen.de/nunn/40_40%2 ... liste.html

340 	Fruit 05/11/03       	2819 	13 	13 	1724 	50.5% 	2815 	40.3%
351 	Rybka 1.0 Beta w32 	2815 	6 	6 	7357 	59.3% 	2750 	34.7%

http://iggor.110mb.com/ratings/NIL42_alph.txt

0    12 Fruit 05/11/03 HD1-2 UCI       : 2803   21  21   558    52.0 %   2789   47.7 %
0    21 Rybka 1.0 Beta UCI             : 2757   20  20   743    54.2 %   2728   35.4 %

To my surprise, when I brought up the Elo difference last year, no one picked up on it.

On a different subject: Serious concerns have been raised regarding Bob being a judge on the ICGA panel that will render final opinion on the matter. These are legitimate complaints given the fact that he has already publicly expressed his opinion, and therefore must recuse himself.

Franklin

Again, read David's letter. I do not get to render any "final opinion". Nor, in fact, any opinion at all. I am a member of a group of 3 that is going to conduct the investigation, giving Vas multiple chances to respond to each point of evidence presented by anyone. The three of us will then produce a report from this and forward it to David, where the ICGA will then convene a new group to decide what should be done in light of the evidence and rebuttals presented....