FIDE Rules on ICGA - Rybka controversy

[BB+]

And 60% is way too less for 2 identical evaluations,

[As hyatt notes, "identical" is not perfectly correct -- to the extent possible, the numerologies were coincided, but there are of course some elements of difference in the eval features]. I think it shows how much search (and/or strength) can really effect an engine's moves. If you simultaneously scaled the Rybka/Fruit search time for strength, then likely the 60% would increase. But I already think 60% is quite high, indeed some engines were only getting 65-70% in "self-similarity" (eg, due to time-slicing, see Footnote 36). Note that the Dailey set of positions is somewhat "positional" (see 4.4) in that there are not too many positions for which there is only one good option (better by a large margin). By looking at the reported CSVN numbers (generally lower than with the Dailey positions), I'd guess their test-set takes this idea further. I think whatever raw numbers you end up with, they have to be put in context, which is what we tried to do in the paper through the statistical analyses.

and as a next surprise the longer time you give an engine the higher the similarity of a suspect program becomes

This is discussed in the paper (mentioned in 6.2.4 in particular, see Table 3): One can note that engines do tend to agree more as time increases... Another point along these lines is with the "Botvinnik" engine (6.3.2), where he is maybe 2700 Elo and the engines at short times (100ms) are only 2300-2400, and this presumably is what causes the low move similarities.

Aside from that, this document is horribly bogus. All that "Rybka code" isn't Rybka code, it's just someone's imagination.

Which document would that have been?

Zach's document, which Levy linked in his first (rather terse) email to Rajlich.

[BB+]

Vas took the Fruit source code, wiped out the Fruit DNA, added 100 elo and released it under his own name.

This would also be my "most likely" reconstruction of events, though something like copying from one window to another is also a strong possibility (and the precise method of copying does not matter for most purposes in any case). Certainly he would then have had to splice in his material imbalance table, and probably it would be easier to use the bitboard code he already had rather than modify Fruit directly [on this and other points I disagree with FL's guess]. So maybe he created a Fruit/Rybka hybrid, and then over time wiped out the Fruit parts. Whatever the creation method, the various amounts of Fruit-like stuff in Rybka 1.0 make it hard for me to think that he didn't actually take some parts of the Fruit code, even in your sense.

And therefore they signed the letter. The heart of the complaint is not the Fruit ideas in Rybka, the heart of the complaint is that they believed Vas copied.

Fabien's open letter (and the ensuing TalkChess thread, linked to by the Programmer's Open Letter) already make it clear that it is not about "code copying" in the strictest sense, and the oft-mentioned bitboard/mailbox distinction again makes it clear that the base-level "code" cannot be exactly the same. Letouzey's statement (about Strelka) was: ...it was not a verbatim copy of the source code. All the code had been typed (can't say "designed" though, see below) by an individual. So legally there was no issue that I knew of. It was however a whole re-write (copy with different words if you like, similar to a translation) of the algorithms. Not just an extraction of a couple of ideas as is common, and normal. I really don't see how anyone could think that the Rybka/Fruit complaint is merely about "code copying", at the very least it had to be about a "translation", which as FL says is "copy[ing] with different words".

And NO PROGRAMMER took offense in 2005 because it's generally agreed it's okay to do that AS LONG as you do not COPY and many programmers by their own admission did the same, taking from Fruit.

No programmer took offense in 2005, partly because Rajlich assured everyone: As far as I know, Rybka has a very original search and evaluation framework. Furthermore, no one had much sense that Raljich had essentially re-used almost all of the Fruit evaluation features -- rather they thought he had taken some/many ideas here-and-there ("as is common", to quote FL from above), not a wholesale import of them.

Things changed when programmers started to believe Vas COPIED.

Things changed when people (thanks to disassembly) realised the great extent to which early Rybka versions were co-incident with Fruit, both in various coding quirks, and also in the evaluation function, to an extent much more than anyone else had expected (or would have done themselves). Things really changed when Fabien Letouzey found out about this.

And some smart dude found the stick to punish Vas for what he (allegedly) did (copying Fruit) namely the cryptic ICGA rule #2 that can be interpreted in various ways. And this is THE MAIN THING I have against the verdict, if you are so convinced Vas copied, then PROOF IT. And the Panel couldn't, see the verdict. That's hypocracy.

The Panel was convinced that Rybka was not original, rather had its origins in Fruit (and Crafty), and thus broke Rule #2. The exact form of "copying" (whether "code copying" or "copy[ing] with different words") is not the principal point. I don't think Rajlich himself would argue that the early Rybka evaluation function was merely "an extraction of a couple of ideas" from Fruit, even if you granted a somewhat large "couple" due to his forwards/backwards statement. Rather he used what was essentially an extraction of the whole Fruit evaluation function, with various modifications (some quite significant, like material imbalance) and tuned numerology.

To take an early example: As is well-known, the program SARGON was published in "open-source" form in 1978, as a book. It was subsequently ported to many different computers. Did any of these people who made such a port try to enter it into a tournament, claiming they were the (sole) "author", rather than Dan and Kathe Spracklen? To your mind, did any of them "copy" SARGON; would you call their work "original"?

22. The programs have different Zobrist hash keys.
23. The programs have different user interface options.

Well, at least these last two were so ridiculous (or added too late) that you didn't have Riis copy them over to his Entertainment Computing article.

[hyatt]

One MAJOR difference Ed seemed to have overlooked:

F - R - U - I - T

R - Y -B - K - A

But even that is not so cool, both have FIVE letters and they share 20% of those letters to boot.

[Rebel]

Who was it who (initially) put emails of Vas on a major chess site? [meaning ChessVibes]

Levy asked (in the name of the ICGA) for Rajlich's comment on the matter of Zach's analysis. Perhaps he should have made it more clear that it was for publication (maybe he did in a later email, I don't necessarily trust your website to have the totality of the correspondence -- for instance, your page lacks Rajlich's quoted email involving: all of the Rybka versions are original, in the sense that I always wrote the source code myself (with the standard exceptions like various low-level snippets, magic numbers, etc).).

The Vas quote, It was not there!, simple as that. Read what the Vas-Levy correspondence page states:

• Email exchange between David Levy and Vasik Rajlich
  As found on the ICGA website one year after the investigation

I double checked the orginal ICGA-WIKI download, it's from August 27, 2012.

It's not my fault the ICGA maintains a sloppy record, forgets to add emails, delete emails. The Vas quote (email?) in question is indeed in the Levy attack on Chessvibes, but NOT in the downloaded version of August 27, 2012. And after that date they canged the contents too, the Vas phone number email I told you about. GONE! Whatever the wiki stats tell you. If you care I would say, take it up with your ICGA friends.

It seems you are now on a level of total mistrust in my person and I don't give a hood. You can go to Rybka forum and ask, there are folks who record everything and surely some of them will have the August 27, 2012 download on their hard disk.

Who was it who put the whole correspondence on the icga website for everybody to see? By your logic David Levy is not decent.

Email correspondence, particularly as a historical proceedings, is different than clipping whatever parts from people's emails. For instance, I would consider it quite regular for you to put up your emails (as Rajlich's representative) to Levy, and indeed also his responses. A bit strangely however, your website only has his responses. Did Levy just spew out these emails spontaneously?

Out of context huh? And you know it. If you are just trying to annoy me, won't happen.

It's still there BTW [ http://icga.wikispaces.com/file/view/le ... thread.txt ]

This is all you mean??? This is just the responses Rajlich gave (or didn't give) to the investigative Report. The first email there is May 13th. All the emails are technically from Levy, though of course contain Rajlich's words as copied in reply.

At least we back what you started. You are still glossing over the copyright breaches of private email by David for what he thought was important for his case against Vas. As long as you don't condemn Levy for that then you have no point blaming me for occasionally doing similar what I thought was right, to expose the dirty laundry of the ICGA. Make sense? Or is it so hard to say?

[Rebel]

In my eyes we had an agreement to have a debate about the AFC test ... in email and in principle meant for publication.

I don't recall any such "agreement" per se. See below.

And if memory served me well (Chris may correct me) you quit after the second round without giving a reason.

We had one preliminary email exchange [thus I quit after the first round, not the second], with each side (ostensibly) trying to write the opponent's views. I am pretty sure I gave a reason for quitting. Just give me a few minutes while I search your website to find my email....

I can appreciate good humor

OK, here we go, intertwined with some emails from/concerning Dalke/Riis, on Feb 23 2012 you "invited [me] into such a debate" with ChrisW regarding AFC. The email was addressed to me and Dalke, and the CC list was Riis and Whittington (by this point, I think Dalke and I had broke off contact in any case, while Riis was more interested in my work on Losing Chess). Approximately one hour also later you sent an email regarding Rybka/Fruit and a statement of Zach's saying: I prefer to end our conversation after reading your answer. I replied: I think I prefer to end our conversations also.

Oh really

This is about your complaint to Chessbase about Soren's article and you were all worked up about something Soren had said and you were quite demanding to Chessbase to make changes, then you asked me to do the same because I offered the 4 PDF's for download, or so. I believe I refused. Seriously, Levy was offered a rebuttal by the Chessbase folks which he took and basically you did most of the work your chance to respond. I realy didn't understand why you dragged me (and notable Dalke) into that discussion but it was quite revealing how sensitive you often are. But indeed, the case became the inducement for the AFC debate.

Regarding the AFC debate,

I am not willing to go through all the emails and trust your perception isn't too far from mine other than that Dalke surely was not involved and regarding my comment about Bob you forget to mention he was quoting you from the "private" panel talks in public. Did he had your permission? If not, that would have been leaking.

**edit ** I looked it up, I was praising you there why make a snide remark over it now?

http://74.220.23.57/forum/viewtopic.php?p=456992#456992

[Chris Whittington]

In my eyes we had an agreement to have a debate about the AFC test ... in email and in principle meant for publication.

I don't recall any such "agreement" per se. See below.

And if memory served me well (Chris may correct me) you quit after the second round without giving a reason.

We had one preliminary email exchange [thus I quit after the first round, not the second], with each side (ostensibly) trying to write the opponent's views. I am pretty sure I gave a reason for quitting. Just give me a few minutes while I search your website to find my email....

I can appreciate good humor

OK, here we go, intertwined with some emails from/concerning Dalke/Riis, on Feb 23 2012 you "invited [me] into such a debate" with ChrisW regarding AFC. The email was addressed to me and Dalke, and the CC list was Riis and Whittington (by this point, I think Dalke and I had broke off contact in any case, while Riis was more interested in my work on Losing Chess). Approximately one hour also later you sent an email regarding Rybka/Fruit and a statement of Zach's saying: I prefer to end our conversation after reading your answer. I replied: I think I prefer to end our conversations also.

Oh really

This is about your complaint to Chessbase about Soren's article and you were all worked up about something Soren had said and you were quite demanding to Chessbase to make changes, then you asked me to do the same because I offered the 4 PDF's for download, or so. I believe I refused. Seriously, Levy was offered a rebuttal by the Chessbase folks which he took and basically you did most of the work your chance to respond. I realy didn't understand why you dragged me (and notable Dalke) into that discussion but it was quite revealing how sensitive you often are. But indeed, the case became the inducement for the AFC debate.

Regarding the AFC debate,

I am not willing to go through all the emails and trust your perception isn't too far from mine other than that Dalke surely was not involved and regarding my comment about Bob you forget to mention he was quoting you from the "private" panel talks in public. Did he had your permission? If not, that would have been leaking.

**edit ** I looked it up, I was praising you there why make a snide remark over it now?

http://74.220.23.57/forum/viewtopic.php?p=456992#456992

The AFC breakdown was probably my fault. The idea of both sides starting by summarising what they thought the other side thought was a good one. My credit. But. We don't like each other's style, that's clear. I read the Mark Watkin's summary of my "thoughts" and was annoyed, basically at the amount of work required to hack through the extensive verbiage to pick out the points, some of which didn't seem to be my thoughts anyway, and surely expressed in a way I would never express them Thus, my reply was kind of fast and terse, partly because I think that way (which I suspect Mark finds equally annoying), a partly because I didn't feel it was going to go anywhere. Mea culpa, should have persisted. As to the occasional "psychology" or what gets referred to as psychology - well, theory of mind is important, why the participants think what they think (pre-notion, bias, tendency to naturally fall into particular camps), because, ultimately, one can read into the mass of data and documents pretty much what one wants, it's almost as complex as is there or isn't there an afterlife, or should I vote left or right.

It also annoys me the way this conversation here goes. It's all just a bit too personal, as if whether Levy leaking email or Ed quoting email (and btw I am quite convinced it was NOT Ed that sent out the panel deliberations) was more important than the extreme bias (obviously secretariat/panel, but also Ronald's views on the Attack of the Clones article). And with bias at that level, should not the verdict be set aside, and, if necessary, reviewed by a non-partisan group, or even a balanced group of both-sided partisans? Rhetorical. Of course it should, the question is how to set up the review.

[Rebel]

And 60% is way too less for 2 identical evaluations,

[As hyatt notes, "identical" is not perfectly correct -- to the extent possible, the numerologies were coincided, but there are of course some elements of difference in the eval features]. I think it shows how much search (and/or strength) can really effect an engine's moves.

Sure, but like I stated the effect of search is surprisingly low. I quickly went once more to your document and couldn't find an experiment that would give an indication of that. Neither can I offer the results of my experiments in this area. I probably posted them at Talkchess EO at the time and forgot to save it. Anyway, by head, I made several fundamental changes to search in mine (excluding Nullmove, LMR, futility pruning) and compared them with the SYM tool. The similarity remained extremely high, evidence EVAL still is the beating heart of a chess program. Anyway, I think more research in this area is needed to come to some sort of consensus regarding the effets of search. The results of one engine (mine) is an indication but certainly not necessarily a pattern.

I would say it's crucial to judge the 60% similarity of Fruit with the Rybka values vs Rybka. Especially in the light of Zach's statement - Simply put, Rybka's evaluation is virtually identical to Fruit's 60% looks bad. Or it must be the influence of Rybka's MIT, I can hardly imagine it would make a big difference, but did you cancel the MIT in your test?

Need to go, dinner time. Will pick up the rest of your post later.

[BB+]

From the 16 who signed I checked 8 of whom I had a sort of relationship then, or in the past and all of them believed Vas did exactly that, the Fabien accusation, that Vas took the Fruit source code, wiped out the Fruit DNA, added 100 elo and released it under his own name. And therefore they signed the letter.

Of the 16, I think the following 10 quite well understand the "copy with other words" sense of the Rule #2 violation: Letouzey, Wegner, Uniacke, Meyer-Kahlen, Dailey, Pijl, Ban, Cozzie, Isenberg, Zwanzger. [I have physically talked to 7 of those 10, with Uniacke/Dailey/Cozzie only by other means]. I would also include Romstad on this list, but I think he only posted a couple of lines post-verdict on the subject. This leaves Schröder, Theron, Schäfer, Bushinsky, and Böhm. I still respect the privacy regarding "votes" in the Panel, but I am sure that you can determine which (if any) of these last 5 participated in that vote.

As I indicated in my previous post, one can both think that Rajlich "took" Fruit, etc., yet simultaneously realise that the extent of this "taking" is unprovable (without source code), and that the "code with other words" standard already suffices for Rule #2. If you want the "court" analogy, you might think it is quite probable that a defendant was guilty of a greater crime, but also realise that the supporting evidence is strongest for a lesser (though still substantial) one.

[BB+]

As found on the ICGA website one year after the investigation

OK, as I think you can tell, I am trying my best to verify the details of this. What do you mean by "the ICGA website"? Is this the icga.wikispaces.com wiki, or the icga.leidenuniv.nl site, or? The wiki history has no information about relevant changes/deletions around Aug 27, 2012 (or more recently), and searching the entire history from start to current did not show anything either. The specific file you indicated very clearly says that it was uploaded on June 16, 2011, has not been modified since, and (unlike your webpage) has only the concluding part of the correspondence.

You are still glossing over the copyright breaches of private email by David for what he thought was important for his case against Vas.

You still have not provided any evidence of publishing of private e-mail [which I would not necessarily term "copyright breaches", either in general or the case given here], either by Levy or Lefler (or the ICGA). Given the lack of evidence, it is easy to "gloss over" the attendant conclusion of the hypothetical.

It's not my fault the ICGA maintains a sloppy record, forgets to add emails, delete emails. ... Whatever the wiki stats tell you. If you care I would say, take it up with your ICGA friends.

Actori incumbit probatio. I reiterate, despite my best efforts, I have found no evidence of your claim, and indeed the evidence that exists seems to contradict it. [And you can surely expect me to make a sniping remark that this isn't the first time I've run into such a problem with some of your claims ].

You can go to Rybka forum and ask, there are folks who record everything and surely some of them will have the August 27, 2012 download on their hard disk.

Aren't the folks at the Rybka forum rather "biased"? From where would they have obtained this download: from the ICGA "wiki", the ICGA "website", relayed from you... ?

Anyway, I interpret your "August 27, 2012 download" to mean that you downloaded everything from some ICGA site at that date. Your EC complaint is dated August 31, cites the above file that has been unmodified since June 16, 2011 and says since August 27, 2012 these emails were removed, however a copy is available including witnesses. So you are claiming that in the intervening 4 days the wiki (somehow) changed? [Incidentally, if I am interpreting this correctly, the Complaint's footnote 17 (regarding the "voting") says that: The ICGA’s web-page covering this topic was blocked August 27, 2012, but in reality said information was always "blocked" to the public -- yet another reason that I find the various claims involved herein to be rather poorly supported].

[hyatt]

And 60% is way too less for 2 identical evaluations,

[As hyatt notes, "identical" is not perfectly correct -- to the extent possible, the numerologies were coincided, but there are of course some elements of difference in the eval features]. I think it shows how much search (and/or strength) can really effect an engine's moves.

Sure, but like I stated the effect of search is surprisingly low. I quickly went once more to your document and couldn't find an experiment that would give an indication of that. Neither can I offer the results of my experiments in this area. I probably posted them at Talkchess EO at the time and forgot to save it. Anyway, by head, I made several fundamental changes to search in mine (excluding Nullmove, LMR, futility pruning) and compared them with the SYM tool. The similarity remained extremely high, evidence EVAL still is the beating heart of a chess program. Anyway, I think more research in this area is needed to come to some sort of consensus regarding the effets of search. The results of one engine (mine) is an indication but certainly not necessarily a pattern.

I would say it's crucial to judge the 60% similarity of Fruit with the Rybka values vs Rybka. Especially in the light of Zach's statement - Simply put, Rybka's evaluation is virtually identical to Fruit's 60% looks bad. Or it must be the influence of Rybka's MIT, I can hardly imagine it would make a big difference, but did you cancel the MIT in your test?

Need to go, dinner time. Will pick up the rest of your post later.

I don't find the effects of search "quite low". If you change one thing, perhaps. But add singular extensions and see how THAT changes the overall tenor of the program's play. Or Donninger's threat extension. Or whack away on the LMR reduction matrix. Even changing the history counter update approach will significantly change both scores and moves played, thanks to LMR. How do I know? I have been debugging this very issue of late. A subtle change that causes the program to not update the killer moves when ply-1 completes, changes move ordering, which changes history counters thanks to LMR/LMP changes, which changes scores and PVs, all from not updating ONE killer move at the end of an iteration...

Which means ponder hit is certainly not so useful. Doing fixed-depth position comparisons gets a bit closer to the truth, but the search STILL has a powerful influence on moves and scores when LMR and such are factored in. Tiny changes balloon into major differences.