Thoughts on Fruit=Rybka EVAL

General discussion about computer chess...
Jeremy Bernstein
Site Admin
Posts: 1226
Joined: Wed Jun 09, 2010 7:49 am
Real Name: Jeremy Bernstein
Location: Berlin, Germany
Contact:

Re: Thoughts on Fruit=Rybka EVAL

Post by Jeremy Bernstein » Thu Sep 01, 2011 2:14 pm

Uly wrote:@ernest: You're the only guy getting that joke, which makes it a bad one.
kingliveson wrote:It sounds like a broken record, but Ed, Chris, Ansari, Uly, and the rest of the doubters, why dont you guys ask Vas to release Rybka 1.0 or earlier source code?
Because we know he doesn't have it. Yes, we know he just deleted old source codes as soon as he had new stable ones. The Rybka codes are just no longer in this world, Vas has as much access to them as everyone else.

Now, Vas could show that the code of Rybka 4 is clean, but that would be irrelevant as he's being challenged on ancient versions (from where only 2.3.2a seems relevant), and because it's anybody's clue how much relation that code has with Rybka Cluster code (which would be the playing one).

And, anyway, all that would be moot, since I don't think Vas would ever play again on an ICGA event even if they apologize, and I think it'll be a loss for the ICGA as soon as a true World Championship is created for the audience, not for the programmers.
Why would the ICGA apologize? They simply enforced their rules.

And. Achieving the title of "World Champion" by cheating at the world championship is lame, and no self-respecting world championship would allow rule-breakers to walk away with the prize money, titles and advertising cred. Correcting that error, even years after the fact, is not only desirable, but necessary to maintain the integrity of the championship.

Rybka's not even the top-rated engine at the moment (yes, yes, clusterblahblahblah, but it's untested and unrated). So, once again, what exactly does the ICGA have to a) apologize for and b) fear?

I'd be more inclined to fear that Rybka is facing impending obsolescence, not the ICGA championship.

Jeremy

User avatar
Uly
Posts: 838
Joined: Thu Jun 10, 2010 5:33 am

Re: Thoughts on Fruit=Rybka EVAL

Post by Uly » Thu Sep 01, 2011 3:36 pm

Jeremy Bernstein wrote:Why would the ICGA apologize? They simply enforced their rules.
I mean, IF in the end it turns out Vas didn't break their rules. To me, the coin is still in the air.

But if it turns out Vas did cheat, I'll be the first one to apologize for doubting the ICGA's decision.

alfons
Posts: 58
Joined: Thu Jun 10, 2010 3:53 pm

Re: Thoughts on Fruit=Rybka EVAL

Post by alfons » Thu Sep 01, 2011 5:24 pm

But if it turns out Vas did cheat, I'll be the first one to apologize for doubting the ICGA's decision.
Yes, this seems crucial - but how 'else' could turn it out? I mean various papers got published, so would if it fair to say, that all are awaiting the confession of the miscreant eagerly? How could that ever be the case?

I have the feeling that the matter will be forever in limbo - at least in the rybkaforum...

hyatt
Posts: 1242
Joined: Thu Jun 10, 2010 2:13 am
Real Name: Bob Hyatt (Robert M. Hyatt)
Location: University of Alabama at Birmingham
Contact:

Re: Thoughts on Fruit=Rybka EVAL

Post by hyatt » Thu Sep 01, 2011 6:31 pm

Actually, quite a few on the Rybka Forum have changed their minds after seeing the evidence, and listening to the discussions. Just a very small group of die-hards remain, and their arguments are now getting "way out there" and not directed toward the evidence at all...

User avatar
kingliveson
Posts: 1388
Joined: Thu Jun 10, 2010 1:22 am
Real Name: Franklin Titus
Location: 28°32'1"N 81°22'33"W

Re: Thoughts on Fruit=Rybka EVAL

Post by kingliveson » Thu Sep 01, 2011 6:43 pm

Uly wrote:...
kingliveson wrote:It sounds like a broken record, but Ed, Chris, Ansari, Uly, and the rest of the doubters, why dont you guys ask Vas to release Rybka 1.0 or earlier source code?
Because we know he doesn't have it. Yes, we know he just deleted old source codes as soon as he had new stable ones. The Rybka codes are just no longer in this world, Vas has as much access to them as everyone else.
Uly, with all due respect, why is this lie continuously perpetuated? Lukas Cimiotti posted this recently:

Code: Select all

Vas and I discussed whether or not he should give source code to the ICGA. He really didn't like that idea. My idea was removing all comments and maybe changing all names of variables to make the code harder to understand. But as the guys that disassembled Rybka hadn't understood several parts of the code, we agreed it's safer to not give anything to our competitors.
So Vas only defended himself by saying: I did nothing wrong.
Image

Additionally, Vas later wrote when Strelka source was released:

Code: Select all

I've taken a look this morning at the Strelka 2.0 sources. The picture is quite clear.

Vast sections of these sources started their life as a decompiled Rybka 1.0. The traces of this are everywhere. The board representation is identical, and all sorts of absolutely unique Rybka code methods, bitboard tricks and even exact data tables are used throughout. Significant portions of the search and evaluation logic are not fully disassembled - the author has left in hardcoded constants and used generic names (such as "PawnStruScore0" & "PawnStruScore1", "PassedPawnValue0" through "PassedPawnValue7", etc) which show that he hasn't yet fully understood what is happening.
What do you suppose he was comparing Strelka source code to? Vas has stated he doesn't really care regarding "Rybka 1.0, which is ancient, incomplete, prototype-level code." So again, he wont be revealing anything new or secrets. We know he has the source, and I ask that you join, as well as encouraging others to call on Vas to release the source code.

Now, Vas could show that the code of Rybka 4 is clean, but that would be irrelevant as he's being challenged on ancient versions (from where only 2.3.2a seems relevant), and because it's anybody's clue how much relation that code has with Rybka Cluster code (which would be the playing one).
If one accrues wealth on stolen $100k invested "legitimately," does that negate fact of the dubious principal which the individual is/was under investigation? Sure Rybka 4 is probably clean, but let's not digress.
And, anyway, all that would be moot, since I don't think Vas would ever play again on an ICGA event even if they apologize, and I think it'll be a loss for the ICGA as soon as a true World Championship is created for the audience, not for the programmers.
Apology? Is there a joke hidden here somewhere? You can't be serious.
PAWN : Knight >> Bishop >> Rook >>Queen

BB+
Posts: 1484
Joined: Thu Jun 10, 2010 4:26 am

Re: Thoughts on Fruit=Rybka EVAL

Post by BB+ » Fri Sep 02, 2011 6:12 am

I will make two remarks. The Panel deliberately left it up to Rajlich [even with the 2007 statement of Cimotti] as to say whether any executables that it had were sufficient approximations to the relevant WCCC entries. Of most particular interest, naturally, was whether or not the EVAL in (say) R232/R232a was the same as that which played the games in WCCC 2007. E.g., it would obviously be quite irrelevant whether one of them had id name Rybka 2.3.2a and the other id name Rybka 2.3.2a-2007WCCC-build as UCI strings, or whether (like Naumov) there were watermarks involved, or whether one of them was named Rybka while the other was only called Rybka (??!), or ...

Secondly, the investigation of R232a (and indeed, many/most of the plethora of the intervening versions from R1 onward) was hardly an "afterthought", but rather the questions of versions "for ICGA purposes" was one of the most debated topics in the Panel. I think I personally glanced at almost everything Dann Corbit had available, and then focussed more on maybe 5 or 10 versions that seemed most relevant (like something temporally near WCCC 2006 -- though that was "Rajlich", not "Rybka" of course ;) ), and/or when changes to EVAL were made.

I could also note that, if consensus building were a desideratum, Bob Hyatt is about the least likely person in computer chess who I could nominate :mrgreen: -- indeed, internally in the Panel he was his usual "lightning rod" for contention, if for no other reason, because no one could bear to see him be correct. :)

Finally, the question of R4 and its use Pradu's magic multiplication code was discussed. It was firstly not clear whether the WCCC entry also used Pradu's code, and furthermore this was considered by some to be more like "black box" code.

BB+
Posts: 1484
Joined: Thu Jun 10, 2010 4:26 am

Re: Thoughts on Fruit=Rybka EVAL

Post by BB+ » Fri Sep 02, 2011 11:58 am

On another issue, one main reason why the Panel chose to start at R1 was that this was the Rybka version that Fabien specifically claimed was Fruit-derivative (recall that he was the genesis of the complaint to the ICGA), and also that various of the technical evidence had already been produced for this version. If Pradu Kannan had claimed that R4 was Buzz-derivative [he was contacted, and said the usage was OK by him], the investigation might have gone the other direction. :P

The Open Letter actually stated that R1 might be enough by itself(!): It is very likely that later Rybka versions have derived and benefited from Rybka 1.0 beta and hence in the circumstances our view is they should also be considered derivatives of Fruit 2.1 until proven otherwise -- but the consensus in the Panel was that an approximate WCCC entry should be investigated, and preferably one that won the WCCC. The Panel then worked forward through Rybka versions [and backward, when the R161 evidence arrived], until the approximate 2007 WCCC winner was reached. All the R1-R232a versions were found to have (at the very least) the same problem of "evaluation overlap" with Fruit 2.1, with perhaps the later versions decreasing this very slightly. In particular, the Rybka 2.3.2a version from the 2007 WCCC specifically had its evaluation function completely analysed, and this was found to be "nonoriginal" with respect to the Fruit 2.1 evaluation -- it is simply incorrect to say that "almost everything (95%?) was based on R1beta" [and indeed, it would have saved me many hours of evidence tracking/presentation if that had been the case].

As others have noted, there has been some "private" comparisons of R3 and later, but these were not considered in depth by the Panel for various reasons. Firstly, there is no longer as much reason to assume that R3/R4 have much resemblance to the 2008-10 WCCC entries as with R232(a)/2007. Secondly, there was nothing "obvious" to point to R3/R4 being a derivative (other than the Buzz/R4 issue), and enumerating evidence to prove a negative is often quite tedious. I was willing to make a statement that the evaluation in R3 (at least), having been re-written by LK, was likely now OK [though it seems that LK's name did not appear on the Rybka entry, or at least the ICGA never listed him as being a Rybka co-author -- but I digress], but to evidence this before the Panel would have required an enumeration of the R3 eval function, which seemed imprudent w/o VR giving an OK. It seems that minor UCI parsing peculiarities (moves[-1] and "0.0", if indeed either is actually copied) still remain in R3. Some (not all) of the PST patterns remain the same, LK didn't modify every evaluation consideration, etc. At the very least there would be something to discuss, as to whether there was still too much Fruit-influence.

Indeed to verify (say) R3 to the level of investigation as with the rest of the Panel deliberations would have involved showing/perusing large amounts of R3 disassembly, to ensure that everything was indeed "original". Furthermore, R3 is not freely available, and this could be a conflict for various Panel members (e.g., ZW does not own a copy, and is now a competitor to Rybka). And (as noted), there was almost a 2-month gap from R3 release to WCCC 2008, with various improvements made, such as bugs being fixed in private versions. For that matter, it's not completely insane [well, at least compared to some other theories] to suggest that R3 was "clean" due to [say] ChessBase concerns(!), but the "private" Rybka versions retained derivative parts (Fruit or otherwise). :roll:

The Panel chose instead to leave the question open, and let the Board decide what to do (e.g., request further investigation if they deemed that necessary for 2008-10). The Report lists various suggestions (ranging from Gerd Isenberg's to Mark Uniacke's) in this regard. When VR did not make a substantive response, the Board opted to close the case with their verdict. I will again link to the Wikipedia page on evidential burden, and note that Rajlich was disqualified and banned based on his behaviour, particularly with Fruit-nonoriginality for the [approximate] 2006-7 WCCC entries, and his ongoing refusal to address Rybka's origins (both with Crafty and Fruit) to the satisfaction of the Board with both these or later entries.

As #1 in the Verdict says: Vasik Rajlich is hereby disqualified from the World Computer Chess Championships (WCCC) of 2006, 2007, 2008, 2009 and 2010, and only then does it enumerate the consequences for his entries therein. One might also note that the Panel statutes say nothing about the ICGA or even tournament participation(!):
The purpose of the Panel shall be to:
[a] Investigate and discuss allegations of cloning or creating a derivative of strategy games programs;
Report to the ICGA as to the veracity or otherwise of such allegations;
[c] Make recommendations to the ICGA as to what action if any should be taken against those found by the Panel to have been guilty of cloning or creating a derivative;
[d] Publish the findings of the Panel.
Indeed, whether an ICGA entry was necessary [either in general, or with respect to recommended actions] was an issue the Panel debated. The case of the KCC Paduk programmer being denied entry in 2009, due to events of a decade prior that were never addressed, is one example of precedent here.

I personally think the verdict was harsh, but inevitable in the circumstances. Given the mountainous evidence for R1-R232a, it seems unreasonable for the onus of evidence to remain with the ICGA, and contrarily seems quite reasonable for Rajlich to be requested to (say) show source code [e.g., privately to Ken Thompson, or some other distinguished person] for at least some later version, be it R3/R4/cluster or what. As this seems to be a non-starter from what Lukas Cimotti has said, the verdict was, as I say, inevitable.

User avatar
Uly
Posts: 838
Joined: Thu Jun 10, 2010 5:33 am

Re: Thoughts on Fruit=Rybka EVAL

Post by Uly » Fri Sep 02, 2011 5:29 pm

kingliveson wrote:Uly, with all due respect, why is this lie continuously perpetuated?
Because it's true.
Lukas Cimiotti wrote: Vas and I discussed whether or not he should give source code to the ICGA. He really didn't like that idea.
It goes like this (an example since nobody knows the exact details):

You have the source code of version 1.0
You make a change for version 1.01 - It's buggy
You make a change for version 1.02 - It's buggy
[...]
You make a change for version 1.05 - It's stable
You make a change for version 1.06 - It's buggy
You make a change for version 1.07 - It's buggy
[...]
You make a change for version 1.1 - It's stable
You don't need anymore the codes for versions 1.0-1.04, you delete them.

You still have the sources of versions 1.05-1.1.

But when you are at version 1.5, you have deleted those and all non-recent source codes.

In this example, 1.1's source code it's what I'd refer to as "current source", i.e. the source of the last stable version.

I've seen people say that what Vas was doing was stupid but I don't see why, for years he didn't need at all the source code of old versions, it was only when he broke time management that it turned out the source code of Rybka 3 would have come in handy.

Lukas was talking about current source, and I'm glad they didn't give it since proving THAT version of Rybka was clean wouldn't have changed anything.
kingliveson wrote:What do you suppose he was comparing Strelka source code to?
Current source
kingliveson wrote:We know he has the source.
He doesn't, he has just stopped deleting source codes recently, so he has current source and the source of Rybka 4(.1).
kingliveson wrote: Sure Rybka 4 is probably clean, but let's not digress.
Fact is that in such case they just banned a legitimate program that doesn't violate their rules.
kingliveson wrote: Apology? Is there a joke hidden here somewhere? You can't be serious.
I'm serious, only close minded people would say "the chance of Vas being innocent is 0", and I doubt they would bet their life on it.

hyatt
Posts: 1242
Joined: Thu Jun 10, 2010 2:13 am
Real Name: Bob Hyatt (Robert M. Hyatt)
Location: University of Alabama at Birmingham
Contact:

Re: Thoughts on Fruit=Rybka EVAL

Post by hyatt » Fri Sep 02, 2011 9:30 pm

Uly wrote:
kingliveson wrote:Uly, with all due respect, why is this lie continuously perpetuated?
Because it's true.
Lukas Cimiotti wrote: Vas and I discussed whether or not he should give source code to the ICGA. He really didn't like that idea.
It goes like this (an example since nobody knows the exact details):

You have the source code of version 1.0
You make a change for version 1.01 - It's buggy
You make a change for version 1.02 - It's buggy
[...]
You make a change for version 1.05 - It's stable
You make a change for version 1.06 - It's buggy
You make a change for version 1.07 - It's buggy
[...]
You make a change for version 1.1 - It's stable
You don't need anymore the codes for versions 1.0-1.04, you delete them.
Nice fairy tale. But any reasonable computer scientist does not work like that. Have you ever noticed how many incremental versions of Crafty are on my ftp box? Care to guess how many intermediate versions there were between those released versions (and there are a LOT of released versions, by the way)? I keep 'em all. If I add a small feature and it doesn't work, and I tweak it and it does, I likely only save that last version, but each version only addresses ONE thing. And when it is completed, it is saved. Software Engineering 101 explains that idea in great detail.

You still have the sources of versions 1.05-1.1.

But when you are at version 1.5, you have deleted those and all non-recent source codes.
Only if you are an idiot software developer.


In this example, 1.1's source code it's what I'd refer to as "current source", i.e. the source of the last stable version.

I've seen people say that what Vas was doing was stupid but I don't see why, for years he didn't need at all the source code of old versions, it was only when he broke time management that it turned out the source code of Rybka 3 would have come in handy.

Lukas was talking about current source, and I'm glad they didn't give it since proving THAT version of Rybka was clean wouldn't have changed anything.
kingliveson wrote:What do you suppose he was comparing Strelka source code to?
Current source
kingliveson wrote:We know he has the source.
He doesn't, he has just stopped deleting source codes recently, so he has current source and the source of Rybka 4(.1).
Don't be naive. If one goes to MIT, which has a "fair" computer science program, one is forced to be exposed to modern software development ideas. Keeping old versions is a very basic part of that, because you never know when you will want to go back to one of those old versions to grab some code that you deleted from later versions. This concept just doesn't pass the "stink test" at all, sorry.

kingliveson wrote: Sure Rybka 4 is probably clean, but let's not digress.
Fact is that in such case they just banned a legitimate program that doesn't violate their rules.
Fatal mistake there. The ICGA doesn't "ban" a program. It "bans" an author. If the author is caught cheating, he can be banned, retroactively, and it is not a "partial ban". Which means that even clean programs could have wins/titles vacated. This is about the author's behavior, using his programs as the evidence to prove that behavior occurred, and then the punishment is again handed down to the author only. I'm not sure how one would "punish" a piece of software, maybe requiring that it spend 2 years running under Windows or something?
kingliveson wrote: Apology? Is there a joke hidden here somewhere? You can't be serious.
I'm serious, only close minded people would say "the chance of Vas being innocent is 0", and I doubt they would bet their life on it.
The chance really is zero. That horse left the barn with 1.6.1, without even discussing later versions.

User avatar
Uly
Posts: 838
Joined: Thu Jun 10, 2010 5:33 am

Re: Thoughts on Fruit=Rybka EVAL

Post by Uly » Sat Sep 03, 2011 4:47 pm

hyatt wrote:because you never know when you will want to go back to one of those old versions to grab some code that you deleted from later versions.
And Vas never needed to do that until the first Rybka 4 version (based on Rybka Cluster code that manages its time with a different concept and can't just be used for UCI Rybka) in where time management no longer worked and he had to rewrite it, and the new rewrite was very problematic. Weeks were lost testing the new time management code, tweaking it and finding optimal parameters for it, and in the end in the worst cases Rybka 4's time management is so bad it's costing it 40 elo or more in some conditions.

If Vas had any source code of any Pre-Cluster Rybka version, it would have been trivial to copy its TM code to Rybka 4. But as he didn't, he had to code new one from scratch, which turned out to be terrible.

Since then Vas has stopped deleting code, but he could have continued deleting it if the incident didn't happen.

Post Reply