Thoughts on Fruit=Rybka EVAL

[Chris Whittington]

Indeed, you said well, the emphasis is on occasion. Not within a time frame of 5½ months having tons of other things on your mind among that an elo improvement of 100 elo.

I would think that any such cosmetic re-arranging (including unit-testing) would take a day at most, but as you're the professional in computer chess engine development, you might like to correct me if wrong. It also seems (see node counts and depth) that Rajlich did in fact have "obfuscation" as one of the things on his mind during this period [though maybe just externally]. And as above, it's not clear to me (at this stage) that the EVAL is really re-ordered to much extent beyond the necessary from bitboard/performance issues. Finally, I discussed/rebutted your contention that 5.5 months for 100 Elo was "too short", and essentially got a non sequitur in response.

One can also note that Rybka gained 110 Elo (2816 to 2928 on CEGT 40/20) from Rybka 1.0 Beta (Dec 5 2005) to Rybka 1.2f (May 5 2006), again a 5 month period. Is your argument now going to be that this latter number just shows how Vas is a genius at adding ELO?

Perhaps there are valid reasons the ICGA did not, illness, not enough man power, etc. and I understand the ICGA is not a multi-million dollar organization with unlimited resources, it's not my intention to scoffer. Fact remains it should have happened and it did not. As such the ICGA became part of the problem. This whole drama could have been avoided right after the 2006 WC cycle with much less consequences and emotional impact.

Just to keep this humorous: how about this conspiracy theory? VR bribed the ICGA Board in 2006, and only now with the new Board (from the 2008 Meeting) has there started to be a clean-up campaign! So the current ICGA is not the problem, just the corrupt one from 5 years ago. Have you considered this possibility? I might note that ChrisW (back in March, after relating his opinion of the dysfunctional aspects of the CC world -- no argument from me there!) essentially asked me not to take part in the tribunal/Panel [and instead "work with others to bring peace in the face of the hate"] -- should I interpret this as "shut up"?

Perhaps it was foolish of me to assume that there were standards here in open chess which meant that you might ask before quoting PMs.

Not that my mind had changed from the sentiment expressed "work with others to bring peace in the face of the hate". Actually, this forms the basis of my Albert Speer charge against you, that you are operating in a technical vacuum, without regard for other factors. I posted some of these elsewhere.

If the cc world is dysfunctional, then it's leadership is dysfunctional, but you are in full cooperation with this "leadership". This "leadership" will take your "technical contribution" wherever it pleases. Perhaps there is another phrase for you, this time from Lenin. Useful Idiot.

[Rebel]

As much as you and Chris make a show about being outside/above the moody, psycho-complicated, paranoid and generally crusty/grumpy atmosphere which characterizes computer chess, you guys really epitomize it. Congratulations on going with your gut feeling, in the face of outrageously, overwhelmingly convincing evidence to the contrary.

Jeremy, you are a programmer and able to understand the signature every programmer leaves in his work, in this case the order in EVAL. See my new topic on Rybka Forum. Empty your mind first from the VIG stance and start reading, then tell me if that is all bull.

[hyatt]

I have changed the "overall order" of my evaluation multiple times. Once specifically to improve lazy evaluation. I originally ordered things as naturally as possible. But some eval components are expensive (piece scoring where you loop over all pieces, for example) yet the scores are not that big, other parts are less expensive, but produce bigger scores (pawn scoring for example, thanks to passed pawns and such). I re-ordered so that the cheap cost / larger scores were done first, so that I could then take a fairly liberal lazy eval cutoff and skip the slower but smaller scoring parts. I have also changed the order so that A can do some of the work needed to do B later.

I mean, Vas DID improve Fruit by +100 Elo in a few months. Changing the evaluation would likely be part of the process. And, of course, you are making the same faux pas that we were accused of when we originally compared fruit to strelka since Juri had said that he RE'd Rybka 1.0 beta, but he also added some original code, and some code directly from Fruit. That's what forced the original investigation to dig into Rybka binary instead... Going back to square one is not productive since it was already rejected by the Rybka fans of the world, and the investigation relies on Rybka 1.0 beta itself, not some "approximation" to it.

[Chris Whittington]

I have changed the "overall order" of my evaluation multiple times. Once specifically to improve lazy evaluation. I originally ordered things as naturally as possible. But some eval components are expensive (piece scoring where you loop over all pieces, for example) yet the scores are not that big, other parts are less expensive, but produce bigger scores (pawn scoring for example, thanks to passed pawns and such). I re-ordered so that the cheap cost / larger scores were done first, so that I could then take a fairly liberal lazy eval cutoff and skip the slower but smaller scoring parts. I have also changed the order so that A can do some of the work needed to do B later.

I mean, Vas DID improve Fruit by +100 Elo in a few months. Changing the evaluation would likely be part of the process. And, of course, you are making the same faux pas that we were accused of when we originally compared fruit to strelka since Juri had said that he RE'd Rybka 1.0 beta, but he also added some original code, and some code directly from Fruit. That's what forced the original investigation to dig into Rybka binary instead... Going back to square one is not productive since it was already rejected by the Rybka fans of the world, and the investigation relies on Rybka 1.0 beta itself, not some "approximation" to it.

You miss Ed's point, which is really quite a simple one, so I'll repeat it ....

You claim Rybka copied Fruit evaluation function.

You claim there are so many markers in there of copying that Vas has hardly bothered to even try and cover it. It just took a couple of guys and a decompiler to find pages and pages worth of matches. Allegedly.

So, if according to you, Vas can't even be bothered to cover his tracks, the cut and paste copy exercise that you claim, ought also to have left the evaluation function sub-components in some sort of identifiable order. Which would be a smoking gun.

Ed has been asking for several days whether or not they are in order. He hasn't really had an answer, but reading between the lines it would appear that they are not. it would appear that in Rybka the evaluation sub-components ordering is not the same as in Fruit.

There's no smoking gun, but, according to your logic and description of the cut and paste (allegedly) copy, there ought to be.

Therefore, the lack of this smoking gun in the hand of Vas becomes a smoking gun reflecting at the accusers.

Simples.

[hyatt]

Yes I (and many others) claim Rybka copied Fruit evaluation. We ALSO claim that he had to rewrite a good bit of the code to translate from mailbox to bitboards. One certainly might re-organize parts during such a process. And it doesn't change a thing with respect to copying.

The problem with you (and now Ed) is that you don't WANT to see any evidence that supports copying. So you simply refuse to see it, and at every turn, find some "way out there" way where the evidence shows something else, or shows nothing at all.

I'll ask once again, what about Rybka 1.6.1? The "didn't play in the ICGA events card" is no good. It absolutely shows that he copied code, and then vehemently denied doing so. Multiple times. Shows some sort of "character issue" that I don't consider flattering... How are you going to hand-wave that off? That program was distributed. Entered in a tournament. Entered in rating list competitions. Etc... How do you plan on totally ignoring that one too?

[Chris Whittington]

Yes I (and many others) claim Rybka copied Fruit evaluation. We ALSO claim that he had to rewrite a good bit of the code to translate from mailbox to bitboards. One certainly might re-organize parts during such a process. And it doesn't change a thing with respect to copying.

The problem with you (and now Ed) is that you don't WANT to see any evidence that supports copying. So you simply refuse to see it, and at every turn, find some "way out there" way where the evidence shows something else, or shows nothing at all.

I'll ask once again, what about Rybka 1.6.1? The "didn't play in the ICGA events card" is no good. It absolutely shows that he copied code, and then vehemently denied doing so. Multiple times. Shows some sort of "character issue" that I don't consider flattering... How are you going to hand-wave that off? That program was distributed. Entered in a tournament. Entered in rating list competitions. Etc... How do you plan on totally ignoring that one too?

Well, this attack document critique is an adversarial process by definition, we look for weaknesses in your position and you counter attack, in your case usually with personal attacks.

If you would like to butt out and leave the floor to Zach and BB and others capable of thought, then I am sure the adversarial side of things would reduce from 110% to something more manageable. Progress even. But with you in the room, every room, butting in to every thread, loudly pushing your agenda at the expense of anyone else trying to work up a counter argument, we are going to continue banging our mutual heads. You are behaving like the Rolf of 1997, it has been quite productive for our side, watching you snatch defeat from the jaws of victory, but I suspect the time is approaching to make at least some cooperative progress. Which means you out, I'm afraid.

[hyatt]

The discussion would be more fruitful if YOU were to "butt out".

I'll keep asking, however.

What about Rybka 1.6.1 and Crafty 19.x? How are you going to hand-wave and obfuscate that away? It is "the elephant in the room." There is no way to justify it, or excuse it, or deny it, or pretend it didn't happen, or pretend it is 'ok'.

Any response, or just more dancin' and dodgin' around the topic without addressing the question I asked???

Guy is guilty as sin. TWICE. It's really hard to see how you/he can wiggle out of that...

[BB+]

You must have missed prior to 2006 and the implication of that.

Indeed, I think by now we are having a lot of miscommunication. However, I disagree with the "implication" here, that in 2005/6 no one would yet be worried about the need to obfuscate. For instance, one can find many examples of "cloning" in TalkChess posts from 2004 and before.

One notable one is the El Chinito incident. This is notable because:
*) the code copying of El Chinito from Crafty was found by reverse engineering (disassembling the internals)
*) Rajlich commented in that thread
*) indeed, he privately congratulated Zach [in 2009 email conversation] about ZW's assistance in that case
So it seems reasonable to assume that already in 2004 that Rajlich would be aware that dissection (RE) of any program was possible, and that "cloning" could be determined by this. [One can also note that Eugenio Castillo was reconciled to the CC community simply by admitting his fault, and that the El Chinito evidence has much in common with Rybka161/Crafty].

To conclude, I find your contention that no one would, prior to 2006, care about internal obfuscation to cover-up cloning to be simply incorrect. If I were keeping score at this point, I think it would be something like: "Unlikely arguments raised by Ed: 10, Number rebutted (or at least strongly disputed): 10" -- are we now aiming for triple digits? I refer you (again) to the evidential burden, to which I will (from here on) simply refer when you raise a new issue w/o any compelling nature to it. I assumed you (and perhaps ChrisW) in good faith wished to discuss substantive issues, but it has now transpired that the situation appears otherwise.

As for all the compliments directed at me, I understand your frustration. You don't have a good answer to the unburden point I made.

If your unburden point is the "evaluation ordering", I re-iterate (for the 3rd, maybe 4th time) that the evidence says little one way or the other. Perhaps I should just state rather categorically "the evaluation ordering in Rybka 1.0 Beta is essentially the same as Fruit, except for a few insignificant modifications that derive from bitboard and performance issues". Is this a good enough answer? Will it make you at least discuss the issue to argue that I am wrong? Or do I not understand computer chess development sufficiently well?

And again: if I answer you, you change the subject -- if I don't answer [or answer in a wishy-washy way], you claim it's because you've hit upon something unanswerable. By now, I think I've answered everyone of your inquiries, including whether I'm on some "ICGA Payroll" -- but I note that you have yet to publicly deny that Raljich is paying you to "stir up trouble", so I can thus assume he is?

But you could stop sending me those silly PM's.

Sorry, you did say "interesting read" to the first one. I similarly would prefer not have "silly PMs" from you asking me to join the discussion at the Rybka Forum. Are you willing to let me make public what you said to Fabien back in February, when signing the Open Letter?

Perhaps it was foolish of me to assume that there were standards here in open chess which meant that you might ask before quoting PMs.

As a technical point, it was an email, not a PM (remember, you didn't like the PM system?). My recollection is that letters/emails fall (at best) under copyright law, and quoting small portions of them for the purposes of comment are indeed one exception to copyright. I also find your charge that I am "operating in a technical vacuum" to be incorrect speculation. Indeed, one reason why I am interested in (and actively pursuing) physically meeting more of the people in CC is related to this.

[BB+]

Ed has been asking for several days whether or not they are in order. He hasn't really had an answer, but reading between the lines it would appear that they are not. it would appear that in Rybka the evaluation sub-components ordering is not the same as in Fruit.

I think you've jumped the shark. Ed wonders why I responded so quickly -- you say he's been asking for days with no real answer. You argue that I'm in a technical bubble -- yet when I (as I did with the Panel) merely present evidence and let the reader come to an answer himself, voila, suddenly he hasn't really had an answer. If I view the evidence and make an argument myself, perhaps I'm biasing the issue -- if I leave a conclusion for others to make based on their own skills, I'm not answering the question. Tedisome.

Reading the lines themselves, one finds many evaluation sub-components that are in the same order in Fruit 2.1 and Rybka 1.0 Beta. I didn't want to over-emphasise this, as (similar to PST evidence) one can argue that you start with a pattern, and follow it. But now since you contend the opposite, here is a partial listing of parts that are in the same order, for no apparent reason.

First a selection of two sub-components where the ordering is indeed the same:
eval_pattern(), the order of:

trapped bishops: white queenside, then kingside; and black queenside, then kingside             
blocked bishops: white queenside, then kingside; and black queenside, then kingside             
blocked rooks: white queenside, then kingside; and black queenside, then kingside

appears in both, with Fruit having a minor difference in splitting of the first into 7th/6th ranks.

Rook evaluation:

Rooks: compute mobility, SemiOpen, Open [including opposing king], 7th rank
       the latter first checks pawn on 7th, then (if not) whether opp has king on 8th

As with all such aspects, it's not impressive by itself, but there indeed is a match [and e.g., there's no compelling reason to compute open files before 7th rank]. Similarly with other pieces.

Overall ordering, both do the following, with minor other aspects here omitted

material table lookup
add in PST (from incremental calculation)
evaluate pawns and pieces
evaluate king safety
evaluate passed pawns
evaluate patterns
interpolate
adjust for drawish bishop endgames

Some of this is forced by the process. Some of it is not. For instance, one can add in PST values at the beginning or near the end, or any place in-between. Some of these places are more logical than others. One can similarly consider "patterns" at the beginning or the end. Passers only need to be after pawneval. So there's a variety of possible orderings. Not too many, and indeed many programs will have something similar, but there is (as with PST) at least some choice. And Fruit/Rybka make more of the same choices than one expects from chance. If I were to make a specific estimate, I could be accused of "mathematising" the whole process -- and if I don't, perhaps it's just a qualitative feeling of mine.

As for the "evaluation ordering" differences, Rybka changes orderings with pieces, due to bitboards. Rybka also moves the pawn eval (particularly shelter/storm) and king safety computation, due to performance issues [Fabien notes that Fruit 2.1 was still in developmental form in various ways].

If this were Ed, I might be expecting the subject to change again (or maybe a personal insinuation/attack, by this point). If you do in fact want an easy counter-argument to the above, the previous enumeration of evidence should provide a few. If this were the Panel and I were making the argument I made here, I would include such mitigating factors [as being proper from the scientific standpoint], but I see little point in doing this for you and Ed anymore, and so will rather state the above in a more advocatory manner.

[BB+]

[...]
interpolate
adjust for drawish bishop endgames

I was wrong about this. Rybka flips the order of these two, and indeed, at some point Zach already noted that it doesn't matter whether one divides by 2 before or after the interpolation. The ordering difference here could partially be due to the way that Fruit handles "mul" (a more general drawish factor), which is something that Rybka omits [there is rather simply a division by 2]. Again this could be classified (and argued either way) as a change induced from external considerations with Rybka's material imbalance table.