David Levy wrote:
No Miscarriage of Justice - Just Biased Reporting
By David Levy [ICGA President]
Introduction
In his four-part article on Chessbase.com about the Rybka scandal (see chessbase.com/newsdetail.asp?newsid=7791 et seq.) Dr Søren Riis has tried very hard to defend Vasik Ralich’s actions that led the ICGA to find him guilty of breaking ICGA Tournament Rule 2. As a historical review of progress in computer chess Riis’s article contains important and interesting information and comments. Unfortunately, however, his thesis lacks objectivity because it circles the core question and attempts to defend Rajlich by attacking the rule he was accused of breaking, attacking the investigative process in various ways and attacking some of those involved in that process.
When a defendant is brought before a court of Law, what is in question is whether or not (s)he broke the Law and not whether the Law itself is appropriate. And so it is with the ICGA rules. In considering the Rybka case the ICGA’s task was to decide the matter on the basis of its Tournament Rule 2, not to question the rule itself. ICGA Tournament Rule 2 applies to the World Computer Chess Championships and states:
“Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.”
Note that this rule requires programmers to list all authors and the source of code, even “derived” code. Entering a competition such as this brings with it the obligation to abide by the competition rules.
Any readers of Riis’s articles who wish to understand the ICGA’s stance in this matter should visit the ICGA web site at icga.org where they will find a link to the full report of the ICGA Investigation Panel together with its supporting evidence and the judgement made by the ICGA based on that report.
How the Scandal Started
The Rybka scandal came about because when Vasik Rajlich submitted his entry applications for the World Computer Chess Championships in 2006, 2007, 2008, 2009 and 2010 he failed in his duty to include the name of Fabien Letouzey, the programmer of Fruit, in his authorship statements. The correct procedure would have been to obtain Letouzey’s permission to use the Fruit derived code and to name Letouzey within the authorship statement, thereby advising the ICGA of the provenance of the derived code. Sadly Rajlich did neither. Had he conformed to the rule it would then have been up to the ICGA either to accept Rajlich’s entries as they stood or to ask him to remove the Fruit derived code for the purpose of the tournaments.
These are the bare facts of the case and are, I believe, beyond dispute. It was upon these facts that the ICGA based its decisions regarding Rajlich being stripped of the World Championship titles awarded to Rybka and being banned for life from ICGA events.
Incidentally, even if the later versions of Rybka contained reduced amounts of Fruit-derived code, the historical roots of an engine are of import. For as long as Rybka engine development was continual, Letouzey’s name should not have been omitted on the ICGA entry information, it then being at the discretion of the ICGA as to how to classify the level of Fruit’s involvement.
Some Comments on Søren Riis’s Arguments
Søren Riis is a moderator on the Rybka forum and clearly a big Rybka fan. Nothing wrong there. But when one examines the detail of Riis’s arguments one finds that his enthusiasm for Rybka has outweighed his objectivity, causing him to make points that do not affect the simple basic need for laws (and tournament rules) to be adhered to.
A detailed and robust technical rebuttal of Riis’s article is in preparation by the ICGA and will be posted shortly. Here I shall point out the irrelevance to the ICGA rules of some of his key arguments and correct some of his erroneous assumptions.
[1] Riis states that the matter was not investigated by the ICGA until:
“over five years had elapsed since the alleged offense, and four consecutive world computer chess championships had been decisively won in head-to-head competition by Rybka.”
In fact when the ICGA was formally requested to investigate (early 2011) it was only a few months after Rajlich’s fifth WCCC violation of rule 2. And prior to 2011 no complaint had been made to the ICGA regarding Rybka, so there was no call for the ICGA to launch an investigation earlier than it did. The ICGA started its investigation within a month or so of a complaint being made by Fabien Letouzey, the author of Fruit.
[2] Riis states that:
“It is clear that Rybka is an original program by any reasonable standard.”
Rajlich has also stated, in a CCC post of Dec 16 2005:
stmintz.com/ccc/index.php?id=470751
"As far as I know, Rybka has a very original search and evaluation framework.”
And in an interview Rajlich said:
superchessengine.com/vasik_rajlich.htm
“Anyway, if I really had to give a number – my wild guess is that Rybka would be 20 rating points weaker had Fruit not appeared.”
Let us now consider the above comments in the light of the evidence. We shall first repeat the last quotation (by Rajlich) because it is so astounding.
“Anyway, if I really had to give a number – my wild guess is that Rybka would be 20 rating points weaker had Fruit not appeared.”
Rajlich himself has admitted that “I went through the Fruit 2.1 source code forwards and backwards and took many things”. Is he here really expecting us to believe that only 20 Elo points of Rybka’s improvement were due to what he took from Fruit, especially as this contradicts Riis’s statement that, after the publication of the Fruit source code, “everyone else” gained much more?
Next consider the investigation report where it refers to Zach Wegner’s analysis found at webspace.utexas.edu/zzw57/rtc/eval/eval.html
“From looking at the piece evaluation of both engines, we find that they are almost identical.” A partial listing of Fruit identical terms:
Identical formulas for calculating piece-square tables for:
. pawns
. knights
. bishops
. rooks
. queens.
Highly similar formulas for piece square tables for kings.
Identical procedures for calculating king safety:
. count of pieces attacking squares around the opponent king
. adding in an attack factor based on piece type then multiplying by a weight based on attack counts
Identical simple mobility counting for:
. knights
. bishops
. rooks
. queens
Identical measurements of pawn features:
. isolation
. doubling
. open
. highly similar backward, candidate and passed pawns
Rook Evaluation:
. identical methods for R on the 7th
. rooks on half and opened files
. king file proximity
Queens:
. identical Q on the 7th.
Blocked Bishop and Rook terms
And to quote the summary in this part of the report:
“Nearly the entire evaluation function is derived from Fruit. This includes the formulas for calculating piece-square tables, methods and features of evaluating piece mobility, rook king file proximity, rook and queen on the 7th rank, and king safety.”
I shall leave it to readers to study the evidence presented in the investigation report, in Riis’s article and in the ICGA’s forthcoming technical rebuttal, and decide for themselves who they believe.
[3] Riis states:
“Here’s the main point: to convict and sentence a man due to his presumed ethical failings and then attempt to ruin him on a world stage you need a very high standard of evidence.”
I take great exception to the suggestion that the ICGA attempted “to ruin” Rajlich. The ICGA made no such attempt, it merely reported accurately on the evidence it had considered and on its findings. It is also not true that the ICGA convicted and sentenced Rajlich “due to his presumed ethical failings” – the ICGA’s verdict and sanctions were because he broke Tournament Rule 2.
[4] Riis points out that:
“Rybka maintained unbroken supremacy on the chess engine rating lists for five years. However its performance in dozens of competitive tournaments held all over the world was, if anything, even more spectacular. Rybka did not merely win nearly every tournament it entered; it won them with a near-90% success rate. It is difficult to overstate the degree of superiority that the Rybka team exhibited in these years in chess software, mastery of hardware, and even in opening theory.”
All of this paragraph is perfectly true but totally irrelevant when considering Rajlich’s guilt or innocence. How would we view an Olympic athlete found guilty of taking performance enhancing drugs if he performed superbly, winning races by huge margins, breaking world records and taking gold medals? Would he be forgiven his drug taking just because his performances were outstanding? No, of course not!
[5] Riis’s article includes quotes by Vasik Rajlich in which he publicly praises the work of Fabien Letouzey. Riis comments:
“If these acknowledgements were insufficient to satisfy WCCC Rule 2, then we must return again to arguments made earlier on the inadequacies of the rule itself and how it is applied.”
Clearly any such acknowledgements made by Rajlich on Internet forums or other public postings are in no way related to his obligations when submitting his applications to participate with Rybka in World Computer Chess Championship tournaments. What counts when making an entry submission is the information that the entrant submits with his application and any relevant information which he fails to submit. Suspecting that these acknowledgements are insufficient to satisfy Rule 2 Riis then reverts to another of his defensive claims, that Rule 2 itself is inadequate. But as I have pointed out earlier, disagreement with a rule or a law does not serve as an adequate defence for breaking it.
[6] Riis points out that, of the sixteen programmers who petitioned the ICGA early in 2011 to investigate Rybka:
“many . . . were in direct competition with Rybka”.
The implication here is that programmers who were competitors of Rajlich might be biased in their call for an investigation. So if one athlete reports a drug-taking rival to the athletics authorities, should the authorities discount the report because the whistle-blower is a rival? Surely not?
All six of the above points are, I submit, irrelevant to the core question – did Vasik Rajlich break ICGA Tournament Rule 2 or did he not? That is what the investigation panel considered, and it was on the basis of their findings and conclusions that Rajlich was sanctioned by the ICGA. Rajlich was given ample opportunity to present a defence to the allegations – as President of the ICGA I invited him to do so before the investigation started, then again during the investigation when I sent him some of the evidence being considered, then again after the panel’s report had been completed (attaching the report and all of the evidence to which it referred) but before the report was considered by the ICGA executive, and yet again after the ICGA executive had considered the report but before we decided on the appropriate sanctions. These prompts by me were sent (inter alia) on February 2nd, March 2nd, April 4th, May 13th, May 31st and June 9th. But despite being given all these opportunities to provide a serious defence to the allegations against him, over a period of five months, Rajlich consistently declined to do so. What could he reasonably expect to happen as a result?
Biased Reporting
I should now like to turn to some aspects of Søren Riis’s article that I regard as biased reporting.
I regret the bias in Riis’s writing, I also regret Chessbase’s attitude to the Rybka scandal. When this scandal erupted early in 2011 I sent my first posting on the matter to Chessbase but they decided not to publish it. Yes, I know that Chessbase sells Rybka, and therefore they might have worried about losing sales, but as a web site of record in all matters chessic it would surely be good journalistic practice to publish important criticism of parts of one’s own “empire”, together with any comments on that criticism which they feel is appropriate. As an example, consider the recent phone hacking scandal that has engulfed News International, the London based company that publishes ‘The Times’, ‘The Sunday Times’ and other newspapers, and caused News International to close down one of its titles, ‘News of the World’, in the middle of 2011. In the midst of the hacking scandal News International itself and its famous proprietors Rupert Murdoch and his son James Murdoch were all castigated in the media, and extensive reports of the scandal were published throughout the News International media empire itself, even though the criticism faced by the Murdoch group of companies was the worst in the history of the UK media. That was responsible and unbiased reporting, no attempt being made to sweep the hacking scandal completely under the carpet. In contrast, when the Rybka scandal erupted Chessbase.com acted as though no Rybka scandal existed.
On now to some examples of the bias in Riis’s article.
[a] The link Riis provides to the ICGA report does not give his readers access to the important evidential documents on which the report was based. A knowledge and understanding of the evidence is needed by any of Riis’s readers who want to consider both sides of the arguments, and many visitors to Chessbase.com will not yet have seen them. I was rather sorry to see that Riis failed to provide such links, the more so because he did provide many links to the “defense” side.
Riis states:
“It really goes without saying that the panel members voted based on the findings of the ICGA report. . .”
This implies that the members of the panel were not privy to all of the evidence, and as such it is misleading. In fact the members of the panel all had access to all of the evidence documents, so they voted not only on the basis of the report but also on the basis of the evidence on which the report was based.
[c] Riis states:
“A panel was formed. Dr. Hyatt served as panel gatekeeper and determined who was and was not allowed to participate.”
Not true. The decision on who was and was not allowed to participate was taken jointly by the three members of the Secretariat and myself, after each of us had had the opportunity to make comments in favour or against particular individuals. With this statement Riis implies that the composition of the panel was somehow skewed against Vasik Rajlich, but that is also untrue. For example, Chris Whittington, a strong Ralich supporter, asked to join and made the comment that he supposed that I would refuse to admit him. Quite to the contrary, I was in favour of admitting him. Unfortunately, when we asked him as part of the registration procedure to verify his email address, which no longer matched those he used for older forums, he responded using phrases such as "wasting humiliation" and "occasional little hitler"? The Secretariat felt he was unwilling to have civil dialog with others and all three of them felt he should not be a member if he was going to be rude. Then, after a brief period, he was invited to re-apply but declined to do so.
Another pro-Rajlich programmer is Ed Schroeder, who was similarly welcomed to the panel when he joined. Sadly Ed then decided that he did not wish to serve and asked for his name to be removed from the panel.
Naturally Vasik Rajlich was asked to be a member of the panel with full access to all the evidence but he refused multiple requests to join.
[d] Riis criticizes Bob Hyatt in various ways, hardly surprising in view of the fact that he also criticizes the rule that Rajlich broke as well as various other aspects of the investigative process. Suffice it to say that Bob Hyatt is one of the world’s leading experts in the field of computer chess, has been so for some thirty years, has twice won the World Computer Chess Championship, and has contributed hugely to the field in various ways, including making his program Crafty open source. Who has a better understanding of computer chess and its minutiae, and who would be more appropriate as a member of the Secretariat of the Investigation Panel – Bob Hyatt or Søren Riis?
[e] Riis incorrectly states that in my first posting on ChessVibes.com I:
“made a preemptive declaration of Rajlich’s guilt”.
Wrong. I wrote about the rumours that had been circulating regarding Rybka and said that:
“But as I have mentioned, at first the Rybka-Fruit case was mere rumour. More recently, however, these rumours have become firm allegations, made by expert chess programmers and supported by evidence which appears on the surface to be rather compelling, both in its nature and in its volume. At this point in time I do not intend to make any definitive statement of my own on these allegations, but will allow the reader to form their own opinion after reading the following.”
So even though I specifically wrote that I did not intend to make a definitive statement of my own at that time, Riis accuses me of declaring Rajlich guilty in that article. In fact I made no public comment at all about Rajlich’s guilt until after the Investigation Panel had completed its work.
[f] Riis states:
“Not even half of the original committee of 34 voted for a guilty verdict. Was it even clear in advance how many guilty votes were needed to convict?”
There was never any compulsion on the members of the panel to take part in the vote that the ICGA conducted when the report had been completed. The purpose of the vote was to determine how the balance of opinion went amongst those panel members who did wish to vote. The result was that not one single member of the panel voted for a not-guilty verdict. Amongst those panel members who were convinced of Rajlich’s guilt was Ken Thompson, a past World Computer Chess Champion programmer, co-author of Unix, winner of the ACM Turing Award (inter alia) and arguably the most august computer scientist who has ever graced our community with his active participation.
[g] Riis’s bias is perhaps at its most telling in the following paragraph from his article:
“While no one questions the fact that the ICGA gave Rajlich ample opportunity to respond to their charges and he did not, there is much more to the matter than “we queried him and he did not respond.” Rajlich was not merely queried. He was publicly accused by the head of the ICGA and publicly excoriated by a group of individuals who stirred themselves up into a crusading lynch mob. A pile of “evidence” was jubilantly thrown together based on a passionately-held predetermined conclusion of code-copying which happened to be wholly at variance with actual reality. And then Rajlich was offered the opportunity to formally respond.”
There is so much to fault with this paragraph that it is difficult to know where to begin. To describe the panel of computer chess experts, many of whom are eminent academics and including a number of former world champions, as a “crusading lynch mob” is, in my view ridiculous and totally uncalled for. The members of the panel were there because they had expressed an interest in following the course of the investigation, perhaps taking some part in it, and because they were assessed as having some expertise that could be helpful to the discussion. To say that the evidence was “jubilantly thrown together” implies an unfair bias on behalf of those who compiled it. To go a little deeper into this particular sentence, the words “thrown together” imply that the evidence was not carefully prepared, whereas anyone reading all of the supporting documents referred to in the report would readily understand that a huge amount of time and assiduous effort had been devoted to the task of collecting and evaluating that evidence. And the words in the final sentence of this paragraph: “And then Rajlich was offered the opportunity to formally respond.” imply, through the emphasis on the word “then”, that the opportunity for Rajlich to defend himself came only after the investigative process had gone a long way towards its conclusion. In fact, as I have shown above, Rajlich was given several opportunities to defend himself, over a period of five months, before the report was considered by the ICGA executive.
Summary
As I have already mentioned, a robust technical rebuttal to Riis’s article will shortly be distributed by the ICGA, so here I have stayed away from the technicalities and commented on the legalistic aspects of the Rybka case. The ICGA would have welcomed an objective appraisal of its conduct of the investigation and its conclusions, but Søren Riis cannot be described as objective in this matter, not by any stretch of the imagination. The resulting article, full as it is of Riis’s bias, does nothing in my view to make the case for a miscarriage of justice to have taken place. It is, put simply, biased reporting.
END
