Bob asks: Then why don't YOU come up with a rule 2?

[Chris Whittington]

OK, here's my view:

This is Rule 2 from 2007, it has been hacked around since and now has a one-year statute of limitations added plus a few other bits of verbiage.
2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.

There is nothing wrong with Rule 2. "Derived from" should be defined better. "Close derivative" gets near to it, but is also open to (mis)interpretation. The "invalidation from the tournament" sanction is correct.
Rule 2 is essentially designed for use by a TD during a tournament.
The sanction is at the level of a football referee's sanction. TD's have to make quick decisions, might be wrong (is that player really offside?), and the limited sanction is therefore correct.

What is wrong, is not Rule 2, but trying to apply it after the event. An apres event process, if needed at all, should be wholly different. there is no time pressure, the technical advice and any tribunal can and should be produced impartially. The decision process should be impartial. All sides must be heard. Process must be clear beforehand etc etc. In case Rybka, clearly this all went badly wrong. There has to be a statutory structure plus rules in place to do anything at all.

The real question is: Should there be a computer chess body (eg the hierarchical ICGA) forming tribunals post-event and inflicting punishments? Or, should Levy have said "sorry, we don't have the statutes for this, ICGA is not designed for judicial matters, it is six years later, I suggest you take this to a copyright court where it can be properly assessed". The end.

The other question, you have covered already. What happens if Houdini, Ippolit, Rybka, or any other program enters an ICGA tournament? Well, the ICGA accept the entry, publicise it along with the other entrants, and interested participants can petition the TD at the time, or pre-prepare the TD that there is going to be a challenge. In practice, any "guilty" program(mmer) is unlikely to want to give up a week in some other country, plus hotel bills, plus entry fee, to go through such a procedure and will likely stay away. If he doesn't - well the TD will have to make a decision.

Sledgehammers are cracking nuts here. Invalidation from a tournament is quite enough, and the program can be "invalidated" again next year if necessary. ICGA should not get itself involved in post event complaints. Actually the best place to decide/discuss these is on the forums - where peer/user distaste is a perfectly good sanction, and, ultimately, the real truth of derivative or not (in complex cases) will emerge. It's one of the features of internet communication and forums, peer group acts as the sanctioning body via a humiliation process in effect.

So, conclusion: leave Rule 2 as it is. Give no more than ONE week, or better no time at all after the tournament result is in, at which point it is TOO LATE. Use the horizontal, everybody is equal, networking forums to discuss what is and what is not a derivative, on the basis that truth will eventually emerge, and internet humiliation will do its work and settle at an appropriate level (of punishment). No need for Levy/ICGA to risk getting it wrong.

All was ok, Rule 2 was enough, forums do their job.

[terrapene]

OK, here's my view:

This is Rule 2 from 2007, it has been hacked around since and now has a one-year statute of limitations added plus a few other bits of verbiage.
2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.

There is nothing wrong with Rule 2. "Derived from" should be defined better. "Close derivative" gets near to it, but is also open to (mis)interpretation. The "invalidation from the tournament" sanction is correct.
Rule 2 is essentially designed for use by a TD during a tournament.
The sanction is at the level of a football referee's sanction. TD's have to make quick decisions, might be wrong (is that player really offside?), and the limited sanction is therefore correct.

What is wrong, is not Rule 2, but trying to apply it after the event. An apres event process, if needed at all, should be wholly different. there is no time pressure, the technical advice and any tribunal can and should be produced impartially. The decision process should be impartial. All sides must be heard. Process must be clear beforehand etc etc. In case Rybka, clearly this all went badly wrong. There has to be a statutory structure plus rules in place to do anything at all.

The real question is: Should there be a computer chess body (eg the hierarchical ICGA) forming tribunals post-event and inflicting punishments? Or, should Levy have said "sorry, we don't have the statutes for this, ICGA is not designed for judicial matters, it is six years later, I suggest you take this to a copyright court where it can be properly assessed". The end.

The other question, you have covered already. What happens if Houdini, Ippolit, Rybka, or any other program enters an ICGA tournament? Well, the ICGA accept the entry, publicise it along with the other entrants, and interested participants can petition the TD at the time, or pre-prepare the TD that there is going to be a challenge. In practice, any "guilty" program(mmer) is unlikely to want to give up a week in some other country, plus hotel bills, plus entry fee, to go through such a procedure and will likely stay away. If he doesn't - well the TD will have to make a decision.

Sledgehammers are cracking nuts here. Invalidation from a tournament is quite enough, and the program can be "invalidated" again next year if necessary. ICGA should not get itself involved in post event complaints. Actually the best place to decide/discuss these is on the forums - where peer/user distaste is a perfectly good sanction, and, ultimately, the real truth of derivative or not (in complex cases) will emerge. It's one of the features of internet communication and forums, peer group acts as the sanctioning body via a humiliation process in effect.

So, conclusion: leave Rule 2 as it is. Give no more than ONE week, or better no time at all after the tournament result is in, at which point it is TOO LATE. Use the horizontal, everybody is equal, networking forums to discuss what is and what is not a derivative, on the basis that truth will eventually emerge, and internet humiliation will do its work and settle at an appropriate level (of punishment). No need for Levy/ICGA to risk getting it wrong.

All was ok, Rule 2 was enough, forums do their job.

Chris, you think truth can emerge from internet forums?
the Houdini 'author' was only humiliated when he déclined to #2 on ccrl and cegt

[Chris Whittington]

OK, here's my view:

This is Rule 2 from 2007, it has been hacked around since and now has a one-year statute of limitations added plus a few other bits of verbiage.
2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in their submission details. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director.

There is nothing wrong with Rule 2. "Derived from" should be defined better. "Close derivative" gets near to it, but is also open to (mis)interpretation. The "invalidation from the tournament" sanction is correct.
Rule 2 is essentially designed for use by a TD during a tournament.
The sanction is at the level of a football referee's sanction. TD's have to make quick decisions, might be wrong (is that player really offside?), and the limited sanction is therefore correct.

What is wrong, is not Rule 2, but trying to apply it after the event. An apres event process, if needed at all, should be wholly different. there is no time pressure, the technical advice and any tribunal can and should be produced impartially. The decision process should be impartial. All sides must be heard. Process must be clear beforehand etc etc. In case Rybka, clearly this all went badly wrong. There has to be a statutory structure plus rules in place to do anything at all.

The real question is: Should there be a computer chess body (eg the hierarchical ICGA) forming tribunals post-event and inflicting punishments? Or, should Levy have said "sorry, we don't have the statutes for this, ICGA is not designed for judicial matters, it is six years later, I suggest you take this to a copyright court where it can be properly assessed". The end.

The other question, you have covered already. What happens if Houdini, Ippolit, Rybka, or any other program enters an ICGA tournament? Well, the ICGA accept the entry, publicise it along with the other entrants, and interested participants can petition the TD at the time, or pre-prepare the TD that there is going to be a challenge. In practice, any "guilty" program(mmer) is unlikely to want to give up a week in some other country, plus hotel bills, plus entry fee, to go through such a procedure and will likely stay away. If he doesn't - well the TD will have to make a decision.

Sledgehammers are cracking nuts here. Invalidation from a tournament is quite enough, and the program can be "invalidated" again next year if necessary. ICGA should not get itself involved in post event complaints. Actually the best place to decide/discuss these is on the forums - where peer/user distaste is a perfectly good sanction, and, ultimately, the real truth of derivative or not (in complex cases) will emerge. It's one of the features of internet communication and forums, peer group acts as the sanctioning body via a humiliation process in effect.

So, conclusion: leave Rule 2 as it is. Give no more than ONE week, or better no time at all after the tournament result is in, at which point it is TOO LATE. Use the horizontal, everybody is equal, networking forums to discuss what is and what is not a derivative, on the basis that truth will eventually emerge, and internet humiliation will do its work and settle at an appropriate level (of punishment). No need for Levy/ICGA to risk getting it wrong.

All was ok, Rule 2 was enough, forums do their job.

Chris, you think truth can emerge from internet forums?
the Houdini 'author' was only humiliated when he déclined to #2 on ccrl and cegt

What is truth? There are many, as many as readers. My truth is not the same as Bob's truth. It is unlikely there is such a thing as a universal truth. But, philosophical ramblings aside, although there are many truths, each reader can, by reading as little or as much as he wants, arrive at his own, and, in doing, he participates on a "forums truth", a compilation, much as Stockfish is composed of individual ideas and efforts, so is this complex truth concept. I think we know a "forums truth" when we see it, it's just that we have several forums

However it happened, and whatever the "truth" is about Houdini, the facts are that he is not around, it seems. Anyway, I would add ccrl and cegt as "forums" themselves. They are part of the comp chess network.

[hyatt]

In a non-paid organization like a WCCC, doing as you ask is pretty much impossible. If you are in the event, playing two rounds per day, preparing openings between rounds, there is little time to develop a formal protest, you might not even NOTICE what was going on until you get home and play over the games. So there has to be some period of time for a protest to be filed, and there has to be some concrete way of handling such protests.

I suggested requiring everyone to submit source. Each person, in the presence os (say) three others from the ICGA provides their source. It is encrypted on the spot with an m out of n encryption key (you use n pieces of the key to encrypt, but it can be decrypted with m pieces where m < n. For example, with the ICGA, perhaps David would supply one part, the TD (Jaap) the second part, the author the third part, and a 3rd party the 4th part. The programmer doesn't have to worry about his source being exposed since it would take 3 out of 4 to decrypt the data, the programmer could not thwart the process if a protest happens since his part is not required if the other three agree that the examination has to be done. Time requirement and storage requirement is non-trivial to manage.

To make that work, the program has to be run against a secret set of positions and the output saved along with the encrypted source. Then somewhere along the way, during the event, the TD can surprise anyone by asking them to run that set of positions through their program and comparing to verify the source on hand matches the program being used. Time requirement is non-trivial.

The rule should probably include a statute of limitations of something like one year, UNLESS a program violates rule 2 and is verified to do so with the source on hand and the test position output on hand, which would open the door to go back even further. Sort of like the US IRS. If they detect something illegal, they can go back as far as they want, all the way back to your first paycheck if they wish. So you could not extend the trip backward in time unless you are proven guilty of violating rule 2 within 12 months of the event in question. Again a time requirement.

So there are ideas that can work. But before you know it, there are no people left willing to be involved because of the time requirement. The Fruit/Rybka case wasted a LOT of time for everybody. And I do mean a LOT. I would not look forward to going through any more of those, so the RE part HAS to go away. And the only way to do so is to have an archived version of the source that can't be "lost" conveniently. And it has to be safeguarded against intentional or unintentional exposure.

All has to be done by an organization that doesn't pay people.

And none of that directly addresses the complaint you and Ed have been promulgating, namely "what does originality mean?"

[hyatt]

The test on this "rybka rook scoring" idea finished overnight. I've already posted the code and won't repeat since it matches the code Zach published for R1beta. Here's the test results:

4 Crafty-25.0-1 2637 3 3 30144 56% 2588 24%
5 Crafty-25.0-2 2635 3 3 30144 56% 2588 24%
6 Crafty-25.0R01-1 2619 3 3 30144 54% 2588 24%
7 Crafty-25.0R01-2 2618 3 3 30144 54% 2588 25%
8 Crafty-25.0R02-2 2613 3 3 30144 53% 2588 24%
9 Crafty-25.0R02-1 2610 3 3 30144 53% 2588 24%

Crafty-25.0-1 and Crafty-25.0-2 are two "control runs" with the best Crafty version so far. Crafty-25.0R02-1 and Crafty-25.0R02-2 are the SAME version, but with the normal open/half-open file stuff removed, and the rybka "brilliant" scheme used instead. I told you yesterday I didn't think the formulation was good, while you were gushing on and on about how clever it was. It is so clever it loses over 20 Elo.

25.0R01 I left in from a CCC discussion. Someone asked how / when I cleared killer moves. I replied "never". They thought that killers are "local" in nature and that when you reach search depth N you should clear killers for depth N+1. R01 does exactly that. It is a 16-18 Elo loss.

These test were interleaved. 25.0-1, 25.0R01-1 25.0R02-1, 25.0-2, 25.0R01-2, and 25.0R02-2 was the order. Clearly the r1beta "open/half-open file" idea is not good. In fact, it is one of the few eval terms I have seen in recent years that produces such a huge drop in Elo. There are lots of adjectives you could use to describe this idea, but "good" or "brilliant" or "intelligent" are NOT in the list. Based on the results, I believe this was an accident. A bug. An oversight. The "formulation" is broken. If you want to see the entire test BayesElo output, here it is:

Rank Name Elo + - games score oppo. draws
1 Stockfish DD 64 2889 4 4 33984 85% 2611 20%
2 Senpai 1.0 2762 3 3 33696 71% 2611 25%
3 Texel 1.05 64-bit 2716 3 3 33984 64% 2611 26%
4 Crafty-25.0-1 2637 3 3 30144 56% 2588 24%
5 Crafty-25.0-2 2635 3 3 30144 56% 2588 24%
6 Crafty-25.0R01-1 2619 3 3 30144 54% 2588 24%
7 Crafty-25.0R01-2 2618 3 3 30144 54% 2588 25%
8 Crafty-25.0R02-2 2613 3 3 30144 53% 2588 24%
9 Crafty-25.0R02-1 2610 3 3 30144 53% 2588 24%
10 Crafty-24.0-1 2589 3 3 30144 51% 2588 25%
11 Crafty-24.0-2 2587 3 3 30144 50% 2588 24%
12 Crafty-24.1-1 2587 3 3 30144 50% 2588 22%
13 Scorpio_2.7.7 2546 3 3 33984 41% 2611 26%
14 Toga2 2476 3 3 33984 31% 2611 26%
15 Glaurung 2.2 2473 3 3 33696 31% 2611 24%
16 Arasan 17.0 2467 3 3 33984 31% 2611 23%
17 Fruit 2.1 2378 3 3 33984 20% 2611 21%

Now maybe we can get off of THAT particular discussion.

I also ran one more test, I did that same change to fruit 2.1. It dropped about 10 Elo. But fruit is on the low end of the rating scale here and is subject to "compression". If it had players that were ranked below it, I would expect it to drop a similar amount compared to latest Crafty.

So two tests, this idea sucked in both. IMHO it just "sucks" and represents a bug rather than a feature. An easy bug to introduce in bit boards just by choosing the wrong mask.

[BB+]

In practice, any "guilty" program(mmer) is unlikely to want to give up a week in some other country, plus hotel bills, plus entry fee, to go through such a procedure and will likely stay away.

I don't know if this is really true in practise. With Squarknll, the issue was the guy trying to procure a visa to travel outside Cuba. My own experience with maths conference organisation is similar -- for any "big-name" recurring conference (like ANTS) you get any number of random people applying to the event (the best I remember was someone from sub-Saharan Africa whose affiliation was "Go Fish Ministries"). They use their registration in your ("prestigious world-class") event/meeting to try to get some local entity to fund them, etc. Note also that the ICGA often gives travel stipends.

...interested participants can petition the TD at the time, or pre-prepare the TD that there is going to be a challenge.

What are the prior criteria for such a challenge to be deemed receivable? For instance, Rajlich has said he thinks the ICGA should check others (perhaps everyone) the same way that Rybka was investigated, but the impetus for said measure is unclear. Is the decision concerning receivability one in which there is a right to be heard? Is there a right to appeal from the tournament committee to the ICGA Board?

In auto racing, for instance NASCAR, the protocol I remember is that: everyone goes through a pre-race inspection, in-race surveillance, and perhaps minor post-race inspection; for any race, the winner, the points leader, and one random car (out of about 40) go through a more extensive post-race inspection, that can sometime involve impounding the car for a day or two (or perhaps longer, if something like wind-tunnel testing is needed). But then, they have heaps of money.

It's one of the features of internet communication and forums, peer group acts as the sanctioning body via a humiliation process in effect.

It depends upon whether you care about the opinions of your "peers". For instance, with the Rybka case, some forums seem to be more inclined toward end-user opinions than those from programmers (similarly with Houdini).

Sledgehammers are cracking nuts here. Invalidation from a tournament is quite enough, and the program can be "invalidated" again next year if necessary. ICGA should not get itself involved in post event complaints.

In a perfect world, this would be valid. But when obfuscatory methods are used to try to beat the system, the sledgehammer becomes more attuned to the situation. If you disallow post event complaints (at which time there would be more information about an engine, possibly from having met/talked with the author at the event), my guess is that the amount of pre-event screening will need to increase (substantially).

The real question is: Should there be a computer chess body (eg the hierarchical ICGA) forming tribunals post-event and inflicting punishments?

I would venture yes, when there are new pieces of evidence after the conclusion of the championships. For instance, Lewis Hamilton was disqualified from the 2009 Australian Grand Prix: The Trulli/Hamilton case was reopened to examine new evidence, and both drivers were summoned to a stewards' inquiry prior to the Malaysian Grand Prix. ... Or more provocatively, the 2008 Crash Gate, which was only exposed when one of the involved drivers was sacked months later (FIA president Max Mosley stated the sport could take no action based on "speculation" at the time of the race, but then when Piquet made accusations, they "retroactively" brought charges). [NASCAR had its own Spingate in 2013, whose investigation was concluded in a shorter frame].

[hyatt]

I believe NASCAR always impounds the winning car. I don't know that they do wind-tunnel testing very often, but I have seen cases where they went over the car with a fine-toothed comb, checking for things like ride-height, rear spoiler angle, the usual engine checks (CID, the infamous restrictor-plate measurements, body width/height/etc. And there have been clever attempts to work the system. A few years ago someone used shocks that would maintain the NASCAR-mandated minimum ride height, UNTIL you hit the track for a couple of laps. The down-force caused by the speed would force the car 1" or so lower than the allowed minimum. Back off for a lap (such as the victory lap) and when you hit the pits or victory lane, you are back to regulation height. How they figured that one out I don't remember.

And don't forget "deflate-gate". The NE Patriots just got slammed several months after the playoff game (NFL) where they used under-inflated footballs. Quarterback was suspended for the first four games of the NEXT season, several months AFTER the game in question. So a year feels reasonable, IMHO, as a statute of limitations. But IF someone is found to violate the rules, there should be nothing to say that at that point, the statutes are no longer binding and past events can be re-checked as well.

Sort of like the IRS audit system in the US. The general rule is they can go back for the previous 3 years. But if they find something, that opens pandora's box and they can go back as far as they want although they generally don't go back beyond where the problematic issue was first seen (such as a 5 - year depreciation that was done in a fishy way).

[BB+]

I don't know that they do wind-tunnel testing

With restrictor-plate racing, the spoiler angle is so important that teams will come up with ways that the angle changes due to wind effects (particularly down force). They used to measure the angle (as the cars came into pit lane after the cooldown lap, as this is trivial to modify by a mechanic in a second or two) with some weights placed on it, but again, weights need not be a fair surrogate for wind effects. Off the top of my head I don't recall a specific case of wind-tunnel testing (though I vaguely remember at least the idea of this was brought up in one case), but then I don't think they always say everything they do in the inspection.

A few years ago someone used shocks that would maintain the NASCAR-mandated minimum ride height, UNTIL you hit the track for a couple of laps. The down-force caused by the speed would force the car 1" or so lower than the allowed minimum. Back off for a lap (such as the victory lap) and when you hit the pits or victory lane, you are back to regulation height. How they figured that one out I don't remember

Given the above spoiler-plate shenanigans (which were common even back in 1995-2000 when I was watching it), it's not difficult to see how someone would come up with the idea to try this. Undoubtedly it's one of about 100 (or 1000) different ways a team can try to bend the rules, and NASCAR can really only spot-check them, moreover even with a fine-tooth comb it is not easy to catch everything (teams have also been famous for sticking a "obvious" thing for the inspectors to catch pre-race, in the hopes that would be satisfied they did their job, and not look at something really subtle).

On May 6, the NFL published a 243-page investigative report regarding the deflation of footballs used in the AFC Championship game.

Heaven forfend the ICGA might come to this...

The investigation concluded that it was "more probable than not" that New England Patriots equipment personnel were deliberately circumventing the rules. ... Further, Brady was implicated as it being more probable than not that he was aware of the deflation. ... (link)

Well, in any case the suspension is now being appealed, though not to a "neutral" arbitrator as the player's union had desired.

[hyatt]

This nonsense goes on everywhere. I've seen a ton of bogus things in drag racing. Quite a few years ago we had a guy that was cutting 0.00 reaction times every single time. Turns out he took advantage of several things.

(1) automatic transmission, where the usual starting line procedure is that after the second staging light comes on for both cars, both floor the accelerator, but hold the brake, letting the torque converter stall at max engine rpm. Then to leave the line, remove your foot when you feel the green is almost on. The trick was to use what is often called a burnout brake, it is a solenoid built into the front brake lines so that you stomp the brakes hard, press the button, then release the brakes. The rear wheel brakes release, but the front ones stay on. Helps to do a burnout without moving. This guy took that one step further. Solenoid/valve on BOTH brake lines (front and rear). Tiny hidden photo-detector hidden in left door mirror. He would tune this during practice rounds to that it would detect the tree yellow lights coming on and release the brakes at the perfect point to avoid a red light, and have no lost reaction time either. They eventually created a class just for that crap, which seemed a bit pointless to me. Another guy (bracket-racing, where you choose your ET, if you beat that ET you lose (called breaking out), otherwise if you cross the finish line before your opponent, or if he breaks out, you win. You can guess where this is going. With a little bit of computer help, you can dial in a 15.00 second E/T and hit it dead on every time, with the computer controlling the brakes. This used to be a fun class to race in, you didn't need a 5 second car, you just needed to be able to drive. You had to beat your opponent but not break out. Until the computer started making perfect runs. That was NOT easy to detect, particularly when it has never been done before. You can go out and buy a stealth nitrous system (runs the nitrous line underneath the intake, hides the bottle under the seat, etc to make it very hard to detect. So long as they don't do a 500hp shot and smoke the tires at the 1/8 mile mark or something stupid.

Whenever there is a rule, you will soon find a rule-breaker. And you often will wonder "why? what's the fun in doing that?" I understood Taft and group that did the electronics to beat Vegas 40 years ago, because that was about money. But much of drag racing is about test and tune nights where you race your friends to compare cars or driving.

[Rebel]

For instance, Rajlich has said he thinks the ICGA should check others (perhaps everyone) the same way that Rybka was investigated, but the impetus for said measure is unclear.

It's indeed one of the first things Rajlich said, why only me? Check others too.

If you feel you have done nothing wrong in the belief others have done exactly the same (taking ideas) as he pointed out himself (of which there is enough evidence others did likewise) then it's a logical thing to say. Why me only?

As I pointed out on several occassions in the other thread (I am still waiting for your comments) there were no (addapted | new) rules, no guidance from the ICGA to the new arisen situation of exploding knowledge available for everyone with a few strikes on the keyboard and how and to what extend the "KNOWLEDGE" should and could be used.

Before I start repeating myself too much, see: [ http://www.open-chess.org/viewtopic.php?p=21759#p21759 ]

Self reflection and introspection aren't the strongest points of the ICGA. They are not blameless in this case.