A Computer Program to Detect Possible Cheating in Chess

[Jeremy Bernstein]

http://www.nytimes.com/2012/03/20/scien ... chess.html

Here's the prof's page: http://www.cse.buffalo.edu/~regan/chess/fidelity/

[User923005]

http://www.nytimes.com/2012/03/20/scien ... chess.html

Here's the prof's page: http://www.cse.buffalo.edu/~regan/chess/fidelity/

His approach is to see if you play at your level. IOW, if an IM beats a GM and makes excellent moves, then he's guilty of cheating.

No thanks to this method.

[kingliveson]

http://www.nytimes.com/2012/03/20/scien ... chess.html

Here's the prof's page: http://www.cse.buffalo.edu/~regan/chess/fidelity/

His approach is to see if you play at your level. IOW, if an IM beats a GM and makes excellent moves, then he's guilty of cheating.

No thanks to this method.

I have to agree.

[Jeremy Bernstein]

http://www.nytimes.com/2012/03/20/scien ... chess.html

Here's the prof's page: http://www.cse.buffalo.edu/~regan/chess/fidelity/

His approach is to see if you play at your level. IOW, if an IM beats a GM and makes excellent moves, then he's guilty of cheating.

No thanks to this method.

I have to agree.

While we all wish that, one day, we'll just wake up and play GM-level chess based on some innate genius we are certain we had from the start, it doesn't work that way. As I understood, he's interested in sample sizes > 1 game: he's generally analyzing match performance. Which is still a small sample, but it's still possible to identify "outliers" based on comparison with previous (and future) games from the same player.

It's not simply a question of catching someone playing excellent moves. It's a question of catching someone making a large number of _improbably_ excellent moves.

jb

[User923005]

Did you ever have a day when your thinking was especially sharp and you played over your head?
I have.
Of course, I have the opposite sort of days too.

[Jeremy Bernstein]

Did you ever have a day when your thinking was especially sharp and you played over your head?
I have.
Of course, I have the opposite sort of days too.

Of course, that's why he analyzes more than single moves in single games, and why his model predicts (for instance) 8.35/10 matches for a 2700 rated player. This is relevant, from his fidelity page:

The main statistical principle which these pages show has been misunderstood by the chess world is that a move that is given a clear standout evaluation by a program is much more likely to be found by a strong human player. And a match to any engine on such a move is much less statistically significant than one on a move given slight but sure preference over many close alternatives...

Here's the referenced data, which addresses your point: http://www.cse.buffalo.edu/~regan/chess ... esults.txt.

The kind of statistics he's interested in revolve around those unforced, reasonably non-tactical positions where engines might consider 5 (or 20!) moves to be all more or less equivalent, with a slight preference for one move due to some subtle point 20 plies in. He's tracking how often a player picks this type of slightly preferred move.

jb

[pgn4web]

While we all wish that, one day, we'll just wake up and play GM-level chess based on some innate genius we are certain we had from the start, it doesn't work that way. As I understood, he's interested in sample sizes > 1 game: he's generally analyzing match performance. Which is still a small sample, but it's still possible to identify "outliers" based on comparison with previous (and future) games from the same player.

What I dislike very much about this kind of article is the rush to conclusions. I believe that any system such the one described, will at best be able to raise a "legitimate suspicion" that something is going on; however those system are often described as a tool to "detect", "proof" that cheating occurs. In my opinion you can not proof cheating only using those kind of statistical analysis.

To make a comparison: a possible way to detect tax evasion is to compare the declared earnings with the lifestyle of an individual; if I claim to be jobless on state benefits and then I continuously travel first class around the world, that might be an indication to raise a "legitimate suspicion"; I hope there's no legal system where such a "legitimate suspicion" would be enough for conviction or slander. But the tax office might start an investigation.
Back to chess, the issue here is: at the end of a tournament, after this wonderful new system detects that I played much better than it was expected from me from past performance, what can actually be done? Do you "convict" me based on this "legitimate suspicion"? It seems unacceptable to me. How do you go back and proof that actual cheating occurred?

That's were those systems fall short, even assuming the validity of the statistical analysis with such a small sample.

PS: maybe something good will come out of this anyway, in an unexpected form. Writing the sentence "detects that I played much better than it was expected from me from past performance", I immediately thought this is exactly what the Elo rating system is supposed to do. This leads me to two conclusions: maybe cheating could be detected just by looking at the difference between rating performance and players rating: if you perform 300 Elo points more than your rating you "must" have cheated! Alternatively the author of the article quoted might just have invented a new and better rating system (where better means more accurate in predicting player's results).

[Jeremy Bernstein]

While we all wish that, one day, we'll just wake up and play GM-level chess based on some innate genius we are certain we had from the start, it doesn't work that way. As I understood, he's interested in sample sizes > 1 game: he's generally analyzing match performance. Which is still a small sample, but it's still possible to identify "outliers" based on comparison with previous (and future) games from the same player.

What I dislike very much about this kind of article is the rush to conclusions. I believe that any system such the one described, will at best be able to raise a "legitimate suspicion" that something is going on; however those system are often described as a tool to "detect", "proof" that cheating occurs. In my opinion you can not proof cheating only using those kind of statistical analysis.

To make a comparison: a possible way to detect tax evasion is to compare the declared earnings with the lifestyle of an individual; if I claim to be jobless on state benefits and then I continuously travel first class around the world, that might be an indication to raise a "legitimate suspicion"; I hope there's no legal system where such a "legitimate suspicion" would be enough for conviction or slander. But the tax office might start an investigation.
Back to chess, the issue here is: at the end of a tournament, after this wonderful new system detects that I played much better than it was expected from me from past performance, what can actually be done? Do you "convict" me based on this "legitimate suspicion"? It seems unacceptable to me. How do you go back and proof that actual cheating occurred?

That's were those systems fall short, even assuming the validity of the statistical analysis with such a small sample.

PS: maybe something good will come out of this anyway, in an unexpected form. Writing the sentence "detects that I played much better than it was expected from me from past performance", I immediately thought this is exactly what the Elo rating system is supposed to do. This leads me to two conclusions: maybe cheating could be detected just by looking at the difference between rating performance and players rating: if you perform 300 Elo points more than your rating you "must" have cheated! Alternatively the author of the article quoted might just have invented a new and better rating system (where better means more accurate in predicting player's results).

True. However, as a tool to either raise a flag for additional research (e.g. "Bear in mind the policy stated elsewhere on this site that statistical evidence should be secondary to physical or observational evidence of possible wrongdoing.") or to statistically exonerate an accused player (as in the cited Mamedyarov case), it has promise.

What the press says and what the professor says are different, and that's not his fault.

jb

[User923005]

Did you ever have a day when your thinking was especially sharp and you played over your head?
I have.
Of course, I have the opposite sort of days too.

Of course, that's why he analyzes more than single moves in single games, and why his model predicts (for instance) 8.35/10 matches for a 2700 rated player. This is relevant, from his fidelity page:

The main statistical principle which these pages show has been misunderstood by the chess world is that a move that is given a clear standout evaluation by a program is much more likely to be found by a strong human player. And a match to any engine on such a move is much less statistically significant than one on a move given slight but sure preference over many close alternatives...

Here's the referenced data, which addresses your point: http://www.cse.buffalo.edu/~regan/chess ... esults.txt.

The kind of statistics he's interested in revolve around those unforced, reasonably non-tactical positions where engines might consider 5 (or 20!) moves to be all more or less equivalent, with a slight preference for one move due to some subtle point 20 plies in. He's tracking how often a player picks this type of slightly preferred move.

jb

Suppose he/she uses a computer to prepare a line before the match.

[Jeremy Bernstein]

Suppose he/she uses a computer to prepare a line before the match.

Obviously his analysis only begins after book/opening play is over.

jb