Android UCI engine test tournament

[AartBik]

I am generalizing the engine match feature in Chess for Android into an engine tournament feature and, as a test, ran a tournament between a few of the Android UCI engines recently released on Jim Ablett's Chess Projects Website, as well as the built-in Java engine (using the fast-move setting, 32MB hash, and Nalimov, Gaviota, Scorpio, Robbobases endgame tablebases on SD card). More elaborate tournament results to follow (new engines still welcome!).

Code: Select all

                           1          2          3          4          5          6          
1   Stockfish 2.1.1 JA     ********** 11110½11½1 111½111111 1111111111 1111111111 1111111111  47.5/50
2   Komodo32 1.3 JA        00001½00½0 ********** 1111111111 1111111111 1111111111 1111111111  42.0/50
3   Rotor 0.4              000½000000 0000000000 ********** 1111011111 1111111111 1111111111  29.5/50
4   Diablo 0.5.1           0000000000 0000000000 0000100000 ********** 1111111111 1111111111  21.0/50
5   Chess for Android      0000000000 0000000000 0000000000 0000000000 ********** 1111½111½1   9.0/50
6   Simplex 0.9.7 rev 180  0000000000 0000000000 0000000000 0000000000 0000½000½0 **********   1.0/50

[thorstenczub]

so we are only a few steps away from having an ARENA-like program done for chess under android.

wow !!

one of the strongest features of arena is, that you can edit the tournaments,
e.g. throw out an engine, replace it with another engine.
and NOW your results from the last run are not overwritten but
used for the NEW tournament.

this way you can enhance tournaments with new versions of a program.
e.g. you had a big tournament with 6 participants and komodo1.3.
now komodo1.4 is coming out, you put komodo in, and another engine
and run the REST of the games and you will see how komodo1.4 ranks in relation to the old komodo1.3 results in the tournament.

[AartBik]

one of the strongest features of arena is, that you can edit the tournaments,
e.g. throw out an engine, replace it with another engine.

Thorsten, always a few feature requests ahead of me
For now, I am finalizing the last details of the new tournament feature, such as continuing a tournament in progress later (useful if you need the device for something else, or when an UCI engine crashes).

[AartBik]

Below the results of a tournament between the different Android Stockfish versions and compilations (from Jim Ablett and myself).

Code: Select all

Chess for Android Tournament  May 2011

                        1          2          3          4          5          6          
1   Stockfish 2.0       ********** 01½11½01½0 0½1½0100½½ 010½½½½½½1 ½1011½½½½1 ½11001½½11  27.5/50
2   Stockfish 2.1 JA    10½00½10½1 ********** 1½½1½1½100 ½½11½00111 1½011½½010 1½00001½1½  27.0/50
3   Stockfish 2.1.1 JA  1½0½1011½½ 0½½0½0½011 ********** ½½½1½10½½½ 0½0010½½½½ ½½1½101½½1  25.5/50
4   Stockfish 1.9       101½½½½½½0 ½½00½11000 ½½½0½01½½½ ********** 100½1½0½1½ 1½10½0111½  24.5/50
5   Stockfish 2.1       ½0100½½½½0 0½100½½101 1½1101½½½½ 011½0½1½0½ ********** ½½010½0101  24.0/50
6   Stockfish 1.8       ½00110½½00 0½11110½0½ ½½0½010½½0 0½01½1000½ ½½101½1010 **********  21.5/50

[AartBik]

Test tournament with different Android versions of Miguel Ballicora's Gaviota and Michel Van den Bergh's GNU chess.

Code: Select all

                           1          2          3          4          5          
1   GNU Chess 5.07.170.7b  ********** 000½10011½ 11½1110110 1100½01001 0111101111  24.0/40
2   gaviota v0.80.0.107    111½01100½ ********** ½010110100 0100½11110 ½101000011  20.5/40
3   GNU Chess 5.07.153.3b  00½0001001 ½101001011 ********** 0101010½01 0111111½10  20.0/40
4   gaviota v0.82-beta2    0011½10110 1011½00001 1010101½10 ********** 00110001½0  19.0/40
5   gaviota v0.83          1000010000 ½010111100 1000000½01 11001110½1 **********  16.5/40

[AartBik]

Yet another test tournament with different Android versions of Jan Brouwer's Rotor and Ben-Hur Carlos Vieira Langoni Jr.'s RedQueen.

Code: Select all

                       1          2          3          4          5          
1   Rotor 0.6a         ********** ½111111011 1011111101 111½111111 11½1111111  35.5/40
2   RedQueen 0.9.8     ½000000100 ********** 1½01½½010½ 11½1½0110½ ½1½1½011½1  20.0/40
3   Rotor 0.4 JA       0100000010 0½10½½101½ ********** 1½0½010110 10011½½101  18.0/40
4   RedQueen 0.9.5     000½000000 00½0½1001½ 0½1½101001 ********** ½1101½10½½  15.0/40
5   RedQueen 0.9.8 JA  00½0000000 ½0½0½100½0 01100½½010 ½0010½01½½ **********  11.5/40

[thorstenczub]

i see you have fun with these tournaments )

the clou of the reuse of the old data is, that your tournament grows and grows,
and is suddenly a rating list instead of a small tournament.

all the old results were taken over to the new tournament.

if in the new tournament a new version of komodo gets a late entry,
there will be many komodo new versus other participants games.
but the result/game stockfish vs. fruit, which was played
in the old tournament, can be used without a replay because
its data can be used without changes for the new tournament.

this way you only have to play the new komodo games, and the tournament
is finished.

i appreciate that you implemented the feature to interrupt and later continue
the tournament.

thanks aart !!!

[AartBik]

I conducted a full tournament between 17 currently available UCI engines for Android as well as the simple built-in Java engine. The tournament consisted of pair-wise matches between all engines for 10 games with random book opening play, one second-per-move, 32MB hash, and 8MB tablebases cache with the 3- and 4-piece Nalimov, Gaviota, Scorpio, Robbobases endgames on SD card. The 1530 games lasted about two days running full time on a Nexus One phone (without a single crash!). The results are shown below.

Code: Select all

1  Stockfish 2.0                   157.5/170
2  RobboLito 0.085g3l              152.5/170
3  IvanHoe-Beta version 999947c    145.0/170
4  Komodo32 1.3 JA                 137.0/170
5  Toga II 1.4.1SE                 130.5/170
6  Pepito v1.59                    104.0/170
7  Daydreamer 1.75 JA              100.5/170
8  Rotor 0.6a                       98.0/170
9  gaviota v0.80.0.107              87.5/170 4647.75
10 GNU Chess 5.07.170.7b            87.5/170 4455.25
11 RedQueen 0.9.8                   75.5/170
12 GreKo 8.0                        71.0/170
13 Jazz v444 JA                     50.5/170
14 BikJump v2.1P                    49.0/170
15 Diablo 0.5.1                     38.5/170
16 ZCT-0.3.2500                     34.0/170
17 Chess for Android                10.5/170
18 Simplex 0.9.7 rev 180             1.0/170

[thorstenczub]

many thanks aart.

[Gargamel]

Hello,

Another tournament :

Htc legend, chess for Android, 16MB hash, no tablebase.

Chess for Android Match
France, 2011.05.18 - 2011.05.21
--------------------------------------------- Score -- Stockfish----- RobboLito-----IvanHoe-Be---- Toga II ---- Chess for
-----------------------------------------------------------------------------------------------------
1: Stockfish 2.1 ------------------------- 30.5 / 40 XXXXXXXXXX =11==10=== 0===10=011 11=1111111 1111111111
2: RobboLito 0.085e4l ---------------- 28.5 / 40 =00==01=== XXXXXXXXXX =0011=01=1 11=11=1111 1111111111
3: IvanHoe-Beta version 999947c -- 27.0 / 40 1===01=100 =1100=10=0 XXXXXXXXXX 1==1=10111 1111111111
4: Toga II 1.4.1SE ----------------------- 14.0 / 40 00=0000000 00=00=0000 0==0=01000 XXXXXXXXXX 1111111111
5: Chess for Android ------------------- 0.0 / 40 0000000000 0000000000 0000000000 0000000000 XXXXXXXXXX
-----------------------------------------------------------------------------------------------------
100 games: +45 =19 -36