IPON Rating's List Bash Script

[Uly]

I saw the thread and have to agree with Uly...publishing the games wont ruin his list.

Whoa! Amazing how we had the same idea in very close times, with the difference that I did nothing and you fully implemented it (it would have taken me more than two weeks to do so and my results wouldn't have been as good as yours so I appreciate the effort!)

Now I'm going to re-post this at Rybka Forum

[Jeremy Bernstein]

Crossposted:

http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=22936

LOL, Uly.

jb

[Uly]

Oh well, Prima went on a posting spree burying the crosspost, I deleted it and may attempt to repost it later.

[Prima]

Crossposted:

http://rybkaforum.net/cgi-bin/rybkaforu ... ?tid=22936

LOL, Uly.

jb

Now my response to other Threads, in Rybka forum, is blamed for truncating a Topic, as if I had the power to. Or as if THAT was my intention. What next? Blame me for Vas' breach of GPL and breaking the ICGA rule #2 repetitively?

It never ends .....

[kingliveson]

uh...what did I miss?

By the way, the weight used for Stockfish is incorrect. It should have been 36 instead of 46, and 40 seems more appropriate.

Code: Select all

engine_weight[18]=40

[Uly]

Now my response to other Threads, in Rybka forum, is blamed for truncating a Topic

No. It was a coincidence, but my cross-post was badly timed (as can be seen when a newly created thread rushes to be buried in the second page, regardless of who is making a posting spree).

I just wanted to explain why I deleted the thread.

[kingliveson]

Important correction to getting draws -- new source attached.

Removed:

Code: Select all

# difference weight table scale
{
black_difference_table[1]=45 # 0-8
black_difference_table[2]=35 # 9-16
black_difference_table[3]=25 # 17-24
black_difference_table[4]=15 # 25-32
black_difference_table[5]=4 # > 32

white_difference_table[1]=10 # > 32
white_difference_table[2]=8 # 25-32
white_difference_table[3]=6 # 17-24
white_difference_table[4]=4 # 9-16
white_difference_table[5]=2 # 0-8
}

# difference weight table scale
{
black_difference_table[1]=45 # 0-8
black_difference_table[2]=35 # 9-16
black_difference_table[3]=25 # 17-24
black_difference_table[4]=15 # 25-32
black_difference_table[5]=4 # > 32

white_difference_table[1]=10 # > 32
white_difference_table[2]=8 # 25-32
white_difference_table[3]=6 # 17-24
white_difference_table[4]=4 # 9-16
white_difference_table[5]=2 # 0-8
}

# Weight look-up table scale
Weight_Lookup_Scale()
{
  if [ $lookup_table_bit -eq 0 ]; then    
    if (( $engine_weight_difference <= 8 )); then
     black_table_weight=${black_difference_table[1]} 
    elif (( $engine_weight_difference >= 9 && $engine_weight_difference <= 16 )); then
     black_table_weight=${black_difference_table[2]}
    elif (( $engine_weight_difference >= 17 && $engine_weight_difference <= 24 )); then
     black_table_weight=${black_difference_table[3]}
    elif (( $engine_weight_difference >= 25 && $engine_weight_difference <= 32 )); then
     black_table_weight=${black_difference_table[4]}
    elif (( $engine_weight_difference > 32 )); then
     black_table_weight=${black_difference_table[5]}
    fi
    white_table_weight=0
  else
    if (( $engine_weight_difference <= 8 )); then
     white_table_weight=${white_difference_table[5]} 
    elif (( $engine_weight_difference >= 9 && $engine_weight_difference <= 16 )); then
     white_table_weight=${white_difference_table[4]}
    elif (( $engine_weight_difference >= 17 && $engine_weight_difference <= 24 )); then
     white_table_weight=${white_difference_table[3]}
    elif (( $engine_weight_difference >= 25 && $engine_weight_difference <= 32 )); then
     white_table_weight=${white_difference_table[2]}    
    elif (( $engine_weight_difference > 32 )); then
     white_table_weight=${white_difference_table[1]}  
    fi
    black_table_weight=0
  fi
}

Replaced with:

Code: Select all

# Draw formula -- decrease to increase win
Draw_Weight_Scale()
{ 
  if [ $scale_side_switch -eq 0 ]; then
    black_scale_weight=`expr $weaker_engine_weight*.425 | bc` 
  else
    white_scale_weight=`expr $weaker_engine_weight*.185 | bc`
  fi
}

The new approach is more efficient; rather than hard-coding constants on how often draws occur, the formula uses the engines' defined weights. The current ratio is tuned to IPON's list previously posted.

[kingliveson]

So what was added this time? User interaction, Gauntlet and Round-Robin tournaments. It is pretty easy to use. Execute it and just follow the prompt -- source and database attached.

-Run a Gauntlet tournament after selecting an engine, then number of rounds. Round Robin tournament will allow all engines to play against each another. A game database will be generated

-Add/remove engine requires 3 changes:
MAXENGINECOUNT=23
chess_engine[23]="Engine Name"
engine_weight[23]=100

-To increase/decrease wins, edit values in Draw_Weight_Scale()

Code: Select all

Chess Engine Rating List Simulator

Enter 1 for Gauntlet or 2 for Round-Robin: 1
List Engines? Enter y or n: n
Enter engine number: 1
Enter number of rounds: 100
Generating game database.......................Complete!

Code: Select all

Chess Engine Rating List Simulator

Enter 1 for Gauntlet or 2 for Round-Robin: 2
List Engines? Enter y or n: n
Enter number of rounds: 300
Generating game database......................Complete!

Simulated Tournament:

Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Houdini 2.0 STD                : 2649    9   9  6300    81.7 %   2390   19.8 %
  2 Critter 1.2                    : 2604    8   8  6300    77.2 %   2392   22.8 %
  3 Komodo64 3 SSE42               : 2595    8   8  6300    76.2 %   2392   23.6 %
  4 Deep Rybka 4.1 SSE42           : 2589    8   8  6300    75.5 %   2393   23.8 %
  5 Stockfish 2.1.1 JA             : 2584    8   8  6300    75.0 %   2393   24.8 %
  6 Naum 4.2                       : 2484    7   7  6300    62.2 %   2398   29.4 %
  7 Deep Shredder 12               : 2484    7   7  6300    62.1 %   2398   29.4 %
  8 Spike 1.4 32b                  : 2448    7   7  6300    57.0 %   2399   31.3 %
  9 Deep Sjeng c't 2010 32b        : 2443    7   7  6300    56.1 %   2400   31.3 %
 10 Deep Junior 12.5               : 2426    7   7  6300    53.6 %   2401   31.4 %
 11 Gull 1.2                       : 2394    7   7  6300    48.8 %   2402   31.2 %
 12 HIARCS 13.2 MP 32b             : 2387    7   7  6300    47.8 %   2402   32.1 %
 13 Hannibal 1.1                   : 2374    7   7  6300    45.9 %   2403   32.1 %
 14 Protector 1.4.0 x64            : 2366    7   7  6300    44.6 %   2403   31.0 %
 15 spark-1.0 SSE42                : 2362    7   7  6300    44.1 %   2404   30.3 %
 16 Deep Onno 1-2-70               : 2328    7   7  6300    39.1 %   2405   30.2 %
 17 Zappa Mexico II                : 2316    7   7  6300    37.3 %   2406   28.6 %
 18 Loop 13.6/2007                 : 2303    8   8  6300    35.5 %   2406   27.1 %
 19 Strelka 2.0 B                  : 2236    8   8  6300    27.0 %   2410   22.6 %
 20 Crafty 23.3 JA                 : 2177    9   9  6300    20.5 %   2412   19.0 %
 21 Umko 1.2 SSE42                 : 2177    9   9  6300    20.5 %   2412   19.7 %
 22 Jonny 4.00 32b                 : 2075   11  11  6300    12.2 %   2417   12.0 %

At last, your own chess engine rating's list! Just pick weights from lists such as CEGT, CCRL, SWCR, SSDF. When people ask for games, you can always provide database with results useful for generating a stats table such as above.

[kingliveson]

A new feature has been added; it now supports choosing which engines participate in a tournament (Gauntlet or Round-Robin) -- Just like regular GUIs. Source and game database are attached.

Usage/Description:

Code: Select all

This program (bash script) is used for simulating and predicting outcome for chess engine matches and tournaments.
It generates a game (CRLS.pgn) database at the conclusion of each run. 
The results should be compared to rating lists such CEGT, CCRL, SWCR, SSDF, etc.

How to run:
sh CRLS or ./CRLS
# Windows users need to research openSUSE (Linux) LiveCD, or cygwin and notepad++
# There is also http://win-bash.sourceforge.net/

Prompt/Usage:
"Enter 1 for Gauntlet or 2 for Round-Robin: " 
# Gauntlet allows an engine to compete against a group of engines
# Round-Robin allows all engines in a tournament to play each other
"List Engines? Enter y or n: " 
# List all available chess programs 

"Enter engine number (1-25), l (list engines), f (finish), or q (quit): " 
# Choose up to 16 programs. Enter each, one at a time
# Enter l to see available programs
# Enter f after selecting the program(s) 
# Enter p to see already selected programs
# Enter q to quit the program

"Enter number of round(s) (1-999999): " 
# Enter the number of round(s)/game(s) to play

Gauntlet:
"Enter engine number (1-25): " 
# Enter engine number from available engines' list
"Play against all engines? Enter y or n: " 
# If yes, selected engine plays all available programs
"Enter engine number (1-25), l (list engines), f (finish), or q (quit): " 
# Choose program(s) to play against selected engine. 

Round-Robin
"List Engines? Enter y or n: " 
# List all available chess programs
"Play against all engines? Enter y or n: " 
# If yes, all available programs play round-robin style tournament
"Enter engine number (1-25), l (list engines), f (finish), or q (quit): " 
# Select the programs to compete in the tournament

Source Editing

Change Engine Weight
# For example, Critter 1.2:
engine_weight[21]=445 # 40

Add/Remove engine requires 3 changes:
# For example: adding a new program
MAXENGINECOUNT=26
chess_engine[26]="Engine Name"
engine_weight[26]=100

To increase/decrease wins, edit values in Draw_Weight_Scale()

Here are some stats from a simulated tournament of all "installed" engines playing each other:

Code: Select all

Games        : 150000 (finished)

White Wins   :  58442 (39.0 %)
Black Wins   :  44737 (29.8 %)
Draws        :  46821 (31.2 %)
Unfinished   :      0

White Perf.  : 54.6 %
Black Perf.  : 45.4 %

9/10/2011 7:00:20 PM :

    Program                          Elo    +   -   Games   Score   Av.Op.  Draws

  1 Houdini 2.0 STD                : 3023    6   6 12000    79.0 %   2793   25.4 %
  2 Houdini 1.5a                   : 3012    6   6 12000    77.9 %   2793   26.3 %
  3 Critter 1.2                    : 2972    6   6 12000    73.5 %   2795   29.3 %
  4 Komodo64 3 SSE42               : 2972    6   6 12000    73.5 %   2795   29.9 %
  5 Deep Rybka 4.1 SSE42           : 2969    6   5 12000    73.1 %   2795   30.3 %
  6 Stockfish 2.1.1 JA             : 2960    5   5 12000    72.1 %   2795   30.8 %
  7 IvanHoe 9.47b                  : 2957    5   5 12000    71.7 %   2795   31.2 %
  8 Deep Shredder 12               : 2869    5   5 12000    60.0 %   2799   35.3 %
  9 Naum 4.2                       : 2860    5   5 12000    58.6 %   2799   35.1 %
 10 Spike 1.4 32b                  : 2828    5   5 12000    53.9 %   2801   36.4 %
 11 Deep Sjeng c't 2010 32b        : 2826    5   5 12000    53.6 %   2801   36.9 %
 12 Deep Fritz 12 32b              : 2810    5   5 12000    51.3 %   2802   36.8 %
 13 Deep Junior 12.5               : 2806    5   5 12000    50.6 %   2802   36.3 %
 14 Gull 1.2                       : 2785    5   5 12000    47.5 %   2803   36.8 %
 15 HIARCS 13.2 MP 32b             : 2770    5   5 12000    45.3 %   2803   37.3 %
 16 Hannibal 1.1                   : 2757    5   5 12000    43.4 %   2804   35.6 %
 17 Protector 1.4.0 x64            : 2748    5   5 12000    42.0 %   2804   35.7 %
 18 spark-1.0 SSE42                : 2747    5   5 12000    41.8 %   2804   36.0 %
 19 Deep Onno 1-2-70               : 2711    5   5 12000    36.7 %   2806   33.6 %
 20 Zappa Mexico II                : 2701    5   5 12000    35.3 %   2806   31.8 %
 21 Loop 13.6/2007                 : 2683    5   5 12000    32.8 %   2807   30.9 %
 22 Strelka 2.0 B                  : 2611    6   6 12000    24.1 %   2810   24.9 %
 23 Umko 1.2 SSE42                 : 2566    6   6 12000    19.5 %   2812   21.1 %
 24 Crafty 23.3 JA                 : 2559    6   6 12000    18.9 %   2812   21.0 %
 25 Jonny 4.00 32b                 : 2499    7   7 12000    14.0 %   2815   15.7 %

And by the way, what you published as results (which snippets I pasted), can easily be generated without ever playing a single game. I can produce you a script that does exactly that.

Franklin

In advance and accordiing to the other list? I would be really impressed!

BYe
Ingo

I am hoping that after this demonstration, Ingo would reconsider publishing actual games.

[ThinkingALot]

A new feature has been added; it now supports choosing which engines participate in a tournament (Gauntlet or Round-Robin) -- Just like regular GUIs. Source and game database are attached.

It's clear that your simulation is a fake. Gull is underrated .