Thoughts on Fruit=Rybka EVAL

[hyatt]

Overall, I just can't see much either way. There are various places where Rybka follows a different agenda (such as re-factoring the code), but on the other hand, there are an assortment of orderings (most rather minor by themselves) that are candidates for being different, that in fact are not. One could argue that in most of the places where there is a re-ordering, there is some external reason for this, possibly from the performance standpoint (like shelter_storm in pawn_eval), or maybe due to ease of development. However, I'm not completely convinced that every Rybka/Fruit difference could be explained by this.

An alternative example where Rybka does re-order a few Fruit elements is in the UCI parsing, where Fruit works alphabetically (binc/btime/depth/infinite/mate/movestogo/movetime/nodes/ponder/searchmoves/winc/wtime), while Rybka reorders this (winc/wtime/binc/btime/depth/infinite/movestogo/movetime/ponder), placing winc/wtime at the front (and not having mate/nodes/searchmoves). So other than winc/wtime at the front, it's alphabetical in Rybka -- but the winc/wtime change shouldn't be simply ignored. Again it is unclear to me whether there might be some external "testing"(?) reason why one might want to move winc/wtime to the front. Zach or I could track down the order in Rybka 1.6.1, if it is thought relevant.

Thank you for clarification.

To say it in a few words, you basically discovered that comparing the eval of 2 good chess programs contain about the same amount of chess knowledge and its implementation look similar.

I could have told you that from the beginning.

I don't see where that is the case at all. One could certainly compare Crafty and Fruit evals, Crafty is definitely stronger, but not by hundreds of Elo. That will shoot holes in your conclusion.

I don't get this claim of "everybody does the same things, in the same order, in the same way..." That is contrary to anything I have seen in looking at other programs over the years. It is more like "everybody does lots of the same things, but in different ways, in different orders, in different places..."

[kingliveson]

My thought is Vas should release Rybka 1.0 source code, and that is what you guys should be asking. There's no secret in Rybka 1.0, so it is not as if he would be revealing anything new. As far as am concerned, he was correctly found guilty and will remain so until he releases Rybka 1.0 source. If am able to dig up non-consequential code written about a decade ago when learning programming, am sure he has the source at his disposal. Ed, Chris, and the rest of our friends at Rybka forum can speculate from here till kingdom come -- it is meaningless and carries no weight.

Code: Select all

/*
Assignment:         2 (Simple GUI Composer)
Author:             Franklin Titus
Date:               02.07.02
*/

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;



public class siGuCo extends JFrame implements ItemListener, ActionListener,
                                            MouseListener, MouseMotionListener{
  JButton button;
  JList list ;
  JTextField text;
  JComboBox combobox;
  JLabel label;
  String[] num = {"One", "Two", "Three"};

  JLabel color;
  JLabel mouse;
  JLabel sel;

  Object source;
  Font font;

  int i;
  int r, g, b;

  JPanel west = new JPanel();
  JPanel center = new JPanel();
  JPanel south = new JPanel();

  String[] strOpt = {"Button", "List Box", "Text Box", "Combo Box", "Label"};
  private JRadioButton rButtons[];
  private JComboBox  fonts;
  private JButton undo = new JButton("Undo");
  private JButton clear = new JButton("Clear All");
  private ButtonGroup group;
  private JButton BColor = new JButton("Background Color");

  public siGuCo(){
    super("Simple GUI Composer");

    Container c = getContentPane();

    rButtons =new JRadioButton [strOpt.length];
    group = new ButtonGroup();
    west.setLayout(new GridLayout(0,1,0,0));
    for(i = 0; i < rButtons.length; i++){
      rButtons[i] = new JRadioButton(strOpt[i]);
      group.add(rButtons[i]);
      west.add(rButtons[i]);
      rButtons[i].addItemListener(this);
    }

    fonts = new JComboBox(GraphicsEnvironment.getLocalGraphicsEnvironment()
                          .getAvailableFontFamilyNames());

    undo.addActionListener(this);
    clear.addActionListener(this);
    BColor.addActionListener(this);
    center.addMouseListener(this);
    center.addMouseMotionListener(this);

    west.add(fonts);
    west.add(undo);
    west.add(clear);
    west.add(BColor);

    south.setLayout(new FlowLayout(3,20,0));
    sel = new JLabel();
    mouse = new JLabel();
    color= new JLabel() ;
    south.add(sel);
    south.add(mouse);
    south.add(color);

    west.setBackground(Color.blue );
    c.add(west,BorderLayout.WEST);
      center.setBackground(Color.getHSBColor(104.0f,401.0f,140.0f));
    c.add(center);
      south.setBackground(Color.cyan );
    c.add(south,BorderLayout.SOUTH);

    setSize(500,500);
    show();
     r = center.getBackground().getRed();
      g = center.getBackground().getGreen();
      b = center.getBackground().getBlue();
      color.setText( "Background" + " " +r+"R,"+ " " +g+"G," + " " +b+"B");

  }

   public static void main( String args[] ){

      siGuCo app = new siGuCo();

      app.addWindowListener(
         new WindowAdapter() {
            public void windowClosing( WindowEvent e ) {
               System.exit( 0 );
            }
         }
      );

   }

  public void itemStateChanged(ItemEvent e){
     font = new Font(fonts.getSelectedItem().toString() ,Font.PLAIN, 12 );
     source = e.getSource()  ;
     if ( source == rButtons[0] ){
      sel.setText("Button Selected");
      repaint();
      validate();
    }
    else if(  source == rButtons[1] ){
       sel.setText("List Selected");
       repaint();
       validate();
    }
    else if ( source == rButtons[2] ){
      sel.setText("Textfield Selected");
      repaint();
      validate();
    }
    else if ( source == rButtons[3]){
      sel.setText("Combo Box Selected");
      repaint();
      validate();
    }
    else if(  source == rButtons[4] ){
      sel.setText("Label Selected");
      repaint();
      validate();
    }



  }

  public void actionPerformed(ActionEvent butt){

    if (butt.getSource() == undo){
       if(center.getComponentCount()==0 ){
          center.removeAll();
          center.repaint();
          repaint();
          validate();
        }
        else {
          center.remove(center.getComponentCount()-1);
          center.revalidate();
          center.repaint();
          repaint();
          invalidate();
        }
     }
    if (butt.getSource() == clear){
      repaint();
      center.removeAll();
      center.repaint();
      repaint();
      invalidate();
    }

    if (butt.getSource() == BColor){
      center.setBackground( JColorChooser.showDialog(center,"color",Color.cyan )) ;
      r = center.getBackground().getRed();
      g = center.getBackground().getGreen();
      b = center.getBackground().getBlue();
      color.setText( "Background" + " " +r+"R,"+ " " +g+"G," + " " +b+"B");
      south.add(color).
      repaint();
      south.repaint();

    }

  }

  public void mouseClicked( MouseEvent m){
    mouse.setText("Clicked at" + " " + m.getX() + "x" + " " + m.getY()+"y");

  }

  public void mousePressed(MouseEvent m){
     mouse.setText("Pressed at" + " " + m.getX() + "x" + " " + m.getY()+"y");
  }

  public void mouseReleased(MouseEvent m){


   mouse.setText("Released at" + " " + m.getX() + "x" + " " + m.getY()+"y");
   center.setLayout(null);
   if ( source == rButtons[0] && rButtons[0].isSelected()   ){
      button = new JButton("Button");
      sel.setText("Button Selected");
      button.setFont(font);
      button.setSize(106,61);
      button.setToolTipText("This is a Button");
      button.setLocation(m.getPoint());
      center.add(button);
      center.repaint();
      repaint();
      validate();
    }
    else if(  source == rButtons[1] && rButtons[1].isSelected() ){
       list = new JList(num);
       sel.setText("List Selected");
       list.setFont(font);
       list.setVisibleRowCount(2);
       JScrollPane sPane = new JScrollPane(list);
	   sPane.setSize(56,39);
	   sPane.setLocation(m.getPoint());
	   center.add(sPane);
	   center.repaint();
       repaint();
       validate();
    }
    else if ( source == rButtons[2] && rButtons[2].isSelected()){
      text = new JTextField(9);
      sel.setText("Textfield Selected");
      text.setFont(font);
      text.setSize(80,60);
      center.add(text).setLocation(m.getPoint());
      center.repaint();
      repaint();
      validate();
    }
    else if ( source == rButtons[3] && rButtons[3].isSelected()){
      combobox= new JComboBox(num);
      sel.setText("Combo Box Selected");
      combobox.setFont(font);
      combobox.setSize(81,41);
      center.add(combobox).setLocation(m.getPoint());
      center.repaint();
      repaint();
      validate();
    }
    else if(  source == rButtons[4] && rButtons[4].isSelected()){
      label = new JLabel("Label");
      sel.setText("Label Selected");
      label.setFont(font);
      label.setSize(43,34);
      center.add(label).setLocation(m.getPoint());
      center.validate();
      center.repaint();
      repaint();
      validate();
    }



  }

  public void mouseEntered(MouseEvent m){
      mouse.setText("Mouse in Window");
  }

  public void mouseExited(MouseEvent m){
    mouse.setText("Mouse outside Window");
  }

  public void mouseMoved( MouseEvent m){
    mouse.setText("Moved at" + " " + m.getX() + "x" + " " + m.getY()+"y");
  }

  public void mouseDragged( MouseEvent m){
    mouse.setText("Dragged at" + " " + m.getX() + "x" + " " + m.getY()+"y");
  }

}

You want to help Vas? Ask him to release Rybka 1.0 source code.

[hyatt]

No doubt that would answer a lot of questions. I am afraid, however, that the answers would still not be very popular with a few, because there is no doubt that the source will show _exactly_ what the disassembly has shown. It would just be easier to read.

[BB+]

Thank you for clarification.

To say it in a few words, you basically discovered that comparing the eval of 2 good chess programs contain about the same amount of chess knowledge and its implementation look similar.

I could have told you that from the beginning.

You specifically asked me for the ordering of terms in evaluation. I even sent you an message, saying that it was a nontrivial task for me to chase this down and present in a proper manner, and that if you weren't serious about this, that I considered it to be trolling. Then you respond in the manner above ("look similar" and "could have told you"). I conclude you are trolling.

Furthermore, it is not the "amount" of knowledge that matters, but the "type" also. Even if we merely weighed the "amount": One can note that Rybka 3, a good chess program, has much more chess knowledge, with a different implementation. One can note that HIARCS from 2005, a good chess program, has much more chess knowledge, with a different implementation. One can note that...

But you are not a chess programmer.

If I had distributed the engine I wrote, perhaps then you would argue that I am a "jealous competitor" of Vas? This two-way straddle gets tiresome.

And you haven't tried Junior, Hiarcs, Fritz.

If I thought it might convince you, I would ask MarkU, SMK, and the Junior team from a code sample from 2005 (Fritz seems harder). I think they are all sufficiently professional in their archiving that this well may be possible. But then should I expect some new angle would be contrived, such as Vas being an IM so he understands evaluation (why then, pray tell, did the evaluation from an IM like Larry Kaufman look so different?), or something of similar limited value?

But you can take Rebel, most of the eval stuff is an easy read.

One concludes that REBEL is a good chess engine with a notably different (and dissimilar) evaluation function than either Rybka or Fruit. E.g., neither Fruit nor Rybka looks at hanging pieces, neither Fruit nor Rybka has a king safety module that looks much like REBEL's, neither Fruit nor Rybka has handles isolated pawns via a square-based criterion. These are the three principal specific things mentioned in Inside REBEL. It is hard to say too much about implementation as that is more dependent on the source code, as compared to a general description.

[Rebel]

I even sent you an message, saying that it was a nontrivial task for me to chase this down and present in a proper manner, and that if you weren't serious about this, that I considered it to be trolling.

Yes you did sent me a PM.

May I post it ?

[BB+]

Yes you did sent me a PM.
May I post it ?

As the thread is spammed anyway at this point:

Dear Ed,

If you are not being serious about your comments, please stop it, as I would consider it to be rather annoying trolling.

You perhaps do not realise how much time I have spent/wasted into tracking down various technical facts/details regarding the things you bring up, not to mention the amount of time need to draft and re-draft (and re-re-draft) posts so that they are phrased in the proper manner, etc. If you are at best only semi-serious about them, perhaps I shall consider billing you for the time spent in humoring you with an answer?

And if you merely wanted a comparison of the Fruit/Rybka evaluation ordering, just ask for it?!

On a personal note, I've been having partial insomnia (perhaps due to jetlag from recent travels, I slept from 4pm to 9pm today/yesterday) the last few days, and perhaps am not in as a good mood as I might be.

So you specifically requested me to do something, then said "I could have told you that from the beginning" when the result came. This combined with puerile histrionics about it being an "atomic bomb" of evidence for you personally (interspersed with cognitions about all top level evaluation functions being much the same), even though I explained ahead of time that structural differences meant that one could not hope to say too much either way [the most notable similarity in "evaluation ordering" is the overall top-level enumeration, and that eval_pattern proceeds in the same manner -- the most notable difference is that various things are re-arranged here and there, though as I mentioned, some of this seems possibly related to bitboard or performance issues].

As such, I conclude from your response [which I could call it rude, or at least dismissive], that indeed you weren't too serious about the request. Indeed, as it's not even clear whether I am a "computer chess programmer" by your standards in any case, I see no reason to perform any more slave-boy tasks for your passing gratification.

Billing: It took me around 3 hours to prepare/draft/re-check the above, plus all the (amortised) invested capital in knowing about Fruit/Rybka in the first place, etc. Since I enjoyed REBEL so much in the 90s, I will give you a rate of only $100/hour, resulting in a fee of $500. You might recall (from an rgcc post in 1997) that I offered [never got a response] to put up a similar amount for a Rebel/GM match, so we can call it even.

[BB+]

But you can take Rebel, most of the eval stuff is an easy read.

One concludes that REBEL is a good chess engine with a notably different (and dissimilar) evaluation function than either Rybka or Fruit. E.g., neither Fruit nor Rybka looks at hanging pieces, neither Fruit nor Rybka has a king safety module that looks much like REBEL's, neither Fruit nor Rybka has handles isolated pawns via a square-based criterion. These are the three principal specific things mentioned in Inside REBEL. It is hard to say too much about implementation as that is more dependent on the source code, as compared to a general description.

Now I realise that the PDF description ended at this juncture, while the underlying webpage continues more in-depth.

Again I will ask you ahead of time: is it worth my time to perform an extended analysis? If I do so, and then respond with an in-depth critical enumeration of REBEL versus Rybka/Fruit, will this suffice? Or can I expect there to be another swerving of what the principal issue is, such as a return to the "it's all general chess knowledge" position? I'm not being completely facetious here -- in the Panel, there was a initial debate as to whether EVAL_COMP was a worthwhile endeavor, so I produced a few examples, and there was general agreement that it was imperfect, but superior to just leaving the question as a qualitative one. Since you've already said that you don't care about EVAL_COMP, it seems unreasonably burdensome to expect me to carry this out, just to please you.

At any rate, the first few sections (Bishops/Rooks) tend to confirm my contention that REBEL is indeed "dissimilar" in contrast to the "overlaps" with Fruit and Rybka. For instance, the logical idea that the value of open files for rooks increase according to number of pawns (this appears in other programs) is not part of the Fruit/Rybka feature list. You use x-ray mobility (R3 has something similar, with pawn skeletons), while Fruit/Rybka do not. The knight/bishop/rook mobility in REBEL is computed "net" (white minus black), and then passed to a lookup table (which starts linear, then increases as the mobility differential becomes extreme). Again this is not much like what Fruit/Rybka do, where each piece type has its own mobility adjustment and the computation is strictly linear. Another point is that REBEL double-counts vertical rook mobility (again LK put something like this in R3).

For passed pawns, the "own support" is different in nature (Fruit/Rybka measure the distance), and the enemy tropism scaling is not as simple as with Fruit/Rybka. There is no "compelling" bonus, and no "heavy" immobility penalty (from a rook). With the next two sections, there is no "minimal mobility" consideration in Fruit/Rybka, and pins also are not considered. They both have no consideration of rook connections or doubled rooks (in all these cases, various other engines in EVAL_COMP do in fact have at least some overlap with what REBEL does).

In short, I think the REBEL evaluation is about as different from either Fruit or Rybka as any other of the "non-matching" pairs of engines in EVAL_COMP. As I say, if you want something more concrete, I would first prefer some indication that you will take the result seriously.

[BB+]

One might also wonder whether the comparators were not cherry picked

I have not seen any argument to substantiate this. If such an issue is to be raised, the evidential burden would seem to rest upon those who suggest it (the rationale for such a requirement is noted rather pointedly in that link). I am more than willing to discuss the issue of "cherry-picking" [which indeed is but one issue that could be raised], but as of yet I can't see anything beyond the mere voicing of the possibility. One way to overcome the aforesaid burden is to give your own listing of evaluation features, in a similar sense to that which EVAL_COMP used.

[Rebel]

Billing: It took me around 3 hours to prepare/draft/re-check the above, plus all the (amortised) invested capital in knowing about Fruit/Rybka in the first place, etc. Since I enjoyed REBEL so much in the 90s, I will give you a rate of only $100/hour, resulting in a fee of $500.

The PM is interesting for the following reasons,

1. Is basically telling me to shut up.

2. Raises questions, such as why that urgent need to reply to my (or anyone else for that matter) postings? Since when Is that an obligation on the Internet? And since you are talking to an (although ex) but still a trained commercial able to read between the lines in business terms lemme ask you this simple yes or no question:

Did the ICGA compensate you for your work ?

Perhaps with some aftermath duties ?

It would explain.

You might recall (from an rgcc post in 1997) that I offered [never got a response] to put up a similar amount for a Rebel/GM match, so we can call it even.

Sorry, must have missed it. Can't even remember I posted that

[mjlef]

Billing: It took me around 3 hours to prepare/draft/re-check the above, plus all the (amortised) invested capital in knowing about Fruit/Rybka in the first place, etc. Since I enjoyed REBEL so much in the 90s, I will give you a rate of only $100/hour, resulting in a fee of $500.

The PM is interesting for the following reasons,

1. Is basically telling me to shut up.

2. Raises questions, such as why that urgent need to reply to my (or anyone else for that matter) postings? Since when Is that an obligation on the Internet? And since you are talking to an (although ex) but still a trained commercial able to read between the lines in business terms lemme ask you this simple yes or no question:

Did the ICGA compensate you for your work ?

Perhaps with some aftermath duties ?

It would explain.

You might recall (from an rgcc post in 1997) that I offered [never got a response] to put up a similar amount for a Rebel/GM match, so we can call it even.

Sorry, must have missed it. Can't even remember I posted that

Ed,

Your anger is taking away your sense of humor.

Mark