Experience version 3 : investigation

[Amos 4ever]

Clarification please, suppose I have a v2 exp file after converting it to v3 and still using the file playing games, will I have new data stored with v2 again so can I safely reuse the tool to update it again to v3?

Huh !? When Eman 7.00 first loads a v1 experience file, it converts it to a v2 experience file and updates it with new v2 data.

Be serious, once a chess engine or chess system adopts a file format for its experience data, there's no point in jumping back and forth between versions.

Given all the investigations presented above and the possible gains, do you really think I'm going to stop at the version 3.0 which saves only 33% ? loool

I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question

[deeds]

I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question

Does SugaR read v3 experience file format !?

[Amos 4ever]

I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question

Does SugaR read v3 experience file format !?

I suppose so.. I think it makes sense that he should read it
but i haven't tried
anyway forget it, my question was simple but it is clear that you do not want to answer, thanks all the same

[IbaiBuR]

Clarification please, suppose I have a v2 exp file after converting it to v3 and still using the file playing games, will I have new data stored with v2 again so can I safely reuse the tool to update it again to v3?

Huh !? When Eman 7.00 first loads a v1 experience file, it converts it to a v2 experience file and updates it with new v2 data.

Be serious, once a chess engine or chess system adopts a file format for its experience data, there's no point in jumping back and forth between versions.

Given all the investigations presented above and the possible gains, do you really think I'm going to stop at the version 3.0 which saves only 33% ? loool

I'm serious, maybe I didn't explain myself well
I try to simplify like this
if I use with the SugaR engine an esp file converted to version 3
what happens with the new data it stores?
are they stored in the new v3 format?
It seems fair to me to ask this question

I mean... If SugaR doesnt read v3 format (because current format as you may now is 8 bytes for the zobrist hash key, 4 bytes for the rest) as it is different from v2 (SugaR's default) and if you dont change the functions that write in the experience file, it is clear that SugaR will not store the new data in the v3 format, it will conitnue storing it in its default one... You have to change the internal code (experience.h and experience.cpp) for SugaR to handle v3 format

Some changes are quite easy, for example, you could change depth to uint8_t in the experience.h (you only need 1 byte for depth as explained by DeeDs since depth is capped to 245 (with 1 byte you can reach up to 255->2⁸ -1)and used in v3 format)

Hope helps,

Ibai

[IbaiBuR]

For sure, there are a lot more changes to be made, but you should start by changing the types of the variables that create the ExpEntry struct....

Just saying

[deeds]

Does SugaR read v3 experience file format !?

I suppose so.. I think it makes sense that he should read it
but i haven't tried

I doubt !

In v3.0, the signature has changed, the 8 bytes of the position key are in normal order, there are only 2 bytes for moves in normal order, only 3 bytes for scores in normal order, scores use a new value's range (from 0 to 262 144), etc.

[deeds]

v3.0 specs :



position key (8 bytes) : zobrish key hashed by engine, bytes stored in normal order
nb moves (1 byte) : reserved for next versions
move (2 bytes) : same as v2, bytes stored in normal order
score (3 bytes) : new range, original values stored in normal order
depth (1 byte) : same as v2
count (1 byte) : same as v2

[deeds]

v3.1 : let's see if we can break the 33% filesize reduction of v3.0 without losing too much information...


After conversion from v2 to v3.1, the experience files looks like this :




And now, let's check if the conversion is lossless :





The conversion from v2 scores to v3.1 scores isn't lossless because from one format to another, we multiply/divide integers by 2.08, of course...
If the error margin stays within a range of +/- 0.01 (1 cp), it's viable !

[deeds]

Some stats about experience files converted to v3.1 :










...

[deeds]

...




Ok so with v3.1 the filesize gain is quite variable but still better than with v3.0.

As the v3.0 already required a lot of modifications to the source code, i think people can be satisfied with its average filesize gain of 33%.
As i already shared a tool to convert experience file from v2 to v3.0, no need to pollute github more.

From my side, i plan to work on v3.2 maybe with an engine-independent position key or an another use of the count values... But i think newer version than 3.1 will be exclusive to D.C.S.