Some tablebase formats

[kingliveson]

I found that Pepito 1.59 (from 2002) manages to both be licensed under the Gnu GPL and use the Nalimov tablebase code. I have no idea if/how this issuance of the Nalimov code is done in a compatible manner with this license. Even if Nalimov OK's your usage of his (and Kadatch's code), it's not clear to me how you could convey it in conjunction with code that is licensed under the Gnu GPL, as this license is notably "viral" (that is, all code that links in to it must use a compatible license). All the thanx.txt says is "Thanks to Eugene Nalimov for making publicly available your code for probing the endgame tablebases", which if true to nth degree would relief a great hassle for many developers. At the very least, if this distribution was done in a proper manner, it would imply that the Nalimov probing code can be freely used in open source engines/GUIs whose licenses are compatible with the Gnu GPL.

Personally do not understand why getting permission to use Nalimov probing code is made to be so difficult. He does not ask for any compensation and yet...but the good news is there are now alternatives which are even better if you asked me.

[ernest]

but the good news is there are now alternatives which are even better if you asked me.

Why better?
Technically Gaviota tablebases are equivalent to Nalimov, not better.

[Dave Mitchell]

According to Miguel, they compress better, (smaller), and can be used more flexibly with soft or hard TB searches.

[BB+]

According to Miguel, they compress better, (smaller), and can be used more flexibly with soft or hard TB searches.

I think the API is also a bit easier to use, though I really don't know. I should have mentioned that my critique was aimed at the boffins who build such TBs, while for programmers a good API (like with Gaviota) is probably a much greater desideratum than most other technical considerations.

[BB+]

I have no idea if/how this issuance of the Nalimov code is done in a compatible manner with this license.

In retrospect, I realise this is quite obvious via the "dual-license" dodge. For instance, Pepito with Nalimov TBs is released under License X, compatible with Nalimov. The same Pepito code but w/o Nalimov hooks is also released under the Gnu GPL.

[BB+]

I'm still not quite sure what Shredder is doing for the extra compression in the 157MB version. One guess is simply a Huffman layer on top of what I presume is run-length encoding (in the 441MB version), though I can't quite get that much space-reduction in my own experiments. I don't know if the slowdown from such uncompression would be ~10x as with Shredder either.

Another possibility is the use of "predictor functions". For instance, rather than save WLD info and compress it, save the diff of the WLD from the result of a predictor function. This have more redundancy, and thus compress more readily. See a Rybka forum discussion with ZW and Alan Sassler here, which links to discussions at Winboard Forum and TalkChess. In the last, Shawul implied that "guessing results" via predictor functions had little gain for 5 pieces (~10% at most), and thought it diminished with more pieces, and almost disappeared when using "1-sided" bitbases.

More fancy ways include decision-tree methods as used in Knightdreamer (see the PDF thesis linked to in the first link above), but even there the size was not reduced below 200MB.

[ernest]

I'm still not quite sure what Shredder is doing for the extra compression in the 157MB version. One guess is simply a Huffman layer on top of what I presume is run-length encoding (in the 441MB version)

Well, as you know, for the Scorpio bitbases, the 1st version was 361 MB, then Dann Corbit reduced that to 229 MB

[BB+]

Well, as you know, for the Scorpio bitbases, the 1st version was 361 MB, then Dann Corbit reduced that to 229 MB

No I didn't know this. The 361MB (or 330MB I thought) was already doing the "one-sided" trick to make it half of the 970MB (see the Winboard Forum discussion linked above). Did DC do this in a way usable in real time? I mean, I can apply bzip2 to Scorpio bases (or RobboTriple bases) and get this sort of compression ratio, but can the uncompression be done sufficiently fast to make it workable? I seem to recall a 7-zip version of Scorpio bases (for download) were teetering around 200MB. For that matter, what happens if you apply your favourite compression algorithm to Shredderbases? Do they reduce much?

EDIT: looks like no? http://talkchess.com/forum/viewtopic.ph ... 25&t=26936
So it's just the RAR that's smaller, for download.

[ernest]

EDIT: looks like no? http://talkchess.com/forum/viewtopic.ph ... 25&t=26936
So it's just the RAR that's smaller, for download.

Your EDIT preempted my answer I was going to give you the same link
Those Scorpio bitbases are made for disk (don't know if they could be used in memory...)
But their size (on my disk, that is "unrared"), is 223MB

[BB+]

Oh, you are right -- Scorpio bases shouldn't compress much via rar, as they are already essentially in Huffmann form. This is unlike Shredderbases or TripleBases, as those seem to be run-length encoded (and so can be read w/o further expansion, unlike Scorpio which expands compressed 8K blocks on demand). It would still be nice to know how much Shredderbases shrink (if at all) when a compressor is applied.