Experience version 3 : investigation

[deeds]

anyway forget it, my question was simple but it is clear that you do not want to answer, thanks all the same

[Amos 4ever]

You're really bad as an investigator if you haven't yet realized you're working with an outdated learning system
and that Eman Sugar and Hypnos have been using better code for a long time now.

Many efforts in training an engine with a code that learns much less quickly than the one we use now

You should have figured it out with the expex command that the codes are different.

[deeds]

The immense loneliness of the stalker who banned so many people on his forum that he tries to talk to them again on other forums.


On a subject where we talk about filesize reduction, new experience file formats, we still find people who missed several wagons but believe they train faster than others


The funny thing is when they ask to forget their questions but ask them again on PMs


Fortunately at OCF, there is no more public discussion between people, no more training rooms, no more engine trainers who dare to publish their results, no more ranking of opening books, only moderators playing in the ban lottery

[deeds]

v3.1 : let's see if we can break the 33% filesize reduction of v3.0 without losing too much information...

v3.1 reached -38% with an EXP file containing reinforced experience data :


The v2 experience files need to be defragmented before converting :

Else same "total moves" but not same "total positions".

[deeds]

[deeds]

Without checking fragmented experience data, the conversion rate is about 540 pos/ms.
With checking fragmented experience data, the conversion rate is about 1 pos/ms..
So i removed this feature from the expTov31 tool.

[deeds]

After converting the v2.0 experience data to v3.0, the entries are not ordered !?

Yes it's normal. With v2.0 and earlier, the positions/moves were stored in reverse order so the file's ordering was different.

Here is an example to reorder v3.0 experience data :

Code: Select all

Private Sub orderV30(path As String)
	Dim myString As String, tabString() As String, i As Integer, start As Long
	Dim myStreamReader As IO.FileStream
	Dim tabEntryV30() As Byte, index As Long
	Dim indexWrites As Long, tabWrites(149999) As Byte

	myStreamReader = New IO.FileStream(path, IO.FileMode.Open)

	'Experience version 3.0
	'0123456789abcdef012345
	ReDim tabEntryV30(21)
	myStreamReader.Read(tabEntryV30, 0, 22)

	If System.Text.Encoding.UTF8.GetString(tabEntryV30) <> "Experience version 3.0" Then
		MsgBox(nomFichier(path) & " <> experience format v3.0 !?", MsgBoxStyle.Exclamation)
		End
	End If

	index = (myStreamReader.Length - 22) / 16
	ReDim tabString(index - 1)
	ReDim tabWrites((index * 16) - 1)

	Console.Write("Loading... ")
	start = GetTickCount
	myString = ""
	index = 0
	ReDim tabEntryV30(15)
	While myStreamReader.Position < myStreamReader.Length
		'1 entry at the time
		myStreamReader.Read(tabEntryV30, 0, 16)

		'16 bytes to string
		tabString(index) = ""
		For i = 0 To 15
			tabString(index) = tabString(index) & hexa(tabEntryV30(i))
		Next

		index = index + 1
		If (myStreamReader.Position - 22) Mod 16 * 2000000 = 0 Then
			Console.Write(Format(myStreamReader.Position / myStreamReader.Length, "0%") & " ")
		End If
	End While
	myStreamReader.Close()
	Console.WriteLine("OK (" & Trim(Format(GetTickCount - start, "### ### ##0 ms")) & ")")

	Console.Write("Sorting... ")
	start = GetTickCount
	Array.Sort(tabString)
	Console.WriteLine("OK (" & Trim(Format(GetTickCount - start, "### ### ##0 ms")) & ")")

	'hexa string to bytes
	Console.Write("Writing... ")
	start = GetTickCount
	indexWrites = 0
	For index = 0 To UBound(tabString)
		For i = 0 To 30 Step 2
			tabWrites(indexWrites) = Convert.ToByte(tabString(index).Substring(i, 2), 16)
			indexWrites = indexWrites + 1
		Next
	Next
	My.Computer.FileSystem.WriteAllText(path, "Experience version 3.0", False, New System.Text.UTF8Encoding(False))
	My.Computer.FileSystem.WriteAllBytes(path, tabWrites, True)
	Console.WriteLine("OK (" & Trim(Format(GetTickCount - start, "### ### ##0 ms")) & ")")

End Sub

[deeds]

My engine loads v3.0 EXP files but the built-in "exp / expex" commands don't show the experience data !?

Perhaps your engine does not yet fully support the v3.0 experience format.

Here is an example for an output equivalent to the "expex" command :

Code: Select all

Private Sub showExpDataV30(path As String, position As String)
	Dim key As String, move As String, scoreCP As Integer, scoreMAT As Integer, depth As Integer, count As Integer
	Dim myStreamReader As IO.FileStream, positionMax As Integer
	Dim tabEntryV30() As Byte, indexEntry As Integer, start As Long
	Dim tabIndex(255) As Integer
	Dim i As Integer, myString As String, tabString() As String, tabTmp() As String
	Dim prevMove As String, prevDepth As Integer, nbMoves As Integer

	myStreamReader = New IO.FileStream(path, IO.FileMode.Open)

	'Experience version 3.0
	'0123456789abcdef012345
	ReDim tabEntryV30(21)
	myStreamReader.Read(tabEntryV30, 0, 22)
	If System.Text.Encoding.UTF8.GetString(tabEntryV30) <> "Experience version 3.0" Then
		MsgBox(nomFichier(path) & " <> experience format v3.0 !?", MsgBoxStyle.Exclamation)
		End
	End If

	'indexation (first byte of the position's key) when loading v3.0 EXP file
	Console.Write("Indexing... ")
	start = GetTickCount
	indexEntry = 0
	ReDim tabEntryV30(15)
	While myStreamReader.Position < myStreamReader.Length
		myStreamReader.Read(tabEntryV30, 0, 16)
		indexEntry = indexEntry + 1

		If tabIndex(tabEntryV30(0)) = 0 Then
			tabIndex(tabEntryV30(0)) = indexEntry
		End If
	End While
	myStreamReader.Close()
	Console.WriteLine("OK (" & Trim(Format(GetTickCount - start, "### ### ##0 ms")) & ")")
	Console.WriteLine()

	ReDim tabEntryV30(15)
	myStreamReader = New IO.FileStream(path, IO.FileMode.Open)

	indexEntry = Convert.ToByte(gauche(position, 2), 16)
	myStreamReader.Position = 22 + (tabIndex(indexEntry) - 1) * 16

	positionMax = myStreamReader.Length
	Do
		indexEntry = indexEntry + 1
		If tabIndex(indexEntry) <> 0 Then
			positionMax = 22 + (tabIndex(indexEntry) - 1) * 16
			Exit Do
		End If
	Loop While indexEntry < 255 And tabIndex(indexEntry) = 0

	myString = ""
	key = ""
	Do
		myStreamReader.Read(tabEntryV30, 0, 16)
		If tabEntryV30(0) = Convert.ToByte(position.Substring(0, 2), 16) Then
			If tabEntryV30(1) = Convert.ToByte(position.Substring(2, 2), 16) Then
				If tabEntryV30(2) = Convert.ToByte(position.Substring(4, 2), 16) Then
					If tabEntryV30(3) = Convert.ToByte(position.Substring(6, 2), 16) Then
						If tabEntryV30(4) = Convert.ToByte(position.Substring(8, 2), 16) Then
							If tabEntryV30(5) = Convert.ToByte(position.Substring(10, 2), 16) Then
								If tabEntryV30(6) = Convert.ToByte(position.Substring(12, 2), 16) Then
									If tabEntryV30(7) = Convert.ToByte(position.Substring(14, 2), 16) Then

										'16 bytes => string
										If key = "" Then
											For i = 0 To 7
												key = key & hexa(tabEntryV30(i))
											Next
										End If

										'031c => 001 100 011 100 => e2e4
										move = binToMove(hexadecimalToBinaire(hexa(tabEntryV30(9)) & hexa(tabEntryV30(10))))

										scoreCP = Convert.ToInt64(hexa(tabEntryV30(11)) & hexa(tabEntryV30(12)) & hexa(tabEntryV30(13)), 16)
										scoreMAT = 0
										If 162144 <= scoreCP And scoreCP < 262144 Then
											'negative score
											scoreCP = scoreCP - 262144
										ElseIf 139900 <= scoreCP And scoreCP < 140000 Then
											'negative mat
											scoreMAT = scoreCP - 140000
											scoreCP = 0
										ElseIf 120000 < scoreCP And scoreCP <= 121000 Then
											'positive mat
											scoreMAT = scoreCP - 120000
											scoreCP = 0
										ElseIf 0 <= scoreCP And scoreCP <= 100000 Then
											'draw and positive scores
											scoreCP = scoreCP
										End If

										'depth
										depth = tabEntryV30(14)

										'count
										count = tabEntryV30(15)

										If scoreMAT = 0 Then
											myString = myString & move & ":" & Format(depth, "000") & ":cp " & scoreCP & ":" & count & vbCrLf
										Else
											myString = myString & move & ":" & Format(depth, "000") & ":mat " & scoreMAT & ":" & count & vbCrLf
										End If
									End If
								End If
							End If
						End If
					End If
				End If
			End If
		End If
	Loop While myStreamReader.Position < positionMax
	myStreamReader.Close()

	'keep greater depth for duplicated moves
	tabString = Split(trierChaine(myString, vbCrLf, False), vbCrLf)
	myString = ""
	prevMove = ""
	prevDepth = 0
	scoreCP = 0
	For i = 0 To UBound(tabString)
		If tabString(i) <> "" Then
			tabTmp = Split(tabString(i), ":")
			depth = Val(tabTmp(1))
			move = tabTmp(0)
			If InStr(tabTmp(2), "cp", CompareMethod.Text) > 0 Then
				scoreCP = Val(Replace(tabTmp(2), "cp ", ""))
			Else
				scoreCP = Val(Replace(tabTmp(2), "mat ", "")) * 1000
			End If
			If Not (move = prevMove And depth < prevDepth) Then
				myString = myString & Format(scoreCP, "000000000") & ":" & move & ":" & depth & ":" & tabTmp(2) & ":" & tabTmp(3) & vbCrLf
				prevMove = move
				prevDepth = depth
			End If
		End If
	Next

	'order by score
	tabString = Split(trierChaine(myString, vbCrLf, False), vbCrLf)
	myString = "key: " & key & vbCrLf
	nbMoves = 0
	For i = 0 To UBound(tabString)
		If tabString(i) <> "" Then
			tabTmp = Split(tabString(i), ":")
			nbMoves = nbMoves + 1
			myString = myString & nbMoves & " : " & tabTmp(1) & " , depth: " & tabTmp(2) & " , eval: " & tabTmp(3) & " , count: " & tabTmp(4) & vbCrLf
		End If
	Next

	Console.WriteLine(myString)
End Sub

Then your engine will have to calculate the quality.