From: Kerfuffle <redacted>
Date: Tue, 22 Aug 2023 00:01:34 +0000 (-0600)
Subject: Fix convert-llama-ggmlv3-to-gguf.py vocab conversion (#2698)
X-Git-Tag: gguf-v0.4.0~292
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=930523c8e1cbbee5449c055daa894917fac6805e;p=pkg%2Fggml%2Fsources%2Fllama.cpp

Fix convert-llama-ggmlv3-to-gguf.py vocab conversion (#2698)

When converting without metadata, the hex value for bytes entries weren't 0 padded to 2 digits.
---

diff --git a/convert-llama-ggmlv3-to-gguf.py b/convert-llama-ggmlv3-to-gguf.py
index 30038072..fa4a044c 100644
--- a/convert-llama-ggmlv3-to-gguf.py
+++ b/convert-llama-ggmlv3-to-gguf.py
@@ -236,8 +236,7 @@ class GGMLToGGUF:
             if len(vbytes) == 0:
                 tt = 3 # Control
             elif tokid >= 3 and tokid <= 258 and len(vbytes) == 1:
-                hv = hex(vbytes[0])[2:].upper()
-                vbytes = bytes(f'<0x{hv}>', encoding = 'UTF-8')
+                vbytes = bytes(f'<0x{vbytes[0]:02X}>', encoding = 'UTF-8')
                 tt = 6 # Byte
             else:
                 vbytes = vbytes.replace(b' ', b'\xe2\x96\x81')