From: o7si <redacted>
Date: Thu, 1 Jan 2026 17:27:07 +0000 (+0800)
Subject: convert : fix encoding of WPM vocab for BERT models (#18500)
X-Git-Tag: upstream/0.0.7721~117
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=2b2afade9fb38b8d699ed561d20a259561c00fc3;p=pkg%2Fggml%2Fsources%2Fllama.cpp

convert : fix encoding of WPM vocab for BERT models (#18500)

* convert: avoid token collision when stripping ## prefix

* convert: use token types for BERT special tokens check

* Update convert_hf_to_gguf.py

Co-authored-by: Sigbjørn Skjæret <redacted>

---------

Co-authored-by: Sigbjørn Skjæret <redacted>
---

diff --git a/convert_hf_to_gguf.py b/convert_hf_to_gguf.py
index a1080b15..2c961b8f 100755
--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -5287,13 +5287,14 @@ class BertModel(TextModel):
         self.gguf_writer.add_token_type_count(self.hparams.get("type_vocab_size", 1))
 
         # convert to phantom space vocab
-        def phantom(tok):
-            if tok.startswith("[") and tok.endswith("]"):
+        def phantom(tok, toktype):
+            if toktype == gguf.TokenType.CONTROL:
                 return tok
             if tok.startswith("##"):
                 return tok[2:]
             return "\u2581" + tok
-        tokens = list(map(phantom, tokens))
+        assert len(tokens) == len(toktypes)
+        tokens = list(map(phantom, tokens, toktypes))
 
         # add vocab to gguf
         self.gguf_writer.add_tokenizer_model("bert")