From: Gabe Goodhart <redacted>
Date: Sun, 24 Nov 2024 09:02:34 +0000 (-0700)
Subject: convert : XLMRoberta Type Vocab Size (#10458)
X-Git-Tag: upstream/0.0.4488~331
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=9336db462c0c34bbe2055413fe4e16442626c38b;p=pkg%2Fggml%2Fsources%2Fllama.cpp

convert : XLMRoberta Type Vocab Size (#10458)

This matches the key in common bert-based embedding models and may have a
value other than 1 in it.

Branch: XLMRobertaTypeVocabSize

Signed-off-by: Gabe Goodhart <redacted>
---

diff --git a/convert_hf_to_gguf.py b/convert_hf_to_gguf.py
index 9f4b8154..80a179b8 100755
--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -2707,7 +2707,7 @@ class XLMRobertaModel(BertModel):
         self.gguf_writer.add_token_scores(scores)
         self.gguf_writer.add_token_types(toktypes)
         self.gguf_writer.add_add_space_prefix(add_prefix)
-        self.gguf_writer.add_token_type_count(1)
+        self.gguf_writer.add_token_type_count(self.hparams.get("type_vocab_size", 1))
         self.gguf_writer.add_remove_extra_whitespaces(remove_whitespaces)
         if precompiled_charsmap:
             self.gguf_writer.add_precompiled_charsmap(precompiled_charsmap)