From: Jiří Podivín <redacted>
Date: Fri, 12 Jul 2024 08:06:33 +0000 (+0200)
Subject: convert : remove fsep token from GPTRefactForCausalLM (#8237)
X-Git-Tag: upstream/0.0.4488~1108
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=5aefbce27a66473d4b1263ba3f7bdd3d14245975;p=pkg%2Fggml%2Fsources%2Fllama.cpp

convert : remove fsep token from GPTRefactForCausalLM (#8237)

The <filename> token used by Refact doesn't serve
the same purpose as the <file_separator> from CodeGemma.

Signed-off-by: Jiri Podivin <redacted>
---

diff --git a/convert_hf_to_gguf.py b/convert_hf_to_gguf.py
index ebb5ca37..cf930be1 100755
--- a/convert_hf_to_gguf.py
+++ b/convert_hf_to_gguf.py
@@ -1203,11 +1203,10 @@ class RefactModel(Model):
 
         # TODO: how to determine special FIM tokens automatically?
         special_vocab = gguf.SpecialVocab(self.dir_model, load_merges=False,
-                                          special_token_types = ['prefix', 'suffix', 'middle', 'fsep', 'eot'])
+                                          special_token_types = ['prefix', 'suffix', 'middle', 'eot'])
         special_vocab._set_special_token("prefix", 1)
         special_vocab._set_special_token("suffix", 3)
         special_vocab._set_special_token("middle", 2)
-        special_vocab._set_special_token("fsep",   4) # is this correct?
         special_vocab.add_to_gguf(self.gguf_writer)
 
     def set_gguf_parameters(self):