]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
convert : correctly handle LLaMA tokenizer for Jamba (#16470)
authoramirai21 <redacted>
Sat, 11 Oct 2025 08:33:41 +0000 (11:33 +0300)
committerGitHub <redacted>
Sat, 11 Oct 2025 08:33:41 +0000 (10:33 +0200)
commit477a66b03501cf3bd067f8968b77ca4d053ff1bd
treecd1a365462d43529a27c7bfa0248ed18d1418285
parente60f01d941bc5b7fae62dd57fee4cec76ec0ea6e
convert : correctly handle LLaMA tokenizer for Jamba (#16470)

* fix: convert_hf_to_gguf - change Jamba non-sentencepiece mode (tokenizer.json) vocab construction

* fix: convert_hf_to_gguf - jamba non-sentencepiece tokenizer to use _set_vocab_llama_hf func

* fix: convert_hf_to_gguf - removed get_vocab_base_pre from jamba
convert_hf_to_gguf.py