]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)
authorXuan-Son Nguyen <redacted>
Fri, 30 May 2025 10:24:37 +0000 (12:24 +0200)
committerGitHub <redacted>
Fri, 30 May 2025 10:24:37 +0000 (12:24 +0200)
commit07e4351ce663a7802b31f92fd7bc0e555c2044b6
tree5469dfd805023e3dd5e01f5138347c4122ac3078
parent291f2b6913c7ef8350dbf0e77da38f7af131a08e
convert : allow partial update to the chkhsh pre-tokenizer list (#13847)

* convert : allow partial update to the chkhsh pre-tokenizer list

* code style

* update tokenizer out

* rm inp/out files for models not having gguf

* fixed hash for glm

* skip nomic-bert-moe test

* Update convert_hf_to_gguf_update.py

* fix minerva-7b hash

* rm redundant import
43 files changed:
convert_hf_to_gguf.py
convert_hf_to_gguf_update.py
models/ggml-vocab-bert-bge.gguf.inp
models/ggml-vocab-bert-bge.gguf.out
models/ggml-vocab-chameleon.gguf.inp [deleted file]
models/ggml-vocab-chameleon.gguf.out [deleted file]
models/ggml-vocab-command-r.gguf.inp
models/ggml-vocab-command-r.gguf.out
models/ggml-vocab-deepseek-coder.gguf.inp
models/ggml-vocab-deepseek-coder.gguf.out
models/ggml-vocab-deepseek-llm.gguf.inp
models/ggml-vocab-deepseek-llm.gguf.out
models/ggml-vocab-deepseek-r1-qwen.gguf.inp [deleted file]
models/ggml-vocab-deepseek-r1-qwen.gguf.out [deleted file]
models/ggml-vocab-falcon.gguf.inp
models/ggml-vocab-falcon.gguf.out
models/ggml-vocab-gpt-2.gguf.inp
models/ggml-vocab-gpt-2.gguf.out
models/ggml-vocab-gpt-4o.gguf.inp [deleted file]
models/ggml-vocab-gpt-4o.gguf.out [deleted file]
models/ggml-vocab-llama-bpe.gguf.inp
models/ggml-vocab-llama-bpe.gguf.out
models/ggml-vocab-llama-spm.gguf.inp
models/ggml-vocab-llama-spm.gguf.out
models/ggml-vocab-llama4.gguf.inp [deleted file]
models/ggml-vocab-llama4.gguf.out [deleted file]
models/ggml-vocab-mpt.gguf.inp
models/ggml-vocab-mpt.gguf.out
models/ggml-vocab-nomic-bert-moe.gguf.inp [deleted file]
models/ggml-vocab-nomic-bert-moe.gguf.out [deleted file]
models/ggml-vocab-phi-3.gguf.inp
models/ggml-vocab-phi-3.gguf.out
models/ggml-vocab-pixtral.gguf.inp [deleted file]
models/ggml-vocab-pixtral.gguf.out [deleted file]
models/ggml-vocab-qwen2.gguf.inp
models/ggml-vocab-qwen2.gguf.out
models/ggml-vocab-refact.gguf.inp
models/ggml-vocab-refact.gguf.out
models/ggml-vocab-roberta-bpe.gguf.inp [deleted file]
models/ggml-vocab-roberta-bpe.gguf.out [deleted file]
models/ggml-vocab-starcoder.gguf.inp
models/ggml-vocab-starcoder.gguf.out
tests/CMakeLists.txt