]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 7085492)
convert: add MiniCPM5 tokenizer support (#23384)
authorzhangtao2-1 <redacted>
Wed, 27 May 2026 05:08:33 +0000 (13:08 +0800)
committerGitHub <redacted>
Wed, 27 May 2026 05:08:33 +0000 (08:08 +0300)
commit9777256c3130fa3201327bfab44bae187f7caea2
tree818ef397fc108aa8b9d77f898a1e739fdae9403dtree
parent7085492c6f792e7499d790b3007cb08fd58b5953commit | diff
convert: add MiniCPM5 tokenizer support (#23384)

Add minicpm5 pre-tokenizer hash via convert_hf_to_gguf_update.py and
implement hardcoded regex handling in llama-vocab.cpp, consistent with
other BPE pre-tokenizers.

Co-authored-by: zhangtao <redacted>
conversion/base.py diff | blob | history
convert_hf_to_gguf_update.py diff | blob | history
src/llama-vocab.cpp diff | blob | history
src/llama-vocab.h diff | blob | history
Packaging of ggml-org/llama.cpp