]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 0c58ba3)
vocab: fix Gemma4 tokenizer (#21343)
authorPiotr Wilkin (ilintar) <redacted>
Fri, 3 Apr 2026 08:33:03 +0000 (10:33 +0200)
committerGitHub <redacted>
Fri, 3 Apr 2026 08:33:03 +0000 (10:33 +0200)
commitb069b10ab48f25ba119e59d0b8bf35d4f06e093f
treefa0232c22e116109316ec41932b9be2bb099c2bdtree
parent0c58ba3365d2bc717b447b5d70e4d6be09ff3c40commit | diff
vocab: fix Gemma4 tokenizer (#21343)

* seems to work

* fix case with new line

Co-authored-by: sayap <redacted>
* gemma 4: fix pre tok regex

---------

Co-authored-by: Xuan Son Nguyen <redacted>
Co-authored-by: sayap <redacted>
convert_hf_to_gguf.py diff | blob | history
src/llama-vocab.cpp diff | blob | history
src/llama-vocab.h diff | blob | history
src/unicode.cpp diff | blob | history
src/unicode.h diff | blob | history
Packaging of ggml-org/llama.cpp