]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 69b9945)
llama : add support for Tekken pre-tokenizer (#8579)
authorMichael Coppola <redacted>
Sat, 20 Jul 2024 13:43:51 +0000 (09:43 -0400)
committerGitHub <redacted>
Sat, 20 Jul 2024 13:43:51 +0000 (16:43 +0300)
commit940362224d20e35f13aa5fd34a0d937ae57bdf7d
tree309b398e827cd20b3c0708ec1d0ea3155794d104tree
parent69b9945b44c3057ec17cb556994cd36060455d44commit | diff
llama : add support for Tekken pre-tokenizer (#8579)

* llama : Added support for Tekken pre-tokenizer (#8577)

Removed uneeded `vocab.tokenizer_clean_spaces` assignment

* llama : fix order of pre-tokenizers

* * Tekken pre-tokenizer no longer uses clean_up_tokenization_spaces
* Updated chkhsh for Tekken tokenizer

---------

Co-authored-by: Georgi Gerganov <redacted>
convert_hf_to_gguf.py diff | blob | history
convert_hf_to_gguf_update.py diff | blob | history
include/llama.h diff | blob | history
src/llama.cpp diff | blob | history
Packaging of ggml-org/llama.cpp