]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 6fbd432)
command-r : add BPE pre-tokenization (#7063)
authorDAN™ <redacted>
Sun, 5 May 2024 05:19:30 +0000 (01:19 -0400)
committerGitHub <redacted>
Sun, 5 May 2024 05:19:30 +0000 (08:19 +0300)
commit889bdd76866ea31a7625ec2dcea63ff469f3e981
treebcfaea83b66f89181980b64dba57a8c8f8f31a49tree
parent6fbd43221167bf96112f899daf22c127b282cbcfcommit | diff
command-r : add BPE pre-tokenization (#7063)

* Add BPE pre-tokenization for Command-R/R+.

* Bump transformers convert requirement.

* command-r : add individual digits regex

---------

Co-authored-by: Georgi Gerganov <redacted>
convert-hf-to-gguf-update.py diff | blob | history
convert-hf-to-gguf.py diff | blob | history
llama.cpp diff | blob | history
llama.h diff | blob | history
models/ggml-vocab-command-r.gguf [new file with mode: 0644] blob
models/ggml-vocab-command-r.gguf.inp [new file with mode: 0644] blob
models/ggml-vocab-command-r.gguf.out [new file with mode: 0644] blob
requirements/requirements-convert.txt diff | blob | history
tests/CMakeLists.txt diff | blob | history
Packaging of ggml-org/llama.cpp