]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 1fd9c17)
llama : add BPE pre-tokenization for Qwen2 (#7114)
authorRen Xuancheng <redacted>
Wed, 8 May 2024 12:06:43 +0000 (20:06 +0800)
committerGitHub <redacted>
Wed, 8 May 2024 12:06:43 +0000 (15:06 +0300)
commit229ffff872f8ad0d21c997d18ee7a23692ae60a0
treeb4c2f055395cc0ccbc2b0ade78da27d429cdb2e1tree
parent1fd9c1741d864d01cd7ec6d67227b92d7bfabf22commit | diff
llama : add BPE pre-tokenization for Qwen2 (#7114)

* Add BPE pre-tokenization for Qwen2.

* minor : fixes

---------

Co-authored-by: Ren Xuancheng <redacted>
Co-authored-by: Georgi Gerganov <redacted>
convert-hf-to-gguf-update.py diff | blob | history
convert-hf-to-gguf.py diff | blob | history
llama.cpp diff | blob | history
llama.h diff | blob | history
models/ggml-vocab-qwen2.gguf [new file with mode: 0644] blob
models/ggml-vocab-qwen2.gguf.inp [new file with mode: 0644] blob
models/ggml-vocab-qwen2.gguf.out [new file with mode: 0644] blob
tests/CMakeLists.txt diff | blob | history
Packaging of ggml-org/llama.cpp