]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 469c9ad)
Add more tokenizer tests (#3742)
authorGalunid <redacted>
Tue, 24 Oct 2023 07:17:17 +0000 (09:17 +0200)
committerGitHub <redacted>
Tue, 24 Oct 2023 07:17:17 +0000 (09:17 +0200)
commitdaab3d7f45832e10773c99f3484b0d5b14d86c0c
tree432092c5aec7c775ab6e33b968564bd0a1e4a187tree
parent469c9addef75893e6be12edda852d12e840bf064commit | diff
Add more tokenizer tests (#3742)

* Add more tokenizer tests

* Add starcoder

* Update test vocab files

* Restrict bpe tokenizer tests to unicode planes

* Update comment

* Comment cosmetics

* Remove bloom vocab/test
models/ggml-vocab-baichuan.gguf [new file with mode: 0644] blob
models/ggml-vocab-gpt-neox.gguf [new file with mode: 0644] blob
models/ggml-vocab-refact.gguf [new file with mode: 0644] blob
models/ggml-vocab-starcoder.gguf [new file with mode: 0644] blob
tests/CMakeLists.txt diff | blob | history
tests/test-tokenizer-1-bpe.cpp diff | blob | history
Packaging of ggml-org/llama.cpp