]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
tests : multi-thread the tokenizer tests (#5474)
authorGeorgi Gerganov <redacted>
Tue, 13 Feb 2024 13:14:22 +0000 (15:14 +0200)
committerGitHub <redacted>
Tue, 13 Feb 2024 13:14:22 +0000 (15:14 +0200)
commitcf45252a7cfcb998bade46a886e20477cecc538a
tree89400db2102fb39809b61af13a7534e49ec5c27e
parent03bf161eb6dea6400ee49c6dc6b69bdcfa9fd3fc
tests : multi-thread the tokenizer tests (#5474)

* tests : multi-thread the tokenizer tests

ggml-ci

* unicode : fix data race for unidentified codepoints

ggml-ci

* unicode : minor style fixes

ggml-ci
llama.cpp
tests/test-tokenizer-1-bpe.cpp
tests/test-tokenizer-1-llama.cpp
unicode.h