]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: eaa13a4)
llama : speedup tokenization (#2831)
authorKawrakow <redacted>
Sun, 27 Aug 2023 13:50:33 +0000 (16:50 +0300)
committerGitHub <redacted>
Sun, 27 Aug 2023 13:50:33 +0000 (16:50 +0300)
commit463173a6c0ff353055eb90665794884c888c790f
tree4868e5ed0a6924410c91b149a6a630ea75ea06detree
parenteaa13a48ff4136f01c1cdb79cacd61b67ec53095commit | diff
llama : speedup tokenization (#2831)

* Speedup tokenization

On current master it takes ~3.2 seconds to tokenize
Wikitext. With this change it becomes ~525 ms.

* Fixit: it was missing the piece after the last found occurence

---------

Co-authored-by: Iwan Kawrakow <redacted>
examples/perplexity/perplexity.cpp diff | blob | history
llama.cpp diff | blob | history
Packaging of ggml-org/llama.cpp