]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
10X faster BPE tokenizer (#2876)
authorKawrakow <redacted>
Tue, 29 Aug 2023 20:55:03 +0000 (23:55 +0300)
committerGitHub <redacted>
Tue, 29 Aug 2023 20:55:03 +0000 (23:55 +0300)
commite37e69dcc3d52f21222a63cafed2a71b3f6b53c6
tree887b453350fd897b3d1114a3924a2f50ecbb8da8
parent53885d7256909ec3e2176cdc2477f3986c15ec69
10X faster BPE tokenizer (#2876)

* 10X faster BPE tokenizer

* Remove comment that no longer applies

---------

Co-authored-by: Iwan Kawrakow <redacted>
llama.cpp