]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 4f54609)
Implement non-greedy tokenizer that tries to maximize token lengths (#242)
authorthement <redacted>
Fri, 17 Mar 2023 20:05:58 +0000 (21:05 +0100)
committerGitHub <redacted>
Fri, 17 Mar 2023 20:05:58 +0000 (21:05 +0100)
commitc9f670a17755311aa28c411f5c7f3c8c05434770
treea942b84194bc4436df9d38eb3b06175e0e849166tree
parent4f546091102a418ffdc6230f872ac56e5cedb835commit | diff
Implement non-greedy tokenizer that tries to maximize token lengths (#242)

* Implement non-greedy tokenizer that tries to maximize token lengths

* Insert single space in front of the prompt

- this is to match original llama tokenizer behavior

---------

Co-authored-by: Jakub Horak <redacted>
main.cpp diff | blob | history
utils.cpp diff | blob | history
Packaging of ggml-org/llama.cpp