]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 8308f98)
vocab : prevent tokenizer overflow (#14301)
authorRuikai Peng <redacted>
Fri, 20 Jun 2025 14:13:06 +0000 (22:13 +0800)
committerGitHub <redacted>
Fri, 20 Jun 2025 14:13:06 +0000 (07:13 -0700)
commitdd6e6d0b6a4bbe3ebfc931d1eb14db2f2b1d70af
treed879284e7b3451ed7d2ca635656bab7ec4fa74d9tree
parent8308f98c7fb778e54bf75538f5234d8bd20915e9commit | diff
vocab : prevent tokenizer overflow (#14301)

* vocab : prevent stack overflow in tokenize

* vocab : return error instead of aborting on oversized token count

* vocab : INT32_MIN from llama_tokenize on overflow
common/common.cpp diff | blob | history
include/llama.h diff | blob | history
src/llama-vocab.cpp diff | blob | history
Packaging of ggml-org/llama.cpp