]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 568013d)
convert : avoid calls to tokenizer.added_tokens_decoder (#12473)
authorBartowski <redacted>
Thu, 20 Mar 2025 06:36:37 +0000 (02:36 -0400)
committerGitHub <redacted>
Thu, 20 Mar 2025 06:36:37 +0000 (08:36 +0200)
commit732b5fbf5e7f9cf069942f0c5850ee959ef321ba
treecc484fcc61909c90a18acb4bb9a1b36f5138a4e8tree
parent568013d0cd3d5add37c376b3d5e959809b711fc7commit | diff
convert : avoid calls to tokenizer.added_tokens_decoder (#12473)

tokenizer.added_tokens_decoder returns a fresh dict every time relatively slowly (~0.04s on average) which results in massive slowdowns when we have a huge number of added tokens
convert_hf_to_gguf.py diff | blob | history
Packaging of ggml-org/llama.cpp