]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 1269cb1)
convert : import bytes_to_unicode from convert_slow_tokenizer (#26217)
authorCaleb DeLeeuw <redacted>
Tue, 4 Aug 2026 07:34:30 +0000 (00:34 -0700)
committerGitHub <redacted>
Tue, 4 Aug 2026 07:34:30 +0000 (10:34 +0300)
commitb06fbc968b980daa313687152114dd1bfd1edaa9
treea2d92eabe40dd20be6cd37877c90f75f42fce48dtree
parent1269cb1ff1598751f846241be90083ae9ad036fbcommit | diff
convert : import bytes_to_unicode from convert_slow_tokenizer (#26217)

bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.

Applies the same fix to chatglm.py.
conversion/chatglm.py diff | blob | history
conversion/qwen.py diff | blob | history
Packaging of ggml-org/llama.cpp