]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 2b3a665)
py : fix missing added_tokens_dict for SPM and BPE vocabs (#4971)
authorGeorgi Gerganov <redacted>
Wed, 17 Jan 2024 13:45:03 +0000 (15:45 +0200)
committerGitHub <redacted>
Wed, 17 Jan 2024 13:45:03 +0000 (15:45 +0200)
commit4f4bf35f46600441dec2f941e667291eeb9a18d8
treebcbb84ce3d43ab5788303312f8f5b35090292ad3tree
parent2b3a665d3917edf393761a24c4835447894df74acommit | diff
py : fix missing added_tokens_dict for SPM and BPE vocabs (#4971)

* py : fix missing added_tokens_dict for SPM vocab

* py : pad with unknown tokens when data is missing

ggml-ci

* py : fix BPE vocab conversion

ggml-ci

* py : fix padded dummy tokens (I hope)
convert.py diff | blob | history
Packaging of ggml-org/llama.cpp