]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
convert.py : support bpe tokenizer (#2228)
authorldwang <redacted>
Tue, 25 Jul 2023 13:22:09 +0000 (21:22 +0800)
committerGitHub <redacted>
Tue, 25 Jul 2023 13:22:09 +0000 (16:22 +0300)
commitfce48caf9a6b9930eee9e2a5971428cdff403ba8
tree23682a47a7223d0033fc99fb7e0018dc8d05dbf0
parent875086bdb95ce1f3294439811536533199e3b579
convert.py : support bpe tokenizer (#2228)

* support bpe tokenizer in convert

Signed-off-by: ldwang <redacted>
* support bpe tokenizer in convert

Signed-off-by: ldwang <redacted>
* support bpe tokenizer in convert, fix

Signed-off-by: ldwang <redacted>
---------

Signed-off-by: ldwang <redacted>
Co-authored-by: ldwang <redacted>
convert.py