]> git.djapps.eu Git - pkg/ggml/sources/ggml/commit
overview / pkg / ggml / sources / ggml / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: c68c544)
common : support utf-8 + fix gpt_tokenize + fix MPT model import (#179)
authorklosax <redacted>
Sun, 21 May 2023 08:21:51 +0000 (10:21 +0200)
committerGitHub <redacted>
Sun, 21 May 2023 08:21:51 +0000 (11:21 +0300)
commit36a005c930cf8fb1f68f0421c5a848a0c8825f5a
treedffd451aed107c94c3cb1259c533dee5597ad84btree
parentc68c544f914d0f0cb50ea8903be64c5a487c2145commit | diff
common : support utf-8 + fix gpt_tokenize + fix MPT model import (#179)

* Update convert-h5-to-ggml.py

* Import tokens correctly

* gpt_tokenize: Convert input to utf-8 + bug fix

* common : minor style fixes

---------

Co-authored-by: Georgi Gerganov <redacted>
examples/common.cpp diff | blob | history
examples/mpt/convert-h5-to-ggml.py diff | blob | history
Packaging of ggml-org/ggml