]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 46ef5b5)
Improve handling of special tokens in GGML to GGUF converter (#2725)
authorKerfuffle <redacted>
Tue, 22 Aug 2023 23:39:39 +0000 (17:39 -0600)
committerGitHub <redacted>
Tue, 22 Aug 2023 23:39:39 +0000 (17:39 -0600)
commit777f42ba18b29f25c71ff8de3ecf97b8017304c0
treec4622646a366bd1f302293cb8aa7b0420d18b17etree
parent46ef5b5fcf4c366e1fb27726b6394adbbf8fd0eacommit | diff
Improve handling of special tokens in GGML to GGUF converter (#2725)

* Improve UNK, BOS, EOS token handling when converting without metadata.

* Allow importing as a module.

* Remove some obsolete code and minor cleanups.

* Set default UNK token mapping from -1 to 0 in llama.cpp

* Try to handle overflow due to buggy Windows Python with a better error message
convert-llama-ggmlv3-to-gguf.py diff | blob | history
llama.cpp diff | blob | history
Packaging of ggml-org/llama.cpp