]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: d9f65c9)
android : fix utf8 decoding error (#5935)
authorDean <redacted>
Sun, 10 Mar 2024 20:03:17 +0000 (04:03 +0800)
committerGitHub <redacted>
Sun, 10 Mar 2024 20:03:17 +0000 (22:03 +0200)
commit7ab7b733bb48250b2df26c12b00256ef42c76932
tree2b5596009f19da27436565c20c1fb155389ccb67tree
parentd9f65c97c3dc3aa6fa27470b8c6e69b437ec1a27commit | diff
android : fix utf8 decoding error (#5935)

* examples: fix utf8 decoding error

some models have a tokenizer that decodes an id into an incomplete utf8 sequence, need to validate and wait for next token
one example would be: https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat-GGUF/resolve/main/qwen1_5-1_8b-chat-q4_0.gguf and and an example of the token is 18137

* android : minor

---------

Co-authored-by: zhangfuwen <redacted>
Co-authored-by: Georgi Gerganov <redacted>
examples/llama.android/app/src/main/cpp/llama-android.cpp diff | blob | history
examples/llama.android/app/src/main/java/com/example/llama/Llm.kt diff | blob | history
Packaging of ggml-org/llama.cpp