]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: fabf30b)
Tokenizer SPM fixes for phi-3 and llama-spm (#7375)
authorjaime-m-p <redacted>
Mon, 20 May 2024 18:15:57 +0000 (20:15 +0200)
committerGitHub <redacted>
Mon, 20 May 2024 18:15:57 +0000 (20:15 +0200)
commit917dc8cfa67a72fb7c8bf7392270da3bf4833af4
tree170c39ba79ae72036105970d92548da420699ba1tree
parentfabf30b4c4fca32e116009527180c252919ca922commit | diff
Tokenizer SPM fixes for phi-3 and llama-spm (#7375)

* Update brute force test: special tokens
* Fix added tokens
  - Try to read 'added_tokens.json'.
  - Try to read 'tokenizer_config.json'.
  - Try to read 'tokenizer.json'.
* Fix special tokens rtrim

Co-authored-by: Georgi Gerganov <redacted>
* server : fix test regexes
convert-hf-to-gguf.py diff | blob | history
examples/server/tests/features/server.feature diff | blob | history
examples/server/tests/features/slotsave.feature diff | blob | history
llama.cpp diff | blob | history
tests/test-tokenizer-random.py diff | blob | history
Packaging of ggml-org/llama.cpp