]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 6d16169)
Per token attributes (#7685)
authorjaime-m-p <redacted>
Tue, 4 Jun 2024 07:17:17 +0000 (09:17 +0200)
committerGitHub <redacted>
Tue, 4 Jun 2024 07:17:17 +0000 (09:17 +0200)
commit3b38d48609280aa5f8ab7ea135a4351b2a5ee240
treec3fdcd91e9959ee394ea798d78731e611faaa008tree
parent6d1616944d9efd342ed2a4fd318722adfc9febcdcommit | diff
Per token attributes (#7685)

* Add per token attributes enum
* Using phi-3 for testing 'rstrip'
* Using jina-v2 for testing 'lstrip'
* Brute force test for 'lstrip' and 'rstrip'
* Implement 'rstrip' and 'lstrip'
* Update phi-3 GGUF file (obsolete since 917dc8c)
* Replace llama_token_type with llama_token_attribs
llama.cpp diff | blob | history
llama.h diff | blob | history
models/ggml-vocab-phi-3.gguf diff | blob | history
tests/test-tokenizer-random.py diff | blob | history
Packaging of ggml-org/llama.cpp