git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit

]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit

overview / pkg / ggml / sources / llama.cpp / commit

author	Georgi Gerganov <redacted>
	Sat, 4 May 2024 05:32:32 +0000 (08:32 +0300)
committer	GitHub <redacted>
	Sat, 4 May 2024 05:32:32 +0000 (08:32 +0300)
commit	92139b90af4841d7fd060b526bdd443b621770ff
tree	9679c3de1b39970ca73b5bd988c63ddac0359ca6	tree
parent	a2ac89d6efb41b535778bfeaecaae8fe295b6ed3	commit \| diff

tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)

* tests : add test-tokenizer-0.sh

* unicode : add all unicode number ranges

* starcoder : fix pre-tokenizer

* tests : add test that fails with DeepSeek tokenizers

* falcon : fix regex

* unicode : regenerate unicode tables

* refact : add tokenizer model

* lint : fix

* tests : disable failing tests

ggml-ci

* refact : add tests files

ggml-ci

* convert : print -> logging

ggml-ci

* lint : fix

* unicode : digit -> number

* phi-3 : update

41 files changed:

.flake8		diff \| blob \| history
Makefile		diff \| blob \| history
convert-hf-to-gguf-update.py		diff \| blob \| history
convert-hf-to-gguf.py		diff \| blob \| history
llama.cpp		diff \| blob \| history
llama.h		diff \| blob \| history
models/ggml-vocab-bert-bge.gguf.inp		diff \| blob \| history
models/ggml-vocab-bert-bge.gguf.out		diff \| blob \| history
models/ggml-vocab-deepseek-coder.gguf.inp		diff \| blob \| history
models/ggml-vocab-deepseek-coder.gguf.out		diff \| blob \| history
models/ggml-vocab-deepseek-llm.gguf.inp		diff \| blob \| history
models/ggml-vocab-deepseek-llm.gguf.out		diff \| blob \| history
models/ggml-vocab-falcon.gguf.inp		diff \| blob \| history
models/ggml-vocab-falcon.gguf.out		diff \| blob \| history
models/ggml-vocab-gpt-2.gguf.inp		diff \| blob \| history
models/ggml-vocab-gpt-2.gguf.out		diff \| blob \| history
models/ggml-vocab-llama-bpe.gguf.inp		diff \| blob \| history
models/ggml-vocab-llama-bpe.gguf.out		diff \| blob \| history
models/ggml-vocab-llama-spm.gguf.inp		diff \| blob \| history
models/ggml-vocab-llama-spm.gguf.out		diff \| blob \| history
models/ggml-vocab-mpt.gguf.inp		diff \| blob \| history
models/ggml-vocab-mpt.gguf.out		diff \| blob \| history
models/ggml-vocab-phi-3.gguf		diff \| blob \| history
models/ggml-vocab-phi-3.gguf.inp		diff \| blob \| history
models/ggml-vocab-phi-3.gguf.out		diff \| blob \| history
models/ggml-vocab-refact.gguf		diff \| blob \| history
models/ggml-vocab-refact.gguf.inp	[new file with mode: 0644]	blob
models/ggml-vocab-refact.gguf.out	[new file with mode: 0644]	blob
models/ggml-vocab-starcoder.gguf.inp		diff \| blob \| history
models/ggml-vocab-starcoder.gguf.out		diff \| blob \| history
scripts/gen-unicode-data.py	[new file with mode: 0644]	blob
tests/CMakeLists.txt		diff \| blob \| history
tests/test-tokenizer-0-bpe.py	[deleted file]	blob \| history
tests/test-tokenizer-0-spm.py	[deleted file]	blob \| history
tests/test-tokenizer-0.cpp		diff \| blob \| history
tests/test-tokenizer-0.py	[new file with mode: 0644]	blob
tests/test-tokenizer-0.sh	[new file with mode: 0755]	blob
unicode-data.cpp		diff \| blob \| history
unicode-data.h		diff \| blob \| history
unicode.cpp		diff \| blob \| history
unicode.h		diff \| blob \| history

Packaging of ggml-org/llama.cpp

RSS Atom