git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit

author	goerch <redacted>
	Sat, 16 Sep 2023 11:41:33 +0000 (13:41 +0200)
committer	GitHub <redacted>
	Sat, 16 Sep 2023 11:41:33 +0000 (13:41 +0200)
commit	b08e75baea294e366628b898e85c0bd359b58115
tree	417a1a8e7589567ceedba88771056aee080c8e70	tree
parent	e6616cf0db2b63189fc34d0076f654af9adecdf8	commit \| diff

Fixing the last deviations from sentencepiece indicated by test-tokenizer-1 (#3170)

* Fix für #2721

* Reenable tokenizer test for LLaMa

* Add `console.cpp` dependency

* Fix dependency to `common`

* Fixing wrong fix.

* Make console usage platform specific

Work on compiler warnings.

* Adapting makefile

* Remove trailing whitespace

* Adapting the other parts of the makefile

* Fix typo.

* Fixing the last deviations from sentencepiece indicated by test-tokenizer-1

* Simplify logic

* Add missing change...

* Fix ugly compiler warning

* llama_tokenize should accept strings containing NUL now

* Adding huichen's test case

common/common.cpp		diff \| blob \| history
examples/train-text-from-scratch/train-text-from-scratch.cpp		diff \| blob \| history
llama.cpp		diff \| blob \| history
llama.h		diff \| blob \| history
tests/test-tokenizer-0-llama.cpp		diff \| blob \| history
tests/test-tokenizer-1-llama.cpp		diff \| blob \| history