]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
Add tokenizer test + revert to C++11 (#355)
authorGeorgi Gerganov <redacted>
Tue, 21 Mar 2023 15:29:41 +0000 (17:29 +0200)
committerGitHub <redacted>
Tue, 21 Mar 2023 15:29:41 +0000 (17:29 +0200)
commiteb34620aeceaf9d9df7fcb19acc17ad41b9f60f8
treeefa10a024845c09ed2630bbe0675618cc7ba0a50
parent2e664f1ff413995506c9a54f3a8d5b8c64e37a91
Add tokenizer test + revert to C++11 (#355)

* Add test-tokenizer-0 to do a few tokenizations - feel free to expand
* Added option to convert-pth-to-ggml.py script to dump just the vocabulary
* Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests)
* Added utility to load vocabulary file from previous point (temporary implementation)
* Avoid using std::string_view and drop back to C++11 (hope I didn't break something)
* Rename gpt_vocab -> llama_vocab
* All CMake binaries go into ./bin/ now
.github/workflows/build.yml
CMakeLists.txt
Makefile
convert-pth-to-ggml.py
main.cpp
models/ggml-vocab.bin [new file with mode: 0644]
quantize.cpp
tests/CMakeLists.txt [new file with mode: 0644]
tests/test-tokenizer-0.cpp [new file with mode: 0644]
utils.cpp
utils.h