]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
Minor improvements in GPT2 tokenizer (#3567)
authorgoerch <redacted>
Tue, 10 Oct 2023 16:59:52 +0000 (18:59 +0200)
committerGitHub <redacted>
Tue, 10 Oct 2023 16:59:52 +0000 (18:59 +0200)
commit233fc1c69f6f415f35363e18a755f9610e89161b
treed949e9cdaa21419b2a03e7eeb81852cd7a5e6240
parentc5b49360d0d9e49f32e05a9116e90bd0b39a282d
Minor improvements in GPT2 tokenizer (#3567)

* Fixing minor bugs in bpe_gpt2_preprocess

* Don't add bos token in test
llama.cpp
tests/test-tokenizer-0-falcon.cpp
tests/test-tokenizer-0-falcon.py
tests/test-tokenizer-0-llama.cpp
tests/test-tokenizer-0-llama.py