]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
vocab: add normalizer.lowercase support to WPM (#23899)
authoro7si <redacted>
Mon, 1 Jun 2026 11:26:47 +0000 (19:26 +0800)
committerGitHub <redacted>
Mon, 1 Jun 2026 11:26:47 +0000 (14:26 +0300)
commit5aba5364d959756c43eccc45596731e7b4eb3aa4
tree26e76b5c1001cc69b6f54e9858aae5010f0ce596
parent8e6fff84de4a31506e0f90bacbf821731e66d237
vocab: add normalizer.lowercase support to WPM (#23899)

* vocab : add jina-embeddings-v2-base-zh (whitespace tokenizer)

* vocab : add normalizer.lowercase support to WPM

* vocab : default normalizer.lowercase to false for whitespace pre-tokenizer
src/llama-vocab.cpp