]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
vocab : add tokenizer support for jina-embeddings-v2-base-zh (#18756)
authoro7si <redacted>
Sun, 31 May 2026 10:37:35 +0000 (18:37 +0800)
committerGitHub <redacted>
Sun, 31 May 2026 10:37:35 +0000 (12:37 +0200)
commitd4c8e2c29ce2fb9a251a0a4a16d6c857b4f70f8c
tree76f3aaf5f1337311c04b1b1acfeff4d7c2cd05c0
parent3292da09f65e3b3a5128807b510ecdcc6bccc0c2
vocab : add tokenizer support for jina-embeddings-v2-base-zh (#18756)

* vocab : add jina-embeddings-v2-base-zh (whitespace tokenizer)

* lowercase defaults to true

* type fix

---------

Co-authored-by: Sigbjørn Skjæret <redacted>
conversion/base.py
conversion/bert.py
gguf-py/gguf/constants.py
gguf-py/gguf/gguf_writer.py
gguf-py/gguf/vocab.py
src/llama-arch.cpp
src/llama-arch.h
src/llama-vocab.cpp
src/llama-vocab.h