]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: bbce619)
vocab : fix HybridDNA tokenizer (#23466)
authorKashif Rasul <redacted>
Fri, 22 May 2026 09:17:31 +0000 (11:17 +0200)
committerGitHub <redacted>
Fri, 22 May 2026 09:17:31 +0000 (11:17 +0200)
commitafcda09d154a285cd366135f98ffc1d357f7ddbd
tree4027b60ad4a9f22d780a512555b019e2e8c740b0tree
parentbbce619adb409880fb6db850a1c5a5f36a4dc7b1commit | diff
vocab : fix HybridDNA tokenizer (#23466)

* vocab : mark hybriddna k-mers to avoid BPE token collisions

* improved loop

---------

Co-authored-by: Sigbjørn Skjæret <redacted>
conversion/base.py diff | blob | history
src/llama-vocab.cpp diff | blob | history
Packaging of ggml-org/llama.cpp