]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: c4a3a4f)
BERT tokenizer fixes (#6498)
authorJared Van Bortel <redacted>
Tue, 9 Apr 2024 17:44:08 +0000 (13:44 -0400)
committerGitHub <redacted>
Tue, 9 Apr 2024 17:44:08 +0000 (13:44 -0400)
commit1b67731e184e27a465b8c5476061294a4af668ea
tree15a2d877029fb509a34e462c227475bc7d6dc31etree
parentc4a3a4ff47d62d2503ddf9bd91b58c21f04fe3c3commit | diff
BERT tokenizer fixes (#6498)

Key changes:
* BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS
* Nomic Embed conversion: pad vocab instead of slicing embedding tensor
* llama_tokenize: handle added special tokens like HF does
20 files changed:
common/common.cpp diff | blob | history
common/common.h diff | blob | history
convert-hf-to-gguf.py diff | blob | history
convert-persimmon-to-gguf.py diff | blob | history
convert.py diff | blob | history
examples/embedding/embedding.cpp diff | blob | history
examples/imatrix/imatrix.cpp diff | blob | history
examples/infill/infill.cpp diff | blob | history
examples/llava/llava-cli.cpp diff | blob | history
examples/lookahead/lookahead.cpp diff | blob | history
examples/lookup/lookup-create.cpp diff | blob | history
examples/lookup/lookup-stats.cpp diff | blob | history
examples/lookup/lookup.cpp diff | blob | history
examples/main/main.cpp diff | blob | history
examples/perplexity/perplexity.cpp diff | blob | history
examples/server/server.cpp diff | blob | history
examples/speculative/speculative.cpp diff | blob | history
examples/tokenize/tokenize.cpp diff | blob | history
llama.cpp diff | blob | history
llama.h diff | blob | history
Packaging of ggml-org/llama.cpp