git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit

]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit

overview / pkg / ggml / sources / llama.cpp / commit

author	Galunid <redacted>
	Mon, 23 Oct 2023 19:46:00 +0000 (21:46 +0200)
committer	GitHub <redacted>
	Mon, 23 Oct 2023 19:46:00 +0000 (21:46 +0200)
commit	69a6735087c3634963c642fd69f0851ac479cd78
tree	b4a0a0037e1d748e1dbf531ab5c83da5fcd7bd2e	tree
parent	5be6c803fa5378f62a1590f3ad8c6b64c7c0c2ce	commit \| diff

Update special token handling in conversion scripts for gpt2 derived tokenizers (#3746)

We still have the heads up in `README.md` regarding `bpe` tokenizers and this patch is needed for

- a couple of tokenizer tests
- some more `special` and `non-special` added tokens handling (as far as I understand it)

* Update special token handling

* Add mpt

Packaging of ggml-org/llama.cpp

RSS Atom

convert-bloom-hf-to-gguf.py		diff \| blob \| history
convert-gptneox-hf-to-gguf.py		diff \| blob \| history
convert-mpt-hf-to-gguf.py		diff \| blob \| history
convert-refact-hf-to-gguf.py		diff \| blob \| history
convert-starcoder-hf-to-gguf.py		diff \| blob \| history