]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 152d9d0)
llama : support StableLM 2 1.6B (#5052)
authorcompilade <redacted>
Mon, 22 Jan 2024 11:21:52 +0000 (06:21 -0500)
committerGitHub <redacted>
Mon, 22 Jan 2024 11:21:52 +0000 (13:21 +0200)
commitd6bd4d46ddb6926087c11e0f6633ab1c81da58c3
treec75bf322368750ef332c1b981b34f9fd26e6f084tree
parent152d9d05e097e35f1cac21262946d57faec7542acommit | diff
llama : support StableLM 2 1.6B (#5052)

* llama : support StableLM 2 1.6B

* convert : fix Qwen's set_vocab wrongly naming all special tokens [PAD{id}]

* convert : refactor Qwen's set_vocab to use it for StableLM 2 too

* nix : add tiktoken to llama-python-extra

* convert : use presence of tokenizer.json to determine StableLM tokenizer loader

It's a less arbitrary heuristic than the vocab size.
.devops/nix/package.nix diff | blob | history
convert-hf-to-gguf.py diff | blob | history
llama.cpp diff | blob | history
Packaging of ggml-org/llama.cpp