git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit

]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit

overview / pkg / ggml / sources / llama.cpp / commit

author	Mathijs Henquet <redacted>
	Thu, 12 Sep 2024 20:30:11 +0000 (22:30 +0200)
committer	GitHub <redacted>
	Thu, 12 Sep 2024 20:30:11 +0000 (22:30 +0200)
commit	78203641fee3b1f82abaff0c7f667e1b4a286390
tree	16a426a27e25662967717c4cf44d07f8f8db71ca	tree
parent	e6b7801bd189d102d901d3e72035611a25456ef1	commit \| diff

server : Add option to return token pieces in /tokenize endpoint (#9108)

* server : added with_pieces functionality to /tokenize endpoint

* server : Add tokenize with pieces tests to server.feature

* Handle case if tokenizer splits along utf8 continuation bytes

* Add example of token splitting

* Remove trailing ws

* Fix trailing ws

* Maybe fix ci

* maybe this fix windows ci?

---------

Co-authored-by: Xuan Son Nguyen <redacted>

Packaging of ggml-org/llama.cpp

RSS Atom

.github/workflows/server.yml		diff \| blob \| history
examples/server/README.md		diff \| blob \| history
examples/server/server.cpp		diff \| blob \| history
examples/server/tests/features/server.feature		diff \| blob \| history
examples/server/tests/features/steps/steps.py		diff \| blob \| history
examples/server/utils.hpp		diff \| blob \| history