]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
tokenize : add --no-parse-special option (#8423)
authorcompilade <redacted>
Thu, 11 Jul 2024 07:41:48 +0000 (03:41 -0400)
committerGitHub <redacted>
Thu, 11 Jul 2024 07:41:48 +0000 (10:41 +0300)
commit9a55ffe6fbf7f19c865f8f277ca7cd585cc0b094
tree5ac620546d21161194aef81aebc34377fde0d4bf
parent7a221b672e49dfae459b1af27210ba3f2b5419b6
tokenize : add --no-parse-special option (#8423)

This should allow more easily explaining
how parse_special affects tokenization.
examples/tokenize/tokenize.cpp