]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
llama : move vocab, grammar and sampling into separate files (#8508)
authorGeorgi Gerganov <redacted>
Tue, 23 Jul 2024 10:10:17 +0000 (13:10 +0300)
committerGitHub <redacted>
Tue, 23 Jul 2024 10:10:17 +0000 (13:10 +0300)
commit938943cdbf4dd79005a394d732bc226f9e34e0ff
treeab88d3177b4c34068b49e65d9c9eef3f1af9a606
parent751fcfc6c33ea5f43cadd4d976f8fb176871df5e
llama : move vocab, grammar and sampling into separate files (#8508)

* llama : move sampling code into llama-sampling

ggml-ci

* llama : move grammar code into llama-grammar

ggml-ci

* cont

ggml-ci

* cont : pre-fetch rules

* cont

ggml-ci

* llama : deprecate llama_sample_grammar

* llama : move tokenizers into llama-vocab

ggml-ci

* make : update llama.cpp deps [no ci]

* llama : redirect external API to internal APIs

ggml-ci

* llama : suffix the internal APIs with "_impl"

ggml-ci

* llama : clean-up
18 files changed:
Makefile
Package.swift
common/sampling.cpp
examples/gbnf-validator/gbnf-validator.cpp
include/llama.h
src/CMakeLists.txt
src/llama-grammar.cpp [new file with mode: 0644]
src/llama-grammar.h [new file with mode: 0644]
src/llama-impl.h [new file with mode: 0644]
src/llama-sampling.cpp [new file with mode: 0644]
src/llama-sampling.h [new file with mode: 0644]
src/llama-vocab.cpp [new file with mode: 0644]
src/llama-vocab.h [new file with mode: 0644]
src/llama.cpp
src/unicode.cpp
src/unicode.h
tests/test-grammar-integration.cpp
tests/test-llama-grammar.cpp