]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
lookup: complement data from context with general text statistics (#5479)
authorJohannes Gäßler <redacted>
Sat, 23 Mar 2024 00:24:36 +0000 (01:24 +0100)
committerGitHub <redacted>
Sat, 23 Mar 2024 00:24:36 +0000 (01:24 +0100)
commit50ccaf5eacb50a2ca378a4ef0dc7aeb45fead652
tree3ebcfdadf96bb6f3aadd752a1bfe9771ac182d3b
parent56a00f0a2f48a85376f48b5ce77699df781631ae
lookup: complement data from context with general text statistics (#5479)

* lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens

* fixup! lookup: evaluation tools, use corpus/previous gens
13 files changed:
.gitignore
Makefile
common/CMakeLists.txt
common/common.cpp
common/common.h
common/ngram-cache.cpp [new file with mode: 0644]
common/ngram-cache.h [new file with mode: 0644]
examples/lookup/CMakeLists.txt
examples/lookup/lookup-create.cpp [new file with mode: 0644]
examples/lookup/lookup-merge.cpp [new file with mode: 0644]
examples/lookup/lookup-stats.cpp [new file with mode: 0644]
examples/lookup/lookup.cpp
scripts/get-wikitext-103.sh [new file with mode: 0755]