]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
overview / pkg / ggml / sources / llama.cpp / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 34b2a5e)
cuda : improve text-generation and batched decoding performance (#3776)
authorGeorgi Gerganov <redacted>
Fri, 27 Oct 2023 14:01:23 +0000 (17:01 +0300)
committerGitHub <redacted>
Fri, 27 Oct 2023 14:01:23 +0000 (17:01 +0300)
commit2f9ec7e271220a78fe27c9e6ccbcc0dda31cda0f
treee36dcd292a133c5106376c6ddd90c68f446ddae6tree
parent34b2a5e1ee4fe6295fb4420eb91131d743694c65commit | diff
cuda : improve text-generation and batched decoding performance (#3776)

* cuda : prints wip

* cuda : new cublas gemm branch for multi-batch quantized src0

* cuda : add F32 sgemm branch

* cuda : fine-tune >= VOLTA params + use MMQ only for small batches

* cuda : remove duplicated cuBLAS GEMM code

* cuda : add CUDA_USE_TENSOR_CORES and GGML_CUDA_FORCE_MMQ macros

* build : add compile option to force use of MMQ kernels
CMakeLists.txt diff | blob | history
Makefile diff | blob | history
ggml-cuda.cu diff | blob | history
llama.cpp diff | blob | history
llama.h diff | blob | history
Packaging of ggml-org/llama.cpp