]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
vulkan: optimize mul_mat for small values of N (#10991)
authorJeff Bolz <redacted>
Mon, 30 Dec 2024 17:27:11 +0000 (11:27 -0600)
committerGitHub <redacted>
Mon, 30 Dec 2024 17:27:11 +0000 (18:27 +0100)
commit716bd6dec3e044e5c325386b5b0483392b24cefe
tree7d020de6c1eb6d8e28caf8cc7bd3e66863a7c590
parentc250ecb3157f3bae0a45f44c3c953b5414d4c2f7
vulkan: optimize mul_mat for small values of N (#10991)

Make the mul_mat_vec shaders support N>1 (as a spec constant, NUM_COLS) where
the batch_strides are overloaded to hold the row strides. Put the loads from the
B matrix in the innermost loop because it should cache better.

Share some code for reducing the result values to memory in mul_mat_vec_base.
ggml/src/ggml-vulkan/ggml-vulkan.cpp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_base.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q2_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q3_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q4_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q5_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q6_k.comp
tests/test-backend-ops.cpp