]> git.djapps.eu Git - pkg/ggml/sources/ggml/commit
vulkan: optimize mul_mat for small values of N (llama/10991)
authorJeff Bolz <redacted>
Mon, 30 Dec 2024 17:27:11 +0000 (11:27 -0600)
committerGeorgi Gerganov <redacted>
Fri, 3 Jan 2025 12:00:38 +0000 (14:00 +0200)
commit8603ac1c80b2c46957411dcfc63c3e67956f7548
tree5fa46769ea6798a177b856db3768ee967355411a
parent74c83c417838b18fe9e19992b33274c9a04b5a8d
vulkan: optimize mul_mat for small values of N (llama/10991)

Make the mul_mat_vec shaders support N>1 (as a spec constant, NUM_COLS) where
the batch_strides are overloaded to hold the row strides. Put the loads from the
B matrix in the innermost loop because it should cache better.

Share some code for reducing the result values to memory in mul_mat_vec_base.
src/ggml-vulkan/ggml-vulkan.cpp
src/ggml-vulkan/vulkan-shaders/mul_mat_vec.comp
src/ggml-vulkan/vulkan-shaders/mul_mat_vec_base.comp
src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q2_k.comp
src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q3_k.comp
src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q4_k.comp
src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q5_k.comp
src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q6_k.comp
tests/test-backend-ops.cpp