]> git.djapps.eu Git - pkg/ggml/sources/whisper.cpp/commit
vulkan: optimize mul_mat for small values of N (llama/10991)
authorJeff Bolz <redacted>
Mon, 30 Dec 2024 17:27:11 +0000 (11:27 -0600)
committerGeorgi Gerganov <redacted>
Sat, 4 Jan 2025 08:45:01 +0000 (10:45 +0200)
commitcea5f1c52faa27f4ebd4c588539035a220f8bef5
treecdbc481a1bb80c4e4438d36d57c5c792a8d5c669
parent2112462db407d02c82b36d887fb8721923f426e8
vulkan: optimize mul_mat for small values of N (llama/10991)

Make the mul_mat_vec shaders support N>1 (as a spec constant, NUM_COLS) where
the batch_strides are overloaded to hold the row strides. Put the loads from the
B matrix in the innermost loop because it should cache better.

Share some code for reducing the result values to memory in mul_mat_vec_base.
ggml/src/ggml-vulkan/ggml-vulkan.cpp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_base.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q2_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q3_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q4_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q5_k.comp
ggml/src/ggml-vulkan/vulkan-shaders/mul_mat_vec_q6_k.comp