git.djapps.eu Git - pkg/ggml/sources/ggml/commit

]> git.djapps.eu Git - pkg/ggml/sources/ggml/commit

overview / pkg / ggml / sources / ggml / commit

author	Gaurav Garg <redacted>
	Wed, 19 Mar 2025 19:52:06 +0000 (01:22 +0530)
committer	Georgi Gerganov <redacted>
	Thu, 27 Mar 2025 07:35:24 +0000 (09:35 +0200)
commit	807726590102f2df4e93ef8f5e100d28f757994b
tree	46660a8852910d3da86eca00a72b3df51a2b8274	tree
parent	57d281541618f56bec47c7b4e3070e474cc21846	commit \| diff

CUDA: Improve flash decoding kernel GPU occupancy for BS=1 case (llama/12183)

- Find out active blocks per SM using cudaOccupancyMaxActiveBlocksPerMultiprocessor API. Use this value to determine the optimal parallel_blocks value.
- Prefer vector flash attention kernels over MMA kernel for BS=1

Fixes Issue: #12182
---------

Co-authored-by: Johannes Gäßler <redacted>

12 files changed:

Packaging of ggml-org/ggml

RSS Atom

src/ggml-cuda/fattn-common.cuh		diff \| blob \| history
src/ggml-cuda/fattn-mma-f16.cuh		diff \| blob \| history
src/ggml-cuda/fattn-tile-f16.cu		diff \| blob \| history
src/ggml-cuda/fattn-tile-f32.cu		diff \| blob \| history
src/ggml-cuda/fattn-vec-f16.cuh		diff \| blob \| history
src/ggml-cuda/fattn-vec-f32.cuh		diff \| blob \| history
src/ggml-cuda/fattn-wmma-f16.cu		diff \| blob \| history
src/ggml-cuda/fattn.cu		diff \| blob \| history
src/ggml-cuda/ggml-cuda.cu		diff \| blob \| history
src/ggml-cuda/vendors/hip.h		diff \| blob \| history
src/ggml-cuda/vendors/musa.h		diff \| blob \| history
tests/test-backend-ops.cpp		diff \| blob \| history