]> git.djapps.eu Git - pkg/ggml/sources/ggml/commit
Simplify and improve CUDA graphs through use of indirect copy pointers (llama/9017)
authorAlan Gray <redacted>
Thu, 3 Apr 2025 01:31:15 +0000 (02:31 +0100)
committerGeorgi Gerganov <redacted>
Tue, 8 Apr 2025 08:47:46 +0000 (11:47 +0300)
commite5918e0d7a633462f056abd5ef41044c347bc991
tree0893a71e81b11c66b7d5a8da66ca86dad646f305
parent7b4b9fdae1264a4b1499290ce7fdf83b6b1778a8
Simplify and improve CUDA graphs through use of indirect copy pointers (llama/9017)

* CUDA: Simplify and improve CUDA graphs through use of indirect copy pointers

Previously there was complexity in the CUDA graphs implementation due
frequently changing parameters to copy kernels associated with K and V
cache pointers. This patch simplifies by using indirection to avoid
such parameters frequently changing, avoiding the need for frequent
graph updates.

Fixes #12152

* Addressed comments

* fix HIP builds

* properly sync to stream

* removed ggml_cuda_cpy_fn_ptrs

* move stream sync before free

* guard to only use indirection with graphs

* style fixes

* check for errors

---------

Co-authored-by: slaren <redacted>
src/ggml-cuda/common.cuh
src/ggml-cuda/cpy.cu
src/ggml-cuda/cpy.cuh
src/ggml-cuda/ggml-cuda.cu