]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
Simplify and improve CUDA graphs through use of indirect copy pointers (#9017)
authorAlan Gray <redacted>
Thu, 3 Apr 2025 01:31:15 +0000 (02:31 +0100)
committerGitHub <redacted>
Thu, 3 Apr 2025 01:31:15 +0000 (03:31 +0200)
commit3f9da22c2b21a2cef216de50006436ef1cab8764
treef63d5229d3e0aef8416c61e21624524daca99985
parent2a0dc97e56eac6db0a4016f0b45da6d0a0055ef2
Simplify and improve CUDA graphs through use of indirect copy pointers (#9017)

* CUDA: Simplify and improve CUDA graphs through use of indirect copy pointers

Previously there was complexity in the CUDA graphs implementation due
frequently changing parameters to copy kernels associated with K and V
cache pointers. This patch simplifies by using indirection to avoid
such parameters frequently changing, avoiding the need for frequent
graph updates.

Fixes #12152

* Addressed comments

* fix HIP builds

* properly sync to stream

* removed ggml_cuda_cpy_fn_ptrs

* move stream sync before free

* guard to only use indirection with graphs

* style fixes

* check for errors

---------

Co-authored-by: slaren <redacted>
ggml/src/ggml-cuda/common.cuh
ggml/src/ggml-cuda/cpy.cu
ggml/src/ggml-cuda/cpy.cuh
ggml/src/ggml-cuda/ggml-cuda.cu