From: Jim Wu Date: Thu, 6 Aug 2026 08:43:26 +0000 (-0700) Subject: ci : onboard AMD ROCm CI with gfx1151 fixes (#26544) X-Git-Tag: upstream/0.0.10438~145 X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=a1f96d4fc2c9e4101a6666a9d87f547e7e880df6;p=pkg%2Fggml%2Fsources%2Fllama.cpp ci : onboard AMD ROCm CI with gfx1151 fixes (#26544) * ci: prepare for amd rocm ci Signed-off-by: Aaron Teo * ci: fix editorconfig-checker Signed-off-by: Aaron Teo * ci: fix device not recognised Signed-off-by: Aaron Teo * ci: rename gpu-amd to gpu-hip Signed-off-by: Aaron Teo * ci: gpu-hip to gpu-rocm haha Signed-off-by: Aaron Teo * CUDA: allow integrated-GPU host output buffer in debug assert On integrated GPUs (APUs), the scheduler can legitimately place a graph node's output on the host-visible buffer, which ggml_cuda_compute_forward already handles. The debug assert in ggml_cuda_graph_evaluate_and_capture required every node output to be on the device buffer, so a debug build aborts on such a node (e.g. attn_residual ADD -> ROCm_Host on RDNA3.5). The source-tensor assert directly below already permits this via the integrated + cuda_host exception; apply the same exception to the node's own output buffer. Debug-only; no effect on release/compute. Fixes test-recurrent-state-rollback on gfx1151 (Strix Halo). * ci: enable unified memory for ROCm gfx1151 job Work around a coherence issue on integrated RDNA3.5 (gfx1151) where GPU kernels reading mmap-loaded weights can return incorrect output, which makes test-llama-archs (and real inference) intermittently wrong. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 uses managed memory, which restores coherence. Remove once the underlying ROCm/HIP issue is fixed. * test-llama-archs: skip jamba on HIP backend jamba produces incorrect output (~0.55 NMSE vs CPU) on the HIP backend on RDNA3.5 (gfx1151); the SSM kernels need separate investigation. Skip it for now, matching the existing per-backend carve-outs (WebGPU), so the ROCm CI can run the test for the remaining architectures. * ci: use HIP_LAUNCH_BLOCKING for ROCm gfx1151 job The gfx1151 ROCm CI job produced incorrect inference output (qwen3 perplexity ~88 vs ~9.4) due to an async-execution correctness issue in the HIP path. Serializing kernel launches with HIP_LAUNCH_BLOCKING=1 restores correctness. This replaces the earlier GGML_CUDA_ENABLE_UNIFIED_MEMORY workaround, which did not fix batched inference. * test-backend-sampler: skip top-k subtests on HIP backend The ROCm backend does not support the TOP_K/ARGSORT op at vocab scale (no CUB; bitonic argsort is capped at ncols <= 1024), so top-k/top-p backend samplers cannot be offloaded. The penalties, set_sampler, mixed, and top_p subtests assert that offload happened, so they fail on HIP. Skip them until TOP_K is supported on the ROCm backend. * Update tests/test-backend-sampler.cpp Co-authored-by: Aaron Teo * Update tests/test-backend-sampler.cpp Co-authored-by: Georgi Gerganov --------- Signed-off-by: Aaron Teo Co-authored-by: Aaron Teo Co-authored-by: Jim Wu Co-authored-by: Aaron Teo Co-authored-by: Georgi Gerganov --- diff --git a/.github/workflows/build-self-hosted.yml b/.github/workflows/build-self-hosted.yml index 441a897e5..0ef202193 100644 --- a/.github/workflows/build-self-hosted.yml +++ b/.github/workflows/build-self-hosted.yml @@ -71,6 +71,26 @@ jobs: nvidia-smi GG_BUILD_CUDA=1 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp + gpu-rocm: + runs-on: [self-hosted, Linux, AMD] + + steps: + - name: Clone + id: checkout + uses: actions/checkout@v6 + + - name: Test + id: ggml-ci + # HIP_LAUNCH_BLOCKING=1: workaround for an async-execution correctness + # issue on integrated RDNA3.5 (gfx1151) where batched inference returns + # incorrect output (perplexity ~88 vs ~9.4). Serializing kernel launches + # restores correctness. Remove once the underlying ROCm/HIP issue is fixed. + env: + HIP_LAUNCH_BLOCKING: "1" + run: | + rocminfo + GG_BUILD_ROCM=1 GG_BUILD_AMDGPU_TARGETS=gfx1151 bash ./ci/run.sh ~/results/llama.cpp ~/mnt/llama.cpp + gpu-vulkan-nvidia-cm: runs-on: [self-hosted, Linux, NVIDIA] diff --git a/ci/run.sh b/ci/run.sh index e4a34ff0a..68a95ec32 100755 --- a/ci/run.sh +++ b/ci/run.sh @@ -10,6 +10,9 @@ # # with CUDA support # GG_BUILD_CUDA=1 bash ./ci/run.sh ./tmp/results ./tmp/mnt # +# # with ROCm support +# GG_BUILD_ROCM=1 GG_BUILD_AMDGPU_TARGETS=gfx1151 bash ./ci/run.sh ./tmp/results ./tmp/mnt +# # # with SYCL support # GG_BUILD_SYCL=1 bash ./ci/run.sh ./tmp/results ./tmp/mnt # @@ -89,7 +92,7 @@ if [ ! -z ${GG_BUILD_CUDA} ]; then fi if [ ! -z ${GG_BUILD_ROCM} ]; then - CMAKE_EXTRA="${CMAKE_EXTRA} -DGGML_HIP=ON" + CMAKE_EXTRA="${CMAKE_EXTRA} -DCMAKE_HIP_COMPILER=$(hipconfig -l)/clang -DGGML_HIP=ON -DGGML_HIP_ROCWMMA_FATTN=ON" if [ -z ${GG_BUILD_AMDGPU_TARGETS} ]; then echo "Missing GG_BUILD_AMDGPU_TARGETS, please set it to your GPU architecture (e.g. gfx90a, gfx1100, etc.)" exit 1 diff --git a/ggml/src/ggml-cuda/ggml-cuda.cu b/ggml/src/ggml-cuda/ggml-cuda.cu index 561ab7ac5..f4b271146 100644 --- a/ggml/src/ggml-cuda/ggml-cuda.cu +++ b/ggml/src/ggml-cuda/ggml-cuda.cu @@ -4033,7 +4033,11 @@ static void ggml_cuda_graph_evaluate_and_capture(ggml_backend_cuda_context * cud continue; } #ifndef NDEBUG - assert(node->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device)); + // On integrated GPUs (APUs, e.g. RDNA3.5) the scheduler may place a + // node's output on the host-visible buffer, which the compute path + // handles. Allow that here, mirroring the src-tensor check below. + assert(node->buffer->buft == ggml_backend_cuda_buffer_type(cuda_ctx->device) || + (integrated && ggml_backend_buft_is_cuda_host(node->buffer->buft))); for (int j = 0; j < GGML_MAX_SRC; j++) { if (node->src[j] != nullptr) { assert(node->src[j]->buffer); diff --git a/tests/test-backend-sampler.cpp b/tests/test-backend-sampler.cpp index 1165f46f0..e5ae634cd 100644 --- a/tests/test-backend-sampler.cpp +++ b/tests/test-backend-sampler.cpp @@ -1668,9 +1668,18 @@ static std::vector collect_tests_to_run(const std::st } } else { for (const auto & test : BACKEND_TESTS) { - if (test.enabled_by_default) { - selected.push_back(&test); + if (!test.enabled_by_default) { + continue; + } +#ifdef GGML_USE_HIP + // TODO: remove this when https://github.com/ggml-org/llama.cpp/pull/26592 is merged + if (test.name == "penalties" || test.name == "set_sampler" || + test.name == "mixed" || test.name == "top_p") { + fprintf(stderr, "Skipping test '%s' on HIP backend (no backend TOP_K support)\n", test.name.c_str()); + continue; } +#endif // GGML_USE_HIP + selected.push_back(&test); } } diff --git a/tests/test-llama-archs.cpp b/tests/test-llama-archs.cpp index eb90798b6..0e29d221b 100644 --- a/tests/test-llama-archs.cpp +++ b/tests/test-llama-archs.cpp @@ -437,6 +437,14 @@ static bool arch_supported(const llm_arch arch) { } #endif // GGML_USE_WEBGPU + // FIXME: jamba produces incorrect output (~0.55 NMSE vs CPU) on the HIP + // backend on RDNA3.5 (gfx1151); the SSM kernels need investigation. +#ifdef GGML_USE_HIP + if (arch == LLM_ARCH_JAMBA) { + return false; + } +#endif // GGML_USE_HIP + return true; }