examples : Remove leading space from txt output (#3921)
* Fix #587: Remove leading space from txt output
The BPE tokenizer used by Whisper produces tokens with leading spaces,
causing each line in the txt output to start with an unwanted space.
This fix strips leading whitespace (spaces and tabs) from each segment
when writing to txt output files, improving the readability of the
transcription output.
CUDA is compiled with fast math and AMD/HIP is not — this flag lets AMD use fast math too.
We can't use -ffast-math: it implies -ffinite-math-only, which won't compile (ggml uses INFINITY for masking) and produces NaNs. -funsafe-math-optimizations gives the speedup without the NaN problems.
cuda: align snake fusion matcher with the other backends (llama/25460)
* cuda: fix snake fusion type predicate, a and inv_b are F32
The matcher required a->type == x->type while launch_snake reads both
as const float *, matching the CPU and Metal contract where a and inv_b
stay F32. F16/BF16 chains never fused and fell back to the naive path,
and a hypothetical all F16 chain would have read F16 bits as float.
Aligns the predicate and the comment with ggml-cpu.c
* cuda: reject snake fusion on non-contiguous operands
The kernel reads x[idx] and a[c] / inv_b[c] linearly, so a
non-contiguous view passing the matcher would silently read wrong data.
Mirror the contiguity guard already present in the CPU, Vulkan and
Metal matchers.
Hongqiang Wang [Wed, 8 Jul 2026 22:52:21 +0000 (15:52 -0700)]
opencl: Q6_K GEMM/GEMV fix for ne01 of weights that are not multiples of 128. (llama/25464)
* opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno
Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number.
Route Q6_K dense mul_mat with ne01 % 128 != 0 off the noshuffle path:
decode (ne1==1) uses the correct flat GEMV and the matching GEMM (ne1>1)
falls back to CPU (the flat convert has no verified small-batch GEMM kernel
for these shapes). All standard hidden/FFN/vocab dims are multiples of 128
and keep the noshuffle path.
* opencl: reserve alignment slack for the SOA subbuffer carve in alloc size
set_tensor carves quantized weights into per-component subbuffers (d/q,
ql/qh/s/d, ...) whose origins are each rounded up to the device base
address alignment. When a component's size is not a multiple of the
alignment, the carve extends past ggml_nbytes(tensor) and the last
subbuffer overlaps the next tensor in the pool -- e.g. q6_K [1536, 49155]:
size_s = 49155*96 ends 32 bytes past a 128-byte boundary, so the d
subbuffer ends 96 bytes past the tensor's allocation, and whichever of the
two neighboring tensors is uploaded last silently corrupts the other (here:
the last vocab rows' block scales). This affects any quant type whose
component sizes can be misaligned, on any shape with ne01 not a multiple of
the alignment granularity; standard power-of-two dims are unaffected.
Implement get_alloc_size for the OpenCL buffer type and reserve the
worst-case carve slack (4 aligned gaps; 5 components max, q5_K) for
quantized tensors. Costs at most 512 bytes per quantized tensor at the
observed 128-byte alignment.
* opencl: use lm based q6_k mm when ne1 is not multiple of 128
The MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles; at low
tokens-per-expert most tiles are mostly padding. When a tile's upper 16 slots
are all padding (router index 0xFFFFFFFF), skip the second dotx16_reduce8 half.
Numerically identical (skipped lanes are padding). Applied to all eight *_f32_ns
MoE GEMMs; default on, opt out with GGML_OPENCL_MOE_RAGGED_FP16=0.
Replace the two half-tile dotx16_reduce8 calls in the 8 *_f32_ns MoE GEMMs with
four dotx8_reduce4 (8-column) calls, skipping each empty trailing skip-group
independently. Padding is always trailing, so the kernel rounds the valid count
up to the skip granularity and skips fully-padding groups. Byte-identical to the
non-skipped path. New env GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32} (quarter/half/
off); default quarter.
* Merge old mul_mat patterns into the lane-based approach
* Enable fusion for MoE in shared MMVQ
* Restrict scale_view_nodes, enroll MM + ADD into lane-matcher
* Refactor mmvq loads, still does not help non-nvfp4 kernels
* Restrict scale-fusion to NVFP4
This is necessary, as the prolog is quite heavy in GEMV for some
quants/model configs, leading to net perf regression.
We should really be looking to refactor this such that ratio of
prologue/hot-loop/epilogue is better on the hot-loop
front:
This is required so we correctly test fusion of NVFP4.
* Update ggml/src/ggml.c
Co-authored-by: Johannes Gäßler <redacted>
* Add 2nd context for weights as suggested by @JohannesGaessler
This reflects more natural use of ggml compared to artifically
pre-allocating weights into the same context
* Exclude fused tests from gradient mode
I'm unsure of the current state, but naively every fusion pattern
should require its own backpropagation implementation. I don't see these
implemented for the CUDA backend, so we can disable tests to avoid
triggering GGML_ASSERT for
-ffast-math implies -ffinite-math-only under ROCm/clang 22, which
disables INFINITY/NaN and triggers -Wnan-infinity-disabled (errors
under -Werror in CI). Re-enable infinity handling without dropping
the rest of fast-math.
Hongqiang Wang [Tue, 7 Jul 2026 02:57:52 +0000 (19:57 -0700)]
opencl: general flash attention decode performance optimizations (llama/25366)
* opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV
* opencl: improve non FA KQ mv kernels
* opencl: tweaks for multiquery FA
* opencl: some tweaks for FA q1 kernels
* opencl: FA with DK=DV=512 for gemma-4
* opencl: various fixes
* opencl: cleanup
* opencl: fix FA decode crash for DK=512 (gemma-4)
The DK=512 decode-only program does not create the f32_f16 prefill
kernel, so the compiled check in ensure_fa_variant never hit and
supports_op gave inconsistent answers for the same op. block_n is also
unset for DK=512 decode; guard it to avoid an out-of-range read at
dispatch.
* opencl: run DK=512 FA decode on CPU
DK=512 decode is bandwidth-bound and faster on the CPU than the GPU,
increasingly so with depth. Decline it in supports_op; prefill stays on the GPU.
* opencl: compile MQ_GQA=8 FA kernels in a minimal program
The full program compiled with -D MQ_GQA=8 runs the Adreno compiler out
of memory at DK>=256. Only the vec_mq kernels are used from this
program, so compile it with FA_MQ_ONLY, which excludes everything else.
Also include the program name in the compile error log.
* opencl: remove stray token in flash_attn_f32_f16.cl
A stray "." broke the f32_f16 program build.
* opencl: split f16-KV FA decode finer (FD_KV_PER_SPLIT_F16)
The 2048 default under-fills the GPU on single-query f16-KV decode;
use 512 for f16 KV to get more splits. Quantized KV keeps 2048.
metal: add col2im_1d op (f32/f16/bf16) (llama/25176)
* metal: add col2im_1d op (f32/f16/bf16)
Gather kernel mirroring the CPU/CUDA path: each output (t_out, oc)
reads its ceil(K/s0) source columns with an F32 accumulator, a single
write and no atomics. One thread per output element, 256 per
threadgroup.
* metal: check dst contiguity and type match in supports_op for COL2IM_1D
Align the GGML_OP_COL2IM_1D predicate with the CPU, CUDA, and Vulkan
backends: the kernel writes dst with linear indexing and assumes the
same type as src0, so supports_op must also require a contiguous dst
and op->type == op->src[0]->type.
ggml-cpu: Enable tiled matmul on AIX (llama/25199)
The matmul_tiled path uses large local stack buffers for A_pack and B_pack. On AIX this can trigger a segmentation fault, so reduce the buffer footprint there to keep the tiled path usable.
Performance Impact:
~ 2x gains in PP_Speed for FP32, Q4_0 and Q8_0 models tested with llama-bench, llama-batched-bench and llama-cli.
Models used: Llama3.2 3b Instruct F32, qwen 2.5 3b Q4_0 and Q8_0
Tensor parallelism (-sm tensor) combined with -ncmoe (CPU-offloaded MoE
experts) aborts during warm-up on MoE models with
GGML_ASSERT(ggml_is_contiguous(tensor)) in ggml-backend-meta.cpp.
The failing tensor is the MoE router output (ffn_moe_topk): it is mirrored
(GGML_BACKEND_SPLIT_AXIS_MIRRORED, replicated across backends since routing
must be identical) and happens to be a non-contiguous view.
ggml_backend_meta_buffer_{get,set}_tensor asserted contiguity before
consulting the split state, so a mirrored non-contiguous tensor tripped the
assert even though the GGML_BACKEND_SPLIT_AXIS_MIRRORED case right below
already handles it.
Move the split-state lookup above the assert and allow the mirrored case in
both get_tensor and set_tensor.
cuda: enable topk-moe fusion for 288 experts (llama/25267)
* cuda: enable topk-moe fusion for 288 experts
The topk-moe fusion only accepted power-of-2 expert counts (or the
special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash)
fell back to the unfused per-layer routing chain: softmax/sigmoid,
argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that
is ~330 extra tiny graph nodes per token.
288 is a multiple of the warp size, so the existing kernel already
handles it; this adds the missing template instantiation and accepts
288 in the eligibility check.
Measured on gfx1151 with Step-3.7-Flash IQ4_XS (llama-bench,
-b 4096 -ub 4096 -fa 1 -dio 1 -ctk q8_0 -ctv q8_0; machine idle,
before/after paired so pp4096 stays matched as a load control):
Prompt processing is unaffected (the fusion only touches decode
routing). The decode gain is ~+2.4% at shallow context and fades with
depth: by 30k tokens each step is attention-bound over the KV cache,
so removing the fixed routing overhead is no longer visible.
Assisted-By: Claude Fable 5 <redacted>
* Update tests/test-backend-ops.cpp
Co-authored-by: Oliver Simons <redacted>
* Add comment for case 288 in topk-moe.cu
Remove redundant CUDA copies after gated_delta_net. (llama/23940)
* Remove redundant CUDA copies after gated_delta_net.
Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.
The change detects that gated_delta_net -> view -> cpy pattern and makes the CUDA GDN kernel write the state snapshot(s) directly into the recurrent cache, skipping the intermediate tail writes and copy kernels when safe.
Hongqiang Wang [Sat, 27 Jun 2026 22:36:06 +0000 (15:36 -0700)]
opencl: flash attention improvement (llama/25069)
* opencl: rework FA kernel for f16 and f32
* opencl: flash-attention prefill prepass kernels
- flash_attn_kv_pad_f16 pads the tail KV tile to a BLOCK_N multiple
- flash_attn_mask_pad_f16 pads the matching mask tile
- flash_attn_blk_f16 classifies each KV tile per query block as
fully masked / mixed / fully unmasked, so
the main kernel can skip fully-masked tiles
and the mask lookup for fully-unmasked ones
* opencl: FA kernels for q4_0 and q8_0
* opencl: `set_rows` for f32 to q8_0/q4_0
* opencl: dequant kernels for q4_0 and q8_0
* opencl: add FA tile tuning table with override
* opencl: wire host side for FA
* opencl: q4_0 MoE tensors are also SOA'ed
* opencl: cosmetic fix
* opencl: refactor, also clarify some code paths in comments
Gaurav Garg [Sat, 27 Jun 2026 12:16:21 +0000 (17:46 +0530)]
Added a cudaMemcpy2DAsync fast path to ggml_cuda_cpy (llama/25057)
* [CUDA] Added a cudaMemcpy2DAsync fast path to ggml_cuda_cpy
Add a CUDA ggml_cpy fast path for same-type, same-shape strided copies that are just 2D pitched block copies.
When tensors are not fully contiguous but each row is contiguous, it now uses cudaMemcpy2DAsync instead of the slow element-wise scalar copy kernel.
This fixes the GDN recurrent snapshot update with -np 4, where rollback slots are separated by cache stride gaps.
* Add new tests that execute the new optimized strided copy path
* Return unsupported for strided copy in OpenVINO, as new tests are failing
Andreas Kieslinger [Fri, 26 Jun 2026 14:18:30 +0000 (16:18 +0200)]
sched : reintroduce less synchronizations during split compute (llama/20793)
* CUDA: Improve performance via less synchronizations between token (llama/17795)
* Adds CPU-to-CUDA copy capability to
ggml_backend_cuda_cpy_tensor_async()
* Adds function to relax sync requirements between input copies on
supported backends (CUDA for now)
* Exchanges synchronous copy with async copy function.
* Adds macro guards to allow compilation in non-CUDA builds
* Reworked backend detection in ggml-backend.cpp to avoid linking
conflicts
* Relax requirement of checks in async CUDA copies from backend and buffer type to just buffer type, to avoid linking issues
* Minor cleanup
* Makes opt-in to relax use of explicit syncs more general. Backends like
vulkan which require a synchronization between HtoD copies and graph
execution could also adopt this change now.
* Reintroduces stricter check for CPU->CUDA backend async copy via
GGML_DEVICE_TYPE_CPU.
* Corrects initialization of ggml_backend_sync_mode in
ggml_backend_sched_split initialization
* Simplifies synchronizations to adhere to `saaasg` pattern.
* Apply suggestion from @ggerganov (src->buffer to buf_src)
Co-authored-by: Georgi Gerganov <redacted>
* Apply suggestion from @ggerganov (src->buffer to buf_src) v2
Co-authored-by: Georgi Gerganov <redacted>
---------
Co-authored-by: Georgi Gerganov <redacted>
* Apply suggestions from @johannesgaessler code review
Co-authored-by: Johannes Gäßler <redacted>
* Adds single-GPU synchronizations to multi-GPU settings to fix hip backend pipeline parallel bugs.
* Scheduler Hardening: Exclude hip/MUSA from copy_from_host CPU split ->
GPU split optimization
* Scheduler Hardening: Re-adding original additional synchronizations for
non-async backends
* Adds disclaimer to hip/musa exclusion of copy_from_host. Highlights that it is out of
precaution, but that no perf-impact is visible, and that it can be
revisited separately anytime.
---------
Co-authored-by: Georgi Gerganov <redacted> Co-authored-by: Johannes Gäßler <redacted>
Lin Xiaodong [Wed, 1 Jul 2026 11:10:16 +0000 (19:10 +0800)]
whisper : expose internal VAD speech segments (#3916)
When transcribing with params.vad = true, whisper already computes the speech
segments and keeps them in the state. Expose them so callers can reuse those
boundaries (for example to align or clip subtitles to speech) instead of running
a second, separate VAD pass.
Times are on the original audio timeline in centiseconds; the count is 0 when VAD
was not used. test-vad-full.cpp checks the segments are ordered and non-empty.
Daniel Bevenius [Wed, 1 Jul 2026 07:04:21 +0000 (09:04 +0200)]
ci, examples : add package-lock.json to examples addon node (#3919)
* ci, examples : add package-lock.json to examples addon node
This commit commits the package-lock.json file for examples/addon.node.
This is needed to make the CI workflow work properly and after we
started pinning the action versions in the workflow, the CI started
failing because the package-lock.json was not committed. At least that
is what I think is happening.
* ci: use node 20.x and 22.x instead of 16.x and 18.x
* fix CMakeLists.txt target properties for addon.node
Lin Xiaodong [Wed, 1 Jul 2026 06:23:50 +0000 (14:23 +0800)]
whisper : map token timestamps to original time when VAD is enabled (#3910)
whisper_full_get_token_data().t0/t1 are in the VAD-processed timeline (silences
removed), so they don't line up with the original audio. Add
whisper_full_get_token_t0/t1 that map them back.
A token inside a speech segment is interpolated within it; a token that falls in a
removed inter-segment silence snaps to the nearest boundary, so it never lands in
the middle of a cut-out gap. Without VAD the raw times are returned unchanged.
Lin Xiaodong [Wed, 1 Jul 2026 06:21:35 +0000 (14:21 +0800)]
whisper : make voice_length() utf-8 aware for CJK (#3915)
* whisper : make voice_length() utf-8 aware for CJK
voice_length() weights each token by how long its text takes to say, which drives
how a segment's time is shared between its tokens. It looped over raw bytes, so
every CJK character (3 bytes) was counted ~3x and full-width punctuation never
matched, skewing token timestamps for Chinese/Japanese.
Decode one utf-8 code point at a time and give full-width ，。！？ etc. the same
weights as their ASCII counterparts. Pure-ASCII text is unaffected.
* whisper : one statement per line in voice_length()
Daniel Bevenius [Tue, 30 Jun 2026 15:47:22 +0000 (17:47 +0200)]
ci : set GGML_NATIVE=OFF for sanitize builds (#3920)
This commit sets GGML_NATIVE=OFF for all sanitize builds to avoid
the case where the compilation happens using a runner that support
instructions that may not be available on all runners leading to illegal
instruction and failed CI runs.
Tarek Dakhran [Fri, 26 Jun 2026 07:41:56 +0000 (09:41 +0200)]
ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32 (llama/24699)
* ggml-cpu: fix SVE leftover path in ggml_vec_dot_f32
2D convolutions with kernel size 9 produced different results on SVE
enabled ARM devices. After debugging it turned out that ggml_vec_dot_f32
was using data from inactive lanes.
Use svmla_f32_m(pg, sum1, ax1, ay1) so inactive lanes retain sum1.
David Spruill [Thu, 25 Jun 2026 05:35:21 +0000 (01:35 -0400)]
sycl : support --split-mode tensor (llama/24152)
* Sycl tp stage1 (llama/1)
* SYCL: tensor parallelism (--split-mode tensor) for dual-GPU
Adds the comm_init/comm_free/comm_allreduce_tensor trio that the
meta-backend queries via get_proc_address to enable backend-specific
all-reduce, mirroring the pattern used by ggml-cuda.cu.
For N=2 (the common dual-GPU case) implements a degenerate ring
all-reduce with two size-branched paths:
* Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel
chained via depends_on(memcpy_event). 4 SYCL submissions/call.
* Large (nelem >= 32768): BF16-compressed. Each device compresses
FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's
inbox (HALF the PCIe bytes), then decompresses + adds into the
local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved
-- wins for any tensor where PCIe dominates kernel time.
Threshold and BF16 path pattern mirror the CUDA NCCL allreduce.
Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes
(matches both path layouts: FP32 nelem floats; BF16 outbox+inbox =
2 * nelem uint16_t each). Single alloc+free per device keeps the
SYCL pool's strict-LIFO invariant trivial.
Initial impl handles N=2 FP32 contiguous tensors. Other cases return
false, causing the meta-backend to use its generic butterfly fallback.
Per-call sync is intentionally omitted. SYCL in-order queue semantics
ensure that the meta-backend's next compute on the same per-device
queue waits for our final ADD, and the next allreduce's first op on
the same persistent buffer waits via the same queue. Only comm_free
does an explicit final wait.
OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process
in communicator_impl.hpp:47 (condition devices.size() == 1), which is
incompatible with llama.cpp's single-process multi-GPU model.
Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 +
DPC++ nightly):
Llama-3.3-70B in SYCL TP now comfortably beats production layer mode
on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on
both — the BF16 path is what unlocks the many-medium-allreduces
prefill pattern.
Build/CMake: no changes. No new dependencies. ~210 lines added across
ggml-sycl.h and ggml-sycl.cpp.
* Fix comments
* documentation update to address PR feedback
* Bring over my device-to-device memcpy chagnes
* move the dev2dev_memcpy calls to the upstream 7-parameter variety
liminfei-amd [Wed, 24 Jun 2026 09:42:03 +0000 (17:42 +0800)]
vulkan: fail the build when a shader fails to compile (llama/24450)
* vulkan-shaders-gen: fail the build when a shader fails to compile
vulkan-shaders-gen did not detect shader-compile subprocess failures, so a
broken libggml-vulkan could be produced while the build reported success and
the breakage only surfaced at run time. execute_command() discarded the child
exit code (POSIX waitpid passed nullptr for status; the Windows branch never
called GetExitCodeProcess) and string_to_spv decided success only from whether
stderr was empty, so a non-zero exit with empty stderr, or a subprocess that
failed to launch, was treated as success.
Return the child exit code from execute_command() (WEXITSTATUS on POSIX,
GetExitCodeProcess on Windows), treat a non-zero exit or non-empty stderr or a
launch exception as a failure, and record it in an atomic flag. main() checks
the flag after process_shaders() and returns EXIT_FAILURE before writing the
output files, so the build stops instead of emitting a broken backend.
Fixes #24393
Signed-off-by: liminfei-amd <redacted>
* vulkan-shaders-gen: simplify compile_failed access and drop unreachable return
Address review feedback on #24450:
- Access the std::atomic<bool> compile_failed directly (= / implicit bool)
instead of .store()/.load(); the flag stays atomic because the worker
threads in process_shaders() set it concurrently.
- Remove the unreachable trailing return -1 in execute_command(): on POSIX the
child _exit()s after execvp and the parent returns (fork()<0 throws); on
Windows the block returns the exit code.
overview / pkg / ggml / sources / whisper.cpp / log