Hongqiang Wang [Wed, 8 Jul 2026 22:52:21 +0000 (15:52 -0700)]
opencl: Q6_K GEMM/GEMV fix for ne01 of weights that are not multiples of 128. (#25464)
* opencl: fix garbled output for Q6_K weights with ne01 % 128 != 0 on Adreno
Observed with granite-3.1-3b-a800m-instruct, whose vocab is an odd number.
Route Q6_K dense mul_mat with ne01 % 128 != 0 off the noshuffle path:
decode (ne1==1) uses the correct flat GEMV and the matching GEMM (ne1>1)
falls back to CPU (the flat convert has no verified small-batch GEMM kernel
for these shapes). All standard hidden/FFN/vocab dims are multiples of 128
and keep the noshuffle path.
* opencl: reserve alignment slack for the SOA subbuffer carve in alloc size
set_tensor carves quantized weights into per-component subbuffers (d/q,
ql/qh/s/d, ...) whose origins are each rounded up to the device base
address alignment. When a component's size is not a multiple of the
alignment, the carve extends past ggml_nbytes(tensor) and the last
subbuffer overlaps the next tensor in the pool -- e.g. q6_K [1536, 49155]:
size_s = 49155*96 ends 32 bytes past a 128-byte boundary, so the d
subbuffer ends 96 bytes past the tensor's allocation, and whichever of the
two neighboring tensors is uploaded last silently corrupts the other (here:
the last vocab rows' block scales). This affects any quant type whose
component sizes can be misaligned, on any shape with ne01 not a multiple of
the alignment granularity; standard power-of-two dims are unaffected.
Implement get_alloc_size for the OpenCL buffer type and reserve the
worst-case carve slack (4 aligned gaps; 5 components max, q5_K) for
quantized tensors. Costs at most 512 bytes per quantized tensor at the
observed 128-byte alignment.
* opencl: use lm based q6_k mm when ne1 is not multiple of 128
The MoE prefill GEMM groups tokens into TILESIZE_N=32 per-expert tiles; at low
tokens-per-expert most tiles are mostly padding. When a tile's upper 16 slots
are all padding (router index 0xFFFFFFFF), skip the second dotx16_reduce8 half.
Numerically identical (skipped lanes are padding). Applied to all eight *_f32_ns
MoE GEMMs; default on, opt out with GGML_OPENCL_MOE_RAGGED_FP16=0.
Replace the two half-tile dotx16_reduce8 calls in the 8 *_f32_ns MoE GEMMs with
four dotx8_reduce4 (8-column) calls, skipping each empty trailing skip-group
independently. Padding is always trailing, so the kernel rounds the valid count
up to the skip granularity and skips fully-padding groups. Byte-identical to the
non-skipped path. New env GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32} (quarter/half/
off); default quarter.
Oliver Simons [Wed, 8 Jul 2026 12:38:51 +0000 (14:38 +0200)]
Make hip quality check run on all changes (#25403)
Improvement of the CI to run on all hip-related changes as a follow-up to
https://github.com/ggml-org/llama.cpp/pull/25373
so breakage is more likely to be caught in future
server-stream: follow-up on SSE Replay Buffer (#23226) (#25047)
* server-stream : pimpl
* server-stream: prefix free functions with server_stream_
address review from ggerganov: scope the public stream functions under the
server_stream_ prefix, matching server_stream_session_manager_start/stop.
* server-stream: guard session and manager state with the mutex
address review from ggerganov: make done, completed_ts and the GC running flag plain members under their
mutex and set the condvar predicates under the lock. keep cancelled atomic for
the lock-free should_stop poll.
* server-stream: trim comments to the non-obvious
address review from ggerganov: drop comments that restate the code, keep the
concurrency, lifetime and ordering rationale. de-stale a few comments left by the
pimpl: g_stream_sessions is now internal and the /v1/streams listing is gone.
* server-stream: update dev docs for the pimpl and prefix
reflect server_stream_session_manager_start/stop and the server_stream_ prefix,
note the manager is now a file-static singleton hidden in the .cpp
* server-stream: move stream traces to debug level
keep the bring-up traces for diagnostics but off the default log: skip
drain, draining, drain ended, DELETE evict, attach_pipe, and the router
stream resume proxy.
* server-stream: align router stream resume proxy trace with upstream
the child-side bring-up traces are already SRV_TRC on master, move the
router stream resume proxy trace to the same level.
* server-stream: move stream_read_status enum to the cpp
it is only used by the hidden session and consumer types, so it belongs
with them behind the pimpl boundary, not on the public header surface.
* feat: Surface enabled-tools token count in context hover card
* refactor(tools): make toolsStore the sole owner of the OpenAI wire format
Previously mcpStore.getToolDefinitionsForLLM() owned the MCP->OpenAI
shape conversion (plus normalizeSchemaProperties). That created two
sources of truth for what gets sent to the LLM, with the
duplication-prone risk of the deduplicated enabled list (which feeds
the token-count cache) drifting from the bytes actually shipped on
chat.
Now:
- mcpStore: pure protocol state + routing. Drop getToolDefinitionsForLLM
and the inline OpenAIToolDefinition conversion + normalizeSchemaProperties.
Doc comment adjusted to declare wire-format ownership as belonging
to toolsStore. Connection lifecycle, health checks, executeTool,
and the connections/toolsIndex remain.
- toolsStore: owns the wire shape (added earlier this series). mcpEntries()
inlines the MCP tool conversion; uses normalizeJsonSchema (the JSON
Schema util extracted in the prior commit) so missing 'type' fields
are inferred from defaults. mcpTools getter iterates mcpEntries() so
the Settings UI and the deduplicated enabled list see the same
definitions. getEnabledToolsForLLM iterates mcpEntries() instead of
calling mcpStore, so the JSON sent to the LLM is identical to what
toolsStore.refreshEnabledToolsTokenCount tokenizes.
- agentic: the chat-completion tools field's type was annotated as
ReturnType<typeof mcpStore.getToolDefinitionsForLLM>, claiming the
shape was owned by mcpStore. Switch to ReturnType<typeof
toolsStore.getEnabledToolsForLLM>, the actual source.
Assisted-by: Claude
* feat: UI WIP
* feat: UI WIP
* feat: UI WIP
* feat: Adjust reasoning submenu layout and spacing
* feat: Adjust context usage gauge thresholds and styling
* feat: Split context usage gauge stats into current and cumulative breakdowns
* chore: Format
* refactor: Cleanup
* refactor: Cleanup
* feat: improve token gauge accuracy and display
* refactor: remove MCP recommendation gating and simplify server visibility
* feat: add token audit logging to ChatStore for debugging
* refactor: Simplify context token reading to use server promptTokens directly
* feat: Replace last-known token tracking with live server-derived stats for accurate streaming gauges
* feat: UI Improvements
* feat: Move prompt processing stats to the preceding user message
* feat: Fix context token double-counting and refine gauge layout
* refactor: remove always-show-agentic-turns setting and simplify agentic turn display
* feat: track and display cache tokens in context gauge
* feat: add diagnostic logging for chat completion requests
* refactor: improve token audit console output with fresh/cached breakdown
Georgi Gerganov [Wed, 8 Jul 2026 07:00:03 +0000 (10:00 +0300)]
llama-eval : fix crash when answer is None in HTML dump (#25435)
dict.get("key", default) returns None (not default) when the key
exists but its value is explicitly None. This caused an AttributeError
in _escape_html() when a task errored before grading and answer was
set to None.
* Merge old mul_mat patterns into the lane-based approach
* Enable fusion for MoE in shared MMVQ
* Restrict scale_view_nodes, enroll MM + ADD into lane-matcher
* Refactor mmvq loads, still does not help non-nvfp4 kernels
* Restrict scale-fusion to NVFP4
This is necessary, as the prolog is quite heavy in GEMV for some
quants/model configs, leading to net perf regression.
We should really be looking to refactor this such that ratio of
prologue/hot-loop/epilogue is better on the hot-loop
front:
This is required so we correctly test fusion of NVFP4.
* Update ggml/src/ggml.c
Co-authored-by: Johannes Gäßler <redacted>
* Add 2nd context for weights as suggested by @JohannesGaessler
This reflects more natural use of ggml compared to artifically
pre-allocating weights into the same context
* Exclude fused tests from gradient mode
I'm unsure of the current state, but naively every fusion pattern
should require its own backpropagation implementation. I don't see these
implemented for the CUDA backend, so we can disable tests to avoid
triggering GGML_ASSERT for
Alex [Tue, 7 Jul 2026 14:20:42 +0000 (10:20 -0400)]
server : fix draft model fit vs load inconsistency (#25056)
* fix: draft model fit vs load inconsistency
* refactor(server): unify draft/mtp parameter initialization, model, and context load
- moves speculative init to speculative.cpp
- changes server_context_impl model_dft and ctx_dft to use raw pointers
- fix: don't throttle progress callback when loading draft model
- refactor: rename draft model/ctx load method
Before this commit, --cache-ram was not a hard limit:
- The cache always kept at least one entry, even if that entry exceeded the
RAM/token limits.
- Old entries were only evicted for the RAM/token limits after saving the new
one, which could cause the cache to temporarily exceed the RAM/token limits
even if individual entries were below the limit.
Now, ensure that the RAM limit is strict with these changes:
- Skip saving state to cache if by itself it exceeds the RAM limit.
- Evict old entries as necessary to make the new entry fit.
Additionally, token-limit cleanup may now evict the last remaining cache entry
instead of always preserving one.
-ffast-math implies -ffinite-math-only under ROCm/clang 22, which
disables INFINITY/NaN and triggers -Wnan-infinity-disabled (errors
under -Werror in CI). Re-enable infinity handling without dropping
the rest of fast-math.
Hongqiang Wang [Tue, 7 Jul 2026 02:57:52 +0000 (19:57 -0700)]
opencl: general flash attention decode performance optimizations (#25366)
* opencl: vec flash-attention decode kernels for f16/q8_0/q4_0 KV
* opencl: improve non FA KQ mv kernels
* opencl: tweaks for multiquery FA
* opencl: some tweaks for FA q1 kernels
* opencl: FA with DK=DV=512 for gemma-4
* opencl: various fixes
* opencl: cleanup
* opencl: fix FA decode crash for DK=512 (gemma-4)
The DK=512 decode-only program does not create the f32_f16 prefill
kernel, so the compiled check in ensure_fa_variant never hit and
supports_op gave inconsistent answers for the same op. block_n is also
unset for DK=512 decode; guard it to avoid an out-of-range read at
dispatch.
* opencl: run DK=512 FA decode on CPU
DK=512 decode is bandwidth-bound and faster on the CPU than the GPU,
increasingly so with depth. Decline it in supports_op; prefill stays on the GPU.
* opencl: compile MQ_GQA=8 FA kernels in a minimal program
The full program compiled with -D MQ_GQA=8 runs the Adreno compiler out
of memory at DK>=256. Only the vec_mq kernels are used from this
program, so compile it with FA_MQ_ONLY, which excludes everything else.
Also include the program name in the compile error log.
* opencl: remove stray token in flash_attn_f32_f16.cl
A stray "." broke the f32_f16 program build.
* opencl: split f16-KV FA decode finer (FD_KV_PER_SPLIT_F16)
The 2048 default under-fills the GPU on single-query f16-KV decode;
use 512 for f16 KV to get more splits. Quantized KV keeps 2048.
Gather kernel mirroring the CPU/CUDA path: each output (t_out, oc)
reads its ceil(K/s0) source columns with an F32 accumulator, a single
write and no atomics. One thread per output element, 256 per
threadgroup.
* metal: check dst contiguity and type match in supports_op for COL2IM_1D
Align the GGML_OP_COL2IM_1D predicate with the CPU, CUDA, and Vulkan
backends: the kernel writes dst with linear indexing and assumes the
same type as src0, so supports_op must also require a contiguous dst
and op->type == op->src[0]->type.
server: fix deadlock in load_models() when erasing a finished download (#25358)
* server: fix deadlock in load_models() when erasing a finished download
The download monitoring thread acquires the models mutex on its way out,
but load_models() joined it from the erase loop while holding that mutex.
Join it outside the lock via threads_to_join like the other monitoring
threads.
* server: add default timeout to test requests
A hung server now fails the test after 10 minutes instead of stalling
the CI job for hours. Explicit timeouts are unchanged.
The matmul_tiled path uses large local stack buffers for A_pack and B_pack. On AIX this can trigger a segmentation fault, so reduce the buffer footprint there to keep the tiled path usable.
Performance Impact:
~ 2x gains in PP_Speed for FP32, Q4_0 and Q8_0 models tested with llama-bench, llama-batched-bench and llama-cli.
Models used: Llama3.2 3b Instruct F32, qwen 2.5 3b Q4_0 and Q8_0
Tensor parallelism (-sm tensor) combined with -ncmoe (CPU-offloaded MoE
experts) aborts during warm-up on MoE models with
GGML_ASSERT(ggml_is_contiguous(tensor)) in ggml-backend-meta.cpp.
The failing tensor is the MoE router output (ffn_moe_topk): it is mirrored
(GGML_BACKEND_SPLIT_AXIS_MIRRORED, replicated across backends since routing
must be identical) and happens to be a non-contiguous view.
ggml_backend_meta_buffer_{get,set}_tensor asserted contiguity before
consulting the split state, so a mirrored non-contiguous tensor tripped the
assert even though the GGML_BACKEND_SPLIT_AXIS_MIRRORED case right below
already handles it.
Move the split-state lookup above the assert and allow the mirrored case in
both get_tensor and set_tensor.
llama : add guard for K/V rotation input when buffer is unallocated (#25215)
llm_graph_input_attn_kv::set_input and llm_graph_input_attn_kv_iswa::set_input
call set_input_k_rot / set_input_v_rot whenever the rotation tensor pointer is
non-null, but the tensor's buffer can be unallocated (NULL) when a graph only
stores K/V without attending -- e.g. DFlash speculative decoding's KV-injection
pass. set_input_k_rot then calls ggml_backend_buffer_is_host() on a NULL buffer
and aborts with GGML_ASSERT(buffer).
Guard the four k_rot/v_rot inputs with the same "&& ->buffer" check that the
adjacent kq_mask inputs already use in these two functions. When the buffer is
unallocated there is no data to upload, so skipping is correct.
chat: trim messages sent to StepFun parser (fixes long reasoning loops) (#25238)
* chat: trim messages sent to StepFun parser (fixes long reasoning loops)
* add regression test; remove duplicate template
* chat: trim StepFun content parts before rendering
The StepFun trim workaround ran on the already-rendered messages, where
typed content parts have been concatenated into a single string, so the
per-part whitespace could no longer be reached. Move the trim ahead of
rendering and apply it to content_parts text as well as the string
content and reasoning_content. Adds a content-parts regression test.
Co-Authored-By: Piotr Wilkin <redacted> Assisted-By: Claude Fable 5 <redacted>
---------
Nick Towle [Fri, 3 Jul 2026 17:03:51 +0000 (10:03 -0700)]
ui: Improve performance when streaming (#25225)
* ui: Improve performance when streaming
* ui: build sibling info map in branching utils
Moves the node map and sibling map construction from the
.by block into buildSiblingInfoMap() in branching.ts.
The map is built once per structural change and only read
afterwards, so it does not need SvelteMap reactivity. Keeping
the construction in plain TypeScript fixes the
svelte/prefer-svelte-reactivity lint error and groups the
branching logic where it already lives.
cuda: enable topk-moe fusion for 288 experts (#25267)
* cuda: enable topk-moe fusion for 288 experts
The topk-moe fusion only accepted power-of-2 expert counts (or the
special-cased 576), so models with 288 experts (e.g. Step-3.7-Flash)
fell back to the unfused per-layer routing chain: softmax/sigmoid,
argsort, get_rows, sum_rows, div, clamp, scale. At batch size 1 that
is ~330 extra tiny graph nodes per token.
288 is a multiple of the warp size, so the existing kernel already
handles it; this adds the missing template instantiation and accepts
288 in the eligibility check.
Measured on gfx1151 with Step-3.7-Flash IQ4_XS (llama-bench,
-b 4096 -ub 4096 -fa 1 -dio 1 -ctk q8_0 -ctv q8_0; machine idle,
before/after paired so pp4096 stays matched as a load control):
Prompt processing is unaffected (the fusion only touches decode
routing). The decode gain is ~+2.4% at shallow context and fades with
depth: by 30k tokens each step is attention-bound over the KV cache,
so removing the fixed routing overhead is no longer visible.
Assisted-By: Claude Fable 5 <redacted>
* Update tests/test-backend-ops.cpp
Co-authored-by: Oliver Simons <redacted>
* Add comment for case 288 in topk-moe.cu
ui: align persisted config with strict server schema and enable thinking by default (#25242)
* ui: migrate legacy string-encoded booleans in persisted config
* ui: enable thinking by default
Fresh users and legacy conversations without a persisted thinking
preference now default to enabled. The per-conversation toggle and
the persisted localStorage choice keep taking precedence.
server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections (#25241)
* server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections
* server + ui: sse_ping_interval becomes a per-request body field
Address review from ngxson: the global default returns to 30 so API
clients see no behavior change, and the WebUI sends sse_ping_interval: 1
in the request body since it owns the 3s visibility-kick contract and
declares the cadence it needs. Positive values keep the existing > 0
gate, -1 keeps its disabled semantics.
* server: move sse_ping_interval into the request schema
Address review from ngxson: the field is now a typed field_num with
hard limits (-1, INT32_MAX) bound to task_params, seeded from the CLI
default alongside the other inherited parameters. The raw json_value
read and its redundant comment are gone, and schema evaluation brings
type and range validation for free.
Remove redundant CUDA copies after gated_delta_net. (#23940)
* Remove redundant CUDA copies after gated_delta_net.
Currently, GDN writes recurrent state snapshots into its output tail, then the graph immediately copies those snapshots into ssm_states_all. With MTP draft length 3, target decode uses K=4, so that becomes 4 extra ggml_cuda_cpy calls.
The change detects that gated_delta_net -> view -> cpy pattern and makes the CUDA GDN kernel write the state snapshot(s) directly into the recurrent cache, skipping the intermediate tail writes and copy kernels when safe.
Expose the model file type (quantization) name, e.g. "Q8_0" or
"Q4_K - Medium", through a new public C API. The returned pointer is
valid for the lifetime of the model and nullptr when the model is
invalid or the file type is unknown.
Signed-off-by: Adrien Gallouët <redacted>
* Move "(guessed)" to the front in llama_ftype_name
Prepend the "(guessed)" label instead of appending it. This allows removing
the non-thread-safe static std::string, making the function allocation-free.
Pascal [Tue, 30 Jun 2026 14:16:44 +0000 (16:16 +0200)]
common,server: handle bracketed IPv6 literals in URL authority (#25140)
* common,server: handle bracketed IPv6 literals in URL authority
Parse the [host]:port form (RFC 3986) and bracket IPv6 hosts when
formatting a URL authority: listening log, proxy Host header, proxy
log, client rebuild. The per-request remote_addr stays bare.
* common: restore unsupported scheme throw in url parser
Address @ngxson review: keep the explicit reject in port resolution so
the block stays self-contained. Non-http(s) schemes still throw (also
gated at the top of common_http_parse_url).
Aman Gupta [Mon, 29 Jun 2026 08:58:51 +0000 (16:58 +0800)]
DeepSeek V4 (#24162)
* convert: add dsv4 conversion
* add basic setup
* add llm_graph_input_dsv4
* add save-load state
* add sinkhorn eps - correction by @fairydreaming
* add rope fix
* cleanup dead code
* fix bugs
* support pro model: added by @fairydreaming
* remove redundant V cache
* Chat template
* remove debugging leftovers
* Add mechanism for inlining templates based on architecture
* s/deepseek-v4-flash/deepseek4/g
* s/deepseek-v4-flash/deepseek4/g continued
* enable graph reuse
* enable FA
* fix test llama archs
* rename
* compatibility with antirez ds4 GGUFs
* simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func.
* reserve worst-case kv-cache
* revert max split inputs
* address review comments
* add padding to enable FA
* pad only the final value of plan.n_kv to 256
* remove built-in cpp chat template
* cont: remove cpp built-in template
* rm outdated test
* replace ggml_view_3d() with ggml_reshape_3d()
Co-authored-by: Georgi Gerganov <redacted>
* only support n_seq=1 for now
* remove unused var
* cont: remove unused var
* use scale bias
* use correct ptr for can_reuse
* remove gen-chat-inline-templates.py
* simplify graph reuse
* cont: cleanup
* remove unused inputs
* enable partial checkpointing
* add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4
* precompute source_idx + add comment about dummy write
* support multi-seq
* remove restored_trim_pos
* use split_equal when possible
* fix indent
* address review comments
* use LLM_KV
* fix ci
---------
Co-authored-by: Piotr Wilkin <redacted> Co-authored-by: Stanisław Szymczyk <redacted> Co-authored-by: Xuan Son Nguyen <redacted> Co-authored-by: fairydreaming <redacted> Co-authored-by: Georgi Gerganov <redacted>
overview / pkg / ggml / sources / llama.cpp / log