ui: Add a "Default" option for the reasoning selector (#25846)
* ui: add Default reasoning option that defers to the server
The webui always injected enable_thinking, overriding the chat template
default and the --reasoning flag, breaking models that reason
unconditionally (e.g. Gemma 4 E4B) on a fresh client.
Default sends nothing so the server decides, Off and effort levels
force the value as before. All choices are remembered.
Also remove the boolean thinking API from the conversations store and
drop ChatFormReasoningEffortSubmenu.svelte (dead code).
* ui: close the whole menu tree on reasoning level selection
The reasoning levels were raw buttons inside the SubContent, so
selecting one only closed the submenu via manual state while the root
dropdown stayed open. DropdownMenu.Item closes the full tree on select
like the sibling entries and brings native keyboard navigation.
* ui: prevent the add menu tooltip from flashing when the dropdown closes
feat(ui): add symbolic math support to JS sandbox via nerdamer (#25948)
* feat(ui): add symbolic math support to JS sandbox via nerdamer
Preload nerdamer (with decimal.js) in the sandboxed worker,
exposing the `nerdamer` global for symbolic computation:
simplify, expand, factor, diff, integrate, solve, laplace,
ilt, limit, partfrac, gcd/lcm, roots, coefficients, and more.
Mirrors the math.js integration pattern from the
feature/sandbox-symbolic-math branch, but uses nerdamer
for a lighter, more focused symbolic math engine.
* Clarify nerdamer usage in sandbox tool description
Updated the description of the sandbox tool to clarify usage of nerdamer.
* ui: build nerdamer sandbox prelude from vendored source
Replace the vendored all.min.js with the readable nerdamer-prime
source and its two bundled deps (big-integer, decimal.js), licenses
included. A vite plugin bundles and minifies them at build time with
the upstream esbuild flags, exposed as virtual:nerdamer and imported
lazily on first sandbox use. The vendors package.json pins commonjs
so the project level type: module does not break esbuild format
detection. The harness gains a CSP removing network egress from the
worker, and browser tests cover the prelude, exact arithmetic, the
fetch block and the timeout.
* feat(ui): make symbolic math (nerdamer) a user-toggleable setting
- Add SYMBOLIC_MATH_ENABLED setting key and registry entry (checkbox, default false)
- Convert SANDBOX_TOOL_DEFINITION to buildSandboxToolDefinition(includeSymbolicMath)
so the tool description includes/excludes nerdamer API docs dynamically
- Cache sandbox harness per variant ('nerdamer' / 'plain') for instant toggle
- Deprecate SANDBOX_TOOL_DEFINITION constant alias for backward compatibility
- Update tools store to pass symbolic math config into tool definition
* docs(ui): tell LLM to list nerdamer functions first, do not guess
* test(ui): enable symbolic math in sandbox tests via settingsStore config
common: infer the speculative type from the draft repo sidecars (#25989)
With -hfd pointing to a repo that ships mtp-/dflash-/eagle3- sidecars
and no --spec-type given, the draft resolved to a full model while the
sidecar was the intended draft.
When the speculative types are still at their default, discover the
sidecars of the draft repo, pick the first available following the
existing mtp > dflash > eagle3 priority, and set the corresponding
type, so this now works without any extra flag:
llama-server -hf repo:Q3_K_M -hfd repo:Q8_0
An explicit --spec-type disables the inference, and a draft repo
without sidecars keeps resolving to a full model as before.
ggml: enable PowerPC backend variants on AIX (#25983)
* ggml: enable PowerPC backend variants on AIX
Allow the PowerPC CPU backend variants to be built on AIX by extending the platform check in the CMake configuration. This reuses the existing PowerPC backend implementations without changing their behavior.
Also fix a missing semicolon in the PowerPC Q0 matmul implementation.
Device-side embedding lookups require GET_ROWS to handle the k-quants
used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without
it the backend rejects the op and the scheduler falls back to the host,
copying the full embedding matrix back on every token in single-device
graphs.
Factor the super-block dequantizers out of the dequantize_block kernels
in convert.cu into shared device functions in dequantize.cuh and reuse
them from a new k_get_rows_kq kernel : one thread block dequantizes one
(dst row, super-block) pair with the existing thread layouts, 32 threads
for q4_K and 64 for the other k-quants.
Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left
as a TODO.
* cuda: add i-quant support to GET_ROWS
Extends the shared super-block dequantizers to the nine i-quants and
reuses them from k_get_rows_kq with the 32-thread layout of the matching
convert.cu kernels. supports_op gates the k-quant and i-quant path on
ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its
own (QK4_NL sub-blocks). mxfp4 is left as a TODO.
* cuda: add mxfp4 support to GET_ROWS
Moves the mxfp4 dequantizer into the shared super-block helpers and
reuses it from k_get_rows_kq with the 32-thread layout of the matching
convert.cu kernel. mxfp4 joins the ne0 % QK_K gate in supports_op since
its 32-value sub-blocks do not guarantee QK_K-aligned rows on their own.
This closes GET_ROWS type coverage on CUDA: every quantized GGML type
now takes the direct device path.
* cuda: gate the GET_ROWS row size only for 32-value sub-block types
Address review from @pwilkin: the i-quant commit replaced the return
shared by the whole supported type cascade, so f16/f32/bf16/i32 and the
legacy quants also inherited the ne0 % QK_K == 0 gate and any row size
that is not a multiple of 256 fell back to the scheduler. Split the
cascade: unconditional support is restored everywhere, the gate stays
only on iq4_nl and mxfp4 whose 32-value sub-blocks do not guarantee the
QK_K super-blocks the kernel iterates on.
Gerben van V [Tue, 21 Jul 2026 21:58:34 +0000 (23:58 +0200)]
mtmd : use align_corners for qwen3vl vision position embedding interpolation (#25781)
The Qwen3-VL learned position embedding is interpolated to the runtime patch
grid with the default bilinear+antialias (align_corners=False) sampling, while
the transformers reference uses align_corners=True (torch.linspace(0, side-1, T)).
The mismatch scales grounding coordinates about the image center, growing with
image size and per-axis for non-square images (see #16880).
common: resolve draft repo to its requested sidecar (#25955)
With -hfd pointing to a repo shipping speculative sidecars, the draft
resolved to the main model of that repo, since find_best_model()
excludes sidecar files, and the explicit draft plan suppressed the
sidecar discovery on the -hf repo.
The draft plan already discovers its sidecars, they were just never
consumed. Wire them as the draft, following the fallback pattern of
the main plan, so this now works as expected:
server: return 400 instead of 500 on validation error with X-Conversation-Id (#25760)
* server: return 400 instead of 500 on validation error with X-Conversation-Id
set_req() attaches the spipe as soon as the header is present, before the request
body is parsed. When params validation throws, set_next() never runs and next_orig
stays empty, so on_complete() called it and crashed with std::bad_function_call,
turning the prepared 400 JSON into a generic 500.
on_complete() now treats an empty next_orig as "streaming never started" and evicts
the session installed by set_req(), so a failed request leaves nothing behind for
discovery or replay. This also covers valid requests that carry the header but do
not stream, which previously left an empty finalized session in the map until the
GC TTL.
* ui: do not send the backend_sampling placeholder
On a fresh profile the syncable settings hold the empty string placeholder meaning
"let the server decide". Every neighbor field goes through the hasValue() guard
that filters it, except backend_sampling, which sent the placeholder verbatim and
made every default settings completion fail validation.
Guard the field with hasValue() like its neighbors. hasValue(false) is true, so an
explicit false still reaches the server and the intent of #18781 (send both true
and false) is preserved. Only the placeholder is filtered.
Winston Ma [Tue, 21 Jul 2026 15:40:45 +0000 (23:40 +0800)]
vulkan: Refactor vk_queue to use per-instance mutexes and unique handles (#23570)
* Refactor vk_queue to use per-instance mutexes and unique handles
* integrates VK_KHR_internally_synchronized_queues, abstracting the queue submission into a polymorphic interface that completely bypasses host-side mutex locking when driver-side synchronization is supported
* fix compilation error
* fix duplicate pNext chain for VkPhysicalDeviceInternallySynchronizedQueuesFeaturesKHR
* add fallback defines for VK_KHR_internally_synchronized_queues
* add null checks for queues in vk_device_struct destructor
* use unique_ptr for outer queues to enforce exclusive ownership and optimize lifetime
* use static constexpr for eInternallySynchronizedKHR
* add lock guard to ggml_vk_create_aliased_queue for thread safety
* initialize sync_query_features.internallySynchronizedQueues to VK_FALSE
* reuse sync_query_features for internallySynchronizedQueues and simplify chaining
* refactor internallySynchronizedQueues detection
* fix internallySynchronizedQueues query guard
* use eInternallySynchronizedKHR constant
* fix self-referential alias for eInternallySynchronizedKHR
* use macro for eInternallySynchronizedKHR fallback
* fix internallySynchronizedQueues query timing in ggml-vulkan.cpp to prevent device creation mismatch
* reset sync_query_features.pNext before reusing in device creation chain, also removed the redundant second probe call
* refactor internally synchronized queues detection to use chained feature query and avoid redundant API calls
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <redacted>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <redacted>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <redacted>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <redacted>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <redacted>
* Update ggml/src/ggml-vulkan/ggml-vulkan.cpp
Co-authored-by: Jeff Bolz <redacted>
* rename sync_enable_features to internally_synchronized_queues_features
* queue_flags is still computed before has_internally_synchronized_queues is set
* fix trailing whitespace
* replace eInternallySynchronizedKHR macro with static constexpr
* preserve source queue semantics in single-queue aliased transfer queue
* vulkan: fix cmd_pool access via pointer for compute_queue unique_ptr
* vulkan: lock queue during debug label emission when not internally synchronized
CUDA: vectorize same-type get_rows with int4 copy (#25929)
k_get_rows_float did a scalar one-element-per-thread copy and recomputed the
row-invariant work (index load, fast_div_modulo, src/dst row pointers) for
every element. Hoist that out of the per-element loop, and add a vectorized
path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the
contiguous same-type (no-cast) case.
The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at
runtime on 16-byte alignment of the base pointers and all row strides and on
ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small
single-row gather can drop below the device CU count and regress; an
occupancy gate keeps those on the block-rich scalar path.
On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288)
drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan
backend, with no regression on the small conv-state gather; total get_rows
-27%. test-backend-ops GET_ROWS passes (47/47).
* feat: Replace conversation rename flow with unified AlertDialog component
* feat: Add radio group component and consolidate title generation settings
* refactor: Remove JS Sandbox global toggle and migrate legacy user state
* chore: Formatting
* refactor: Cleanup
Co-authored-by: Aleksander Grygier <redacted>
* refactor: Cleanup
* refactor: Marquee selection hook
* feat: UI improvements
* refactor: Bulk db operations
* fix: optimize bulk conversation deletion to handle ancestor chains
* refactor: remove pairedKey mechanism from settings system
* fix: remove redundant onclick handler from dialog cancel button
* chore: pin @lucide/svelte to exact version
* feat: Run JavaScript tool disabled by default
* fix: correct active conversation deletion tracking in bulk delete
* feat: improve shift-key multi-selection support in sidebar via keyboard
* refactor: Retrieve JS Tool enabling via Developer Settings
* nits: sync, dialog wording, cycle guard, and lockfile follow-ups
- Restore titleGenerationUseLLM registry entry so it syncs across devices again
- Mention fork cascade in the bulk delete confirmation dialog
- Clear newParent on cycle guard break so children never point at a deleted conversation
- Align @lucide/svelte in package-lock.json with the exact pin in package.json
ui: fix collapsed user bubble with markdown rendering (#25869)
Edge paragraph margins are now zeroed at the source in
markdown-content.css, but the user bubble and the system message still
carried the -my-4 compensation for them. The uncompensated negative
margins shrank the wrapper 2rem below its content, collapsing
single-line user bubbles into a scrollable sliver and skewing the
system message expand threshold.
* ui: fix Show tool call in progress toggle ignored
The showToolCallInProgress setting was disconnected from the render
path during the agentic content rework: getDefaultExpanded() returns
a hardcoded false for tool call sections and an unconditional effect
auto-expands the currently executing tool call regardless of the
setting.
Drive default expansion of all tool call section types from the
setting and remove the now redundant auto-expand effect. Manual
toggling still takes precedence over the default in both directions.
* ui: rename Show tool call in progress to Always show tool call content
The previous name suggested symmetry with Show thought in progress,
which only applies while inference is running, but tool call content
stays expanded after completion. Rename the label, the settings key
and the syncable server key to alwaysShowToolCallContent. The synced
parameter never worked under its previous name so no migration is
needed.
ui: enable the agentic flow when only the JS sandbox is active (#25865)
The agentic gate counted MCP servers, builtin and custom tools but not
frontend tools, so with the JS sandbox as the only tool source the
agentic flow was skipped, no tools field reached the server and the
chat template rendered without the tool system prompt.
The sandbox is fully client-side: frontendTools derives from the
Developer settings toggle alone, counting it in the gate restores that
single source of truth.
llama-quant : exclude i32 ffn_gate_tid2eid routing table from quantization (#25787)
DeepSeek-V4's ffn_gate_tid2eid tensor is an i32 token-id -> expert-id
index table, not weights. It was never added to the name-based
exclusion list alongside ffn_gate_inp.weight, so llama-quantize tries
to quantize it and fails since i32 cannot convert to a float type.
Georgi Gerganov [Fri, 17 Jul 2026 09:15:30 +0000 (12:15 +0300)]
common : auto-download dflash- and eagle3- HF sidecars (#25811)
* common: auto-download dflash- and eagle3- HF sidecars
Mirror the existing mtp- sidecar logic to support auto-discovery and
download of DFlash (dflash-) and Eagle3 (eagle3-) speculative decoding
sidecars from Hugging Face repos.
Changes:
- Add --dflash and --eagle3 CLI flags to trigger sidecar download
- Add find_best_dflash() and find_best_eagle3() using find_best_sibling
- Exclude dflash- and eagle3- filenames from primary model selection
- Filter dflash- and eagle3- from cached model listings
- Wire download tasks that set speculative.draft.mparams as fallback
kleidiai: Add SME vs SME2 distinction in kernel dispatch (#25478)
The current integration treats SME as a single capability (CPU_FEATURE_SME)
with no distinction between SME(v1) and SME2. The kernels dispatched under
CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect
on SME(v1)-only hardware.
We introduce build-time and runtime distinction between SME and SME2, and
wire SME(v1) and SME2 kernels based on actual hardware support.
conversion: accept BitNetForCausalLM architecture name (#25769)
Microsoft BitNet Hugging Face configs use BitNetForCausalLM while the
converter only registered BitnetForCausalLM, causing conversion to fail
with "Model BitNetForCausalLM is not supported".
Register both spellings in TEXT_MODEL_MAP and the Bitnet model class.
ggml-cuda : restore prop.integrated on HIP builds (#24233)
PR #16308 set info.devices[id].integrated = false unconditionally for all
CUDA/HIP devices as a workaround for corrupted output on Jetson Orin
(#15034). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag is
needed: with the cached field forced to false, supports_buft() refuses
CUDA host buffers on AMD APU/UMA parts, while get_type() already reads
prop.integrated (#23007) — an inconsistency that breaks integrated-GPU
host-buffer use on ROCm.
Guard the workaround so it only applies to non-HIP (CUDA) builds and
restore prop.integrated for HIP, keeping the Jetson workaround intact for
CUDA.
For MoE gate/up projections the src1 activation is broadcast across the
routed experts (ne11 == 1), so ids_src1 maps every one of a token's
n_expert_used slots to the same physical row. The MMQ path therefore
re-quantized each token's activation n_expert_used times.
For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of
once per expert. For NVFP4 a single quantize+scatter kernel
(quantize_scatter_mmq_nvfp4) quantizes each token once and writes the
resulting block_fp4_mmq straight to all n_expert_used slots, using an
inverse token->compact-row map (build_tok2c). MXFP4, and
GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique
rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks).
Both are bit-identical to the previous gather-then-quantize path (identical
source data, deterministic per-block quantization), verified by
test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the
default, gather, and per-expert paths) and by coherent end-to-end
generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original
per-expert path.
Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192
(nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks):
activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused
quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The
fused path avoids materializing and re-reading the 8x compact buffer,
writing the expert copies directly from registers.
* CUDA: bounds-check token ids in build_tok2c_kernel
Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or
t >= n_tokens) and drop entries beyond n_expert_used per token instead of
writing past the token's tok2c region. No behavior change for valid MoE
routing data; test-backend-ops MUL_MAT_ID 790/790.
* Refactor the code based on review comments
- Removed previously added kernels that were not necessary anymore\
- Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows.
* Adding q8_1 support for dedup and addressing review comments
Hongqiang Wang [Wed, 15 Jul 2026 16:08:40 +0000 (09:08 -0700)]
opencl: fix two issues on flash attention for Adreno a7x (#25697)
* opencl: route `sub_group_shuffle_xor` to qcom ext when KHR ext is unavailable
KHR `sub_group_shuffle_xor` is not defined by compiler when
`cl_qcom_subgroup_shuffle` is present, causing certain FA
kernels fail to build. Define the KHR shuffle_xor using
the qcom extension.
* opencl: skip FA kernels with mixed and quant types for A7x to avoid compiler crash
server: fix read_file append_loc space breaking edit_file match (#25705)
read_file with append_loc emits "{n}\u2192 {line}". The space after the
arrow is meant as a separator, but it is indistinguishable from real
indentation. Models strip "{n}\u2192" yet keep the space, so the old_text
passed to edit_file carries a phantom leading space and never matches
(normalize_for_fuzzy_match trims trailing whitespace only, never leading).
Drop the separator space so the arrow abuts content: stripping "{n}\u2192"
now yields the exact line with its real indentation preserved, and the
failure mode cannot occur by construction. Update the description example
to match the new format.
ui: fix thinking menu never appearing in single-model mode (#25637)
In MODEL mode, modelPropsCache is never populated: fetchModelProps
call sites are gated on router-only state (isRouterMode checks,
routerModels always empty), so supportsThinking always reads an
empty chat template once a model is auto-selected.
Read serverStore.props.chat_template directly in non-router mode,
since the global /props already describes the single loaded model.
metal: fuse snake activation (mul, sin, sqr, mul, add) (#25459)
* metal: fuse snake activation (mul, sin, sqr, mul, add)
Mirror the CUDA, Vulkan and CPU snake fusion: same matcher on the naive
5-op chain, same F32 contract on a and inv_b, same F32/F16/BF16 kernel
with F32 compute. Follows the Metal backend idioms: bf16 instantiation
gated behind GGML_METAL_HAS_BF16 and concurrency ranges checked on the
remaining chain nodes before encoding, as done by the bin fusion.
Covered by the existing backend-agnostic SNAKE_FUSE tests.
* metal: absorb snake fusion into ggml_metal_op_bin
Extract the matcher to ggml_metal_op_can_fuse_snake, mirroring the
Vulkan naming, and dispatch the fused path from ggml_metal_op_bin.
The encode loop switch is back to a single call per case.
Address review from ggerganov
* metal: fix indentation in ggml_metal_op_can_fuse_snake
sycl: Increase minimum buffer size for USM system allocations (#25525)
Raise the threshold for minimum buffer size from 1 GiB to 4 GiB, based
on real-world experiments of overcommitting device memory with model
weights larger than available VRAM, for example Qwen3.5-35B-A3B-Q8
running on a B70.
Also add a debug message to better track USM system allocations.
[SYCL] Flash Attention with XMX engine via oneDNN (#25222)
* [SYCL] F16 (default) Flash Attention with XMX engine via oneDNN graph API; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k
* [SYCL] Address review on FA oneDNN path. Result: llama-bench---pp512; 32% increase with fa1; llama-perplexity---0.11% difference; tested model: mradermacher/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf
* PR-25222 revision v2: addressed audits
* [SYCL] flash-attn oneDNN SDPA KV F16 rev 3.0: add BMG gate + multi-device sync. Narrow the scrope of this PR to Battlemage only (bmg; Xe2). Other archs (e.g., alchemist) fall back to existing FA kernel. When device_count >1, apply stream -> wait_and_throw(), validated working path for multi-gpu sync fix by @maxious.
Co-authored-by: maxious <redacted>
* updated comment on bmg gate, noted the issue
Hongqiang Wang [Tue, 14 Jul 2026 19:27:56 +0000 (12:27 -0700)]
opencl: avoid the vec path in GEMV for unaligned row stride (#25671)
The f16 GEMV kernels take a vectorized path for ne00 >= 128 that casts the row
pointers to half4 or float4. When the row stride is not aligned, the wide load
becomes misaligned. On devices that require natural alignment for vector loads,
the kernel reads garbage. This is the case Intel GPUs and the kernels produce
incorrect results there. Adreno happpens to be byte addressable and the kernels
happen to work.
overview / pkg / ggml / sources / llama.cpp / log