Frosty40 [Tue, 16 Jun 2026 05:35:00 +0000 (00:35 -0500)]
sycl: support reordered Q4_K/Q5_K/Q6_K MoE MUL_MAT_ID (#24452)
* sycl: support reordered Q4_K and Q5_K MoE MUL_MAT_ID
Extend reordered-weight handling to fused MoE MUL_MAT_ID for Q4_K and Q5_K expert tensors and add Q5_K reordered DMMV coverage. Unsupported 3D reorder cases now fall back instead of aborting.
accept an optional leading type: function field in
build_json_tools_flat_keys so openai style tool calls parse on
templates whose serialization opens on the name field.
return a clean error and log the unparsed fragment on a final peg
parse failure instead of throwing the raw parser position and input.
keep the raw arguments string in func_args_not_string when it is not
valid json instead of aborting the prompt render.
* chat: surface peg-native parse failures
a final peg parse failure threw the raw parser position and input. log
the unparsed fragment and raise a clearer error instead, so a model
output that does not match the expected format no longer fails silently
with an empty assistant turn.
minimal change, no behavior change on successful parses.
* chat: handle openai style tool calls in peg-native
* nits
* common: scope OpenAI wrapper grammar trigger via autoparser flag
* chat: gate type:function parsing leniency on the analysis flag
Thread accept_openai_wrapper from the generator to build_json_tools_flat_keys
so the leading "type": "function" field is accepted only when openai_wrapper_trigger is set.
Pascal [Mon, 15 Jun 2026 06:11:36 +0000 (08:11 +0200)]
UI/svg block rendering (#24080)
* ui: add svg block visualizer based on allozaur's mermaid PR
* ui: rationalise diagram block styling and pre transforms shared by mermaid and svg
* ui: live render streaming svg blocks
* ui: also render svg authored in xml code fences
* ui: refactor svg block rendering, address review from allozaur
- Move the svg size ceiling and DOMPurify config out of sanitize-svg.ts into /constants.
- Rename the svg-diagram class to svg-block so the name no longer implies diagrams only.
- Replace the svg, xml and svg tag magic strings in the markdown pipeline with shared constants.
- Promote the data-svg-rendered marker and its sibling data attributes to constants.
* ui: render svg blocks in a shadow root for animation and live zoom
Mount each sanitized svg inside an open shadow root so author <style> and
keyframe or smil animations run while staying scoped to the host element.
Relax the sanitizer to forbid only foreignObject and script, which lets
animation, href and external resource refs through for wider compatibility.
Render the inline block and the zoom dialog from the same reactive source,
so a streaming svg keeps drawing live inside the open zoom popup.
Jeff Bolz [Sat, 13 Jun 2026 13:44:15 +0000 (08:44 -0500)]
vulkan: support non-contig unary/glu ops (#24215)
* vulkan: support non-contig unary/glu ops
Change unary/glu ops to pass in all strides and use fastdiv for the index
calculation. Put all unary ops in one file, similar to glu, to share the
code. codex went ahead and added expm1 without me asking, but I had to
make it do a real precision analysis rather than just making stuff up.
unary.comp initially couldn't use generic_unary_head because there wasn't
space for xielu's additional constants. Fixing this required packing the
fastdiv 'L' values.
Georgi Gerganov [Fri, 12 Jun 2026 14:59:56 +0000 (17:59 +0300)]
server : fix reasoning budget WebUI precedence over model.ini (#24517)
When reasoning-budget is set in model.ini, the per-request
thinking_budget_tokens from the WebUI was ignored because the
model.ini value took unconditional precedence.
Swap the precedence so the WebUI per-request value is checked
first, with the model.ini value serving as a fallback default.
Aleksander Grygier [Fri, 12 Jun 2026 13:53:26 +0000 (15:53 +0200)]
ui: PWA support (#23871)
* feat: Add basic PWA support and service worker for offline caching
* feat: Vite PWA implementation WIP
* feat: Improve PWA icons generation
* feat: Add PWA workbox to server routes
* feat: Include `version.json` in static assets
* feat: Add HTTP cache headers for PWA static assets
* feat: Update app name for `apple-mobile-web-app-title`
* feat: Implement PWA versioning and automatic update detection
* chore: Update `.gitignore` files
* feat: Splash Screens
* feat: Add dark mode favicon support
* refactor: Cleanup
* fix: Use dark logo for dark splash screens
* refactor: Simplify favicons SVG code
* fix: Adjust caching and polling for reliable service worker updates
* fix: Add missing favicon entry
* fix: Align PWA service worker configuration with SvelteKit build structure
* fix: Replace hashed bundle paths with versioned static paths
* test: Add PWA tests
* ci: Add build output for unit tests
* refactor: Cleanup
* fix: Server build & release versioning
* chore: Update package-lock.json
* chore: Increase PWA cache size
* chore: Update packages
* feat: Update favicons
* refactor: Post-merge fix
* feat: support explicit build version for PWA cache busting
* fix: CI
* feat: Improve PWA Refresh Alert UI
* feat: Add toggleable build version display
* refactor: Cleanup
* feat: Add version mismatch detection and manual app reload
* refactor: replace dynamic imports with static
* refactor: Cleanup
* feat: Add safe space for `pwa-<size>.png` rendered icons
* fix: use relative paths for PWA assets to support base path deployment
* feat: add PWA mode detection via URL query parameter
* feat: Use ?cache=true for SW-cached PWA assets
* refactor: Build process cleanup
* refactor: Decouple PWA versioning and remove ?cache=true workaround
* chore: Update README logo
* feat: Include PWA Assets generation in build script
* refactor: `usePwa` hook for core layout
* fix: Relativize base vite plugin
* fix: remove unnecessary backslash escapes in test regexes
* test: update static asset paths for API Key test
* refactor: Move SvelteKit PWA Options config to constants
* ui: fix update notification never appearing
Keep the PWA hook object intact instead of destructuring needRefreshByStorage,
which freezes the reactive getter. Also exclude loading.html from PWA
precache to prevent 404 errors and broken SW installation.
Pascal [Fri, 12 Jun 2026 08:20:27 +0000 (10:20 +0200)]
UI/jpeg exif orientation (#24196)
* ui: bake jpeg exif orientation into uploaded images
stb_image in mtmd ignores exif metadata, so rotated smartphone photos
reach the model with raw pixel orientation. The webui now reads the
exif orientation tag at send time and feeds it into the existing
capImageDataURLSize canvas pass: the browser applies the rotation when
decoding, so capped images come out upright for free, and images under
the cap threshold get a single plain redraw when orientation > 1.
At most one re-encode ever happens per image. Upright jpegs with
capping disabled pass through untouched, bit perfect.
Adds jpeg-orientation.ts with a minimal exif parser working on a
bounded base64 prefix (both endianness, returns 1 on any malformed
input) and unit tests against handcrafted jpeg byte streams.
* ui: move jpeg exif constants into lib/constants
* ui: add browser test for jpeg orientation and capping
Covers capImageDataURLSize end to end in chromium with real Pillow
generated jpeg fixtures across exif orientations 1/3/5/6/8: upright
quadrant colors checked pixel-wise, expected dimensions with and
without capping, no orientation tag left in the output, and strict
passthrough when nothing needs rewriting.
Gaurav Garg [Wed, 10 Jun 2026 17:51:16 +0000 (23:21 +0530)]
Remove padding and multiple D2D copies for MTP (#24086)
* Make ggml_gated_delta_net take only the initial recurrent state (D, 1, n_seqs) and passes the snapshot count K as an op parameter instead of inferring it from state->ne[1].
Remove the padding hack and copy all emitted snapshots into the recurrent cache with a single strided ggml_cpy
* Make GDN changes in all backends. Address review comments.
Oliver Simons [Wed, 10 Jun 2026 12:27:08 +0000 (14:27 +0200)]
CUDA: Fix ssm_scan_f32 data-races (#24360)
* Add missing syncthreads before resuing cub_temp_storage
__syncthreads() is required before being allowed to resue TempStorage
smem:
https://nvidia.github.io/cccl/unstable/cub/api/classcub_1_1BlockLoad.html#_CPPv4I0EN3cub9BlockLoad4LoadEv20RandomAccessIteratorRA14ItemsPerThread_1Ti
* Add one more missing __syncthreads
Could also double-buffer, but alternative is to simply ensure all
threads have read smem* before writing to it again in the next loop
iteration
ddh0 [Wed, 10 Jun 2026 07:31:35 +0000 (02:31 -0500)]
speculative : fix "ngram-map-k4v" name in logging (#24253)
This is a non-functional change.
When using `--spec-type ngram-map-k4v`, the log messages at startup and
runtime say `ngram-map-k`. Added logic in the in the constructor of
`common_speculative_impl_ngram_map_k` to pass the correct
`COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K4V` when `config.key_only` is
`false`.
After this change, the log messages use the correct name.
Expose a run_javascript tool to the model, executed entirely in the
browser through the existing agentic loop. Code runs in a Web Worker
inside a sandboxed iframe with an opaque origin, isolated from the
WebUI and its API. Console output, errors and the return value are
fed back as the tool result. The parent enforces a hard timeout by
removing the iframe, which terminates the worker.
Disabled by default, toggle in Settings > Developer.
* ui: address review feedback from allozaur
Use the JsonSchemaType enum for the tool definition parameter types
instead of raw string literals, extending it with STRING and NUMBER.
Move the worker shim and the iframe harness html into their own files
so the service no longer carries inline source blobs.
Replace the remaining magic strings with constants: SANDBOX_EMPTY_OUTPUT
and SANDBOX_TRUNCATION_NOTICE, and reuse NEWLINE_SEPARATOR for joins.
* ui: move sandbox worker shim to a raw imported file
Replace the inline worker template string with a real sandbox-worker.js
imported as raw text, and build the iframe harness from it in
sandbox-harness.ts. The raw worker ships as a string, not a module, so
it is excluded from eslint and the typecheck program.
Pascal [Tue, 9 Jun 2026 09:01:37 +0000 (11:01 +0200)]
ggml : add GGML_OP_COL2IM_1D (#24206)
* cpu: add GGML_OP_COL2IM_1D
Add the overlap-add (scatter-add) step of a 1D transposed convolution.
A ConvTranspose1d factorizes as a GEMM followed by col2im: a weight
pre-permuted to [IC, K*OC] is contracted against the [IC, T_in] input
with mul_mat to produce a column matrix [K*OC, T_in], and col2im_1d
scatters those columns back into the [T_out, OC] signal, with
T_out = (T_in - 1)*s0 + K - 2*p0.
Keeping the contraction as a plain mul_mat leaves the heavy work on the
optimized (and quantizable) matmul kernels, so col2im_1d only does the
cheap overlap-add.
CPU uses a gather formulation parallelized over output channels,
supporting F32, F16 and BF16 with an F32 accumulator.
* tests: add backend coverage for GGML_OP_COL2IM_1D
Add test_col2im_1d next to the conv_transpose_1d cases, covering F32,
F16 and BF16 across eight geometries: the canonical kernel = 2*stride
DAC upsampling shape, overlap, no overlap, cropping (p0 = 1 and
p0 = stride/2), kernel < stride with zeroed gaps, kernel not a
multiple of stride, and a single column unfold.
Perf mode gets three real vocoder stage shapes reporting memory
bandwidth. max_nmse_err relaxes to 5e-4 for F16 and BF16.
* cpu: harden GGML_OP_COL2IM_1D
ggml_col2im_1d validates s0, oc, p0 and input contiguity at graph
build time, before the oc division, protecting every backend at once.
The kernel asserts the contiguity its flat indexing assumes and its
doc states the full output length including the crop term.
The kernel parallelizes over the time axis: the split stays balanced
down to OC = 1, where the previous channel split was single threaded.
Values are bit identical on the three real vocoder chains, two out of
three improve.
* tests: extend the GGML_OP_COL2IM_1D grid
The eval grid grows to eleven geometries: OC = 1 (mono output stage),
K = 1 with stride > 1 (sparse scatter, every gap position zeroed) and
a crop down to T_out = 2 where all the gather bounds act at once.
* tests: add col2im_1d equivalence test
tests/test-col2im-1d.cpp proves mul_mat + col2im_1d matches the
native ggml_conv_transpose_1d on the CPU backend, F32 bit exact, F16
and BF16 through casts of the column matrix. test-backend-ops cannot
cover this for a CPU only op since the CPU backend is its own
reference there.
* rpc: bump protocol patch version for GGML_OP_COL2IM_1D
GGML_OP_COUNT goes from 96 to 97 with the new op, which trips the
static_assert in ggml-rpc.h. Bump RPC_PROTO_PATCH_VERSION since the
op is appended and no existing op code shifts.
fiesh [Tue, 9 Jun 2026 07:45:16 +0000 (09:45 +0200)]
server : do not clear slots without unified KV cache (#24190)
* Always export idle slots to RAM
Without this, a slot's VRAM cache may not be written to RAM. If this
slot happens to be busy then later on, this triggers needless
preprocessing in another slot.
* cont : clean-up
---------
Co-authored-by: Christoph Weiss <redacted> Co-authored-by: Georgi Gerganov <redacted>
overview / pkg / ggml / sources / llama.cpp / log