Pascal [Sat, 8 Aug 2026 22:42:50 +0000 (00:42 +0200)]
server: report the isolate working directory from get_info (#26773)
* server: report the isolate working directory from get_info
Without an explicit cwd, get_info fell back to the server process
working directory even when a tools runtime was configured. That named a
host path no tool would ever run in, since an isolate starts in a
directory of its own.
It now asks the isolate for its working directory in that case, and
keeps the process one only when the tools run on the host.
Pascal [Sat, 8 Aug 2026 14:36:21 +0000 (16:36 +0200)]
server, ui: only offer a working directory when a tool reads it (#26762)
The working directory chip showed up as soon as the server exposed any
builtin tool, so a server started with just get_datetime, or a user who
turned every filesystem tool off in the settings, still got a control
that nothing would read.
Tools now declare whether they resolve their paths and run against the
working directory, next to the write permission they already publish in
the /tools listing. The WebUI shows the chip and enables the /cwd
command only when at least one such tool is both served and left
enabled.
Pascal [Fri, 7 Aug 2026 20:35:52 +0000 (22:35 +0200)]
tts: account for the vocoder pass in the timings line (#26733)
get_output runs the waveform work the pipeline defers to it, from a
single trailing window to a full pass depending on the model. Measuring
it keeps the reported total and the audio to process ratio honest.
robertomeroni [Fri, 7 Aug 2026 18:09:07 +0000 (20:09 +0200)]
metal : fix NORM/RMS_NORM for row lengths that leave a partial simdgroup (#26708)
ggml_metal_op_norm sized the threadgroup with
`nth = std::min(nth, args.ne00_t)`, which can leave nth not a multiple of
the simdgroup size. The kernels finish their row reduction with a
cross-simdgroup step where each lane of the last simdgroup reads one
per-simdgroup partial sum out of shmem_f32:
When the last simdgroup is partial it has fewer lanes than the
threadgroup has simdgroups, so the tail of the partial sums is never
read and the row sum is too small. For ne00_t = 33 nth becomes 33: two
simdgroups, but only one lane in the second, so one of the two partial
sums is dropped. The mean and variance are then wrong for the whole row.
Round ne00_t up to a whole number of simdgroups instead. Rounding up
rather than dropping the clamp keeps the threadgroup as small as
possible: deleting the line would raise nth to the next power of two
(ne00_t = 544 -> 1024 instead of 544), which costs idle lanes on 26 row
lengths below 8192 that were already correct, including 1536 and 3584.
GGML_OP_NORM is affected as well as GGML_OP_RMS_NORM - both dispatch
through ggml_metal_op_norm.
No mainstream LLM hidden size hits this: ne00_t is ne00/4 on the
vectorized path, so 4096, 8192, 2048 and friends all give a multiple of
32. It is reachable from other norm shapes, e.g. 320-channel norms.
Add NORM and RMS_NORM cases for ne0 = 33, 132 and 260 across the
existing eps values. 33 exercises the scalar path and 132/260 the
vectorized one, since only those divide by 4.
Before, on M3 Pro:
test-backend-ops test -b MTL0 -o NORM 25/50
test-backend-ops test -b MTL0 -o RMS_NORM 26/51
After:
test-backend-ops test -b MTL0 -o NORM 50/50
test-backend-ops test -b MTL0 -o RMS_NORM 51/51
test-backend-ops test -b MTL0 13943/13943
Emanuil Rusev [Fri, 7 Aug 2026 13:31:40 +0000 (16:31 +0300)]
webui: load the model selected via ?model= when ?load=true (#26707)
* webui: load the model selected via ?model=
Opening the WebUI with ?model= selects the model but doesn't load it. The load only starts when you send your first message, so you wait for it then.
This loads it as soon as the page opens, while you're still typing your prompt. It's what the model dropdown already does, and it isn't awaited, so the UI still works while the model loads.
This is the path the Llama macOS app uses to open the WebUI, so it's a common way in.
* webui: gate the load behind ?load=true
Loading on landing is opt-in, so a plain ?model= link behaves as before and doesn't allocate memory on its own.
* webui: name the chat URL params
Collects the query params the chat routes read into a URL_PARAMS constant, instead of repeating the literals across three files. NEW_CHAT_PARAM folds into it.
Pascal [Fri, 7 Aug 2026 11:32:52 +0000 (13:32 +0200)]
mtmd: stop feeding the text stream again during Qwen3-TTS generation (#26706)
The reference implementation has two mutually exclusive prompt layouts.
In non streaming mode the prefill carries the whole utterance text plus
tts_eos summed with codec_pad, and the trailing text hidden collapses to
a single tts_pad row. In streaming mode the prefill carries only the
first text token and the trailing rows stream the rest of the text
followed by tts_eos.
The pipeline built the non streaming prefill but the streaming overlay,
so the talker saw the utterance a second time during generation and read
it twice before emitting codec_eos.
The overlay is now the single tts_pad row that matches the prefill.
Pascal [Fri, 7 Aug 2026 10:07:58 +0000 (12:07 +0200)]
ui: read model modalities from the router model list (#26709)
* ui: read model modalities from the router model list
The router advertises input modalities for every model, loaded or not.
Reading them at list build time lets the UI accept image and audio
uploads for a model selected through ?model=, which has no /props yet.
Titaniumtown [Fri, 7 Aug 2026 05:24:40 +0000 (22:24 -0700)]
sycl: *glu flat path (#26354)
* tests: add SWIGLU perf cases
perf mode had no GLU coverage. Adds SWIGLU at 17408 columns, 512 and
2048 tokens, f16 and f32, with the operands both fused and split.
* sycl: consolidate fused-GLU kernels
They differed only in which op_* they called, so take the op as an argument and share a common launcher.
Their block sizes were all 256, so launch geometry is unchanged;
SYCL_GELU_BLOCK_SIZE and SYCL_SILU_BLOCK_SIZE lose their last users so are dropped.
* sycl: contiguous fast path for the fused GLU ops
o0 == n and o1 == n collapse the de-interleave index math to the
identity, so dispatch a flat kernel in that case. It fires for
ggml_glu_split with packed operands; a fused [gate|up] tensor keeps the
strided path. test-backend-ops perf -o SWIGLU on an Arc Pro B70: split
+14% f16 and +4% f32, fused unchanged.
JamePeng [Fri, 7 Aug 2026 04:49:14 +0000 (12:49 +0800)]
metal : avoid `threadgroup` matrix array instantiation in kernel_lightning_indexer (#26646)
- In MSL, declaring an array of matrix types like `threadgroup half4x4` causes
a 'no matching constructor' compilation error because MSL matrix types do not
have zero-argument default constructors and threadgroup variables cannot have
initializers.
- Fix this by declaring a POD `threadgroup half` array instead and casting
to `threadgroup half4x4 *` for matrix indexing.
On integrated GPUs (APUs), the scheduler can legitimately place a graph
node's output on the host-visible buffer, which ggml_cuda_compute_forward
already handles. The debug assert in ggml_cuda_graph_evaluate_and_capture
required every node output to be on the device buffer, so a debug build
aborts on such a node (e.g. attn_residual ADD -> ROCm_Host on RDNA3.5).
The source-tensor assert directly below already permits this via the
integrated + cuda_host exception; apply the same exception to the node's
own output buffer. Debug-only; no effect on release/compute.
Fixes test-recurrent-state-rollback on gfx1151 (Strix Halo).
* ci: enable unified memory for ROCm gfx1151 job
Work around a coherence issue on integrated RDNA3.5 (gfx1151) where GPU
kernels reading mmap-loaded weights can return incorrect output, which
makes test-llama-archs (and real inference) intermittently wrong.
GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 uses managed memory, which restores
coherence. Remove once the underlying ROCm/HIP issue is fixed.
* test-llama-archs: skip jamba on HIP backend
jamba produces incorrect output (~0.55 NMSE vs CPU) on the HIP backend on
RDNA3.5 (gfx1151); the SSM kernels need separate investigation. Skip it
for now, matching the existing per-backend carve-outs (WebGPU), so the
ROCm CI can run the test for the remaining architectures.
* ci: use HIP_LAUNCH_BLOCKING for ROCm gfx1151 job
The gfx1151 ROCm CI job produced incorrect inference output (qwen3 perplexity ~88 vs ~9.4) due to an async-execution correctness issue in the HIP path. Serializing kernel launches with HIP_LAUNCH_BLOCKING=1 restores correctness. This replaces the earlier GGML_CUDA_ENABLE_UNIFIED_MEMORY workaround, which did not fix batched inference.
* test-backend-sampler: skip top-k subtests on HIP backend
The ROCm backend does not support the TOP_K/ARGSORT op at vocab scale (no CUB; bitonic argsort is capped at ncols <= 1024), so top-k/top-p backend samplers cannot be offloaded. The penalties, set_sampler, mixed, and top_p subtests assert that offload happened, so they fail on HIP. Skip them until TOP_K is supported on the ROCm backend.
Daniel Bevenius [Thu, 6 Aug 2026 07:38:06 +0000 (09:38 +0200)]
model-conversion : add --model-name to conversion scripts (#26665)
This commit adds the --model-name flag to the causual and embedding
model conversion scripts.
The motivation for this is that this is the name used for the metadata
field general.name and it can be useful to specify this explicitely if
the default (the basename of the model path) is not what we want.
Pascal [Wed, 5 Aug 2026 22:47:59 +0000 (00:47 +0200)]
mtmd/ggml: add ggml_build_forward_order (#26649)
* ggml: add ggml_build_forward_order
ggml_build_forward_expand marks the tensor and all its ancestors for
compute, so using it as a pure ordering hint (keeping q, k and v
together) defeats ggml_build_forward_select: the unselected branch is
forced to run with inputs that were never uploaded. In the mtmd audio
graph this makes GEN_WAV calls execute the GEN_CODE branch with a
stale inp_code0, hitting the get_rows bound assert on CPU.
Add ggml_build_forward_order, which inserts nodes without the compute
flag; the flag is restored when the branch is actually selected.
Switch the q/k/v hints in clip_graph::build_attn to it.
Pascal [Wed, 5 Aug 2026 19:31:54 +0000 (21:31 +0200)]
server: harden the file_glob_search directory walk (#26626)
* server: don't walk Windows junctions in file_glob_search
std::filesystem reports a junction as a plain directory, so the symlink
guard misses it and a junction pointing back at an ancestor is walked
until the path length gives out
read the reparse tag and treat a symlink and a mount point as links,
leaving any other reparse point walkable so cloud placeholders and dedup
stubs still get searched
look junk directory names up case insensitively on Windows, where NTFS
makes Build the same directory as build
test that a junk directory stays selectable while its contents stay out
of search results
* server: report a directory the walk could not read
a directory that fails to open or to iterate was skipped in silence, so
a caller got a listing that looked complete while a whole subtree was
missing: a path over the platform limit, a volume going away, a name the
filesystem rejects
skip_permission_denied never reaches this path, so an error here is an
incomplete answer rather than a deliberate omission, and it now sets the
truncated flag
* server: simplify the file_glob_search listing plumbing
return a small result struct instead of two out params and a caller path
that only fed an error string, taking list_entries from six parameters
down to three
scope the error code to the directory being read, act on the status code
the entry lookups already returned, and treat an unreadable link state as
a link so the walk never descends on a guess
check the deadline when a directory is popped, not only per entry, so a
tree of empty directories cannot outlive the budget
read the path parameter once, and reject an invalid limit the way an
invalid type is already rejected, instead of silently falling back
normalize the resolved path, so a "." or ".." a caller typed reaches
neither git nor the client, and return the generic path form with '/'
separators on every platform, so the base sent to clients no longer needs
a local fixup
* ui: expire cached picker searches
the cache grew for the lifetime of the component: entries went stale
after the TTL but were never removed, so every distinct query typed in a
session stayed in memory
drop expired entries when a new result is stored
* server: address review from @ngxson
trim comments to one line each, and drop two that restate the code
rename junk_lookup_name to get_effective_name, and move it and the link
check to private static members next to junk_dir_names
merge the Windows and Linux link checks into one is_link, so symlinks are
checked everywhere and junctions only add to it on Windows
* server: convert tool paths as UTF-8 on Windows
a narrow path uses the active code page there, so a file name came back
mangled and a path with an accent could not be opened at all
convert explicitly at every crossing between a std::string, which always
carries UTF-8 here, and fs::path
read the home directory through the wide environment, since the narrow
one returns the profile path in the active code page too
the walker no longer normalizes separators by hand, since paths now come
back in generic form
* server: fold the platform branch inside console_output_to_utf8
match the shape of the other helpers, one definition with the #if inside,
instead of two definitions wrapped in #if and #else
inline the single caller helper and trim the comment
Pascal [Tue, 4 Aug 2026 20:24:55 +0000 (22:24 +0200)]
server: decode Windows OEM output to UTF-8 in built-in tools (#26597)
a child process writes in the OEM code page, which is not UTF-8 on a
western Windows install, so accented output reaches the JSON layer as
invalid bytes and gets replaced there, silently losing the characters
run() spawns without a console, so the child never inherits the console
code page and GetOEMCP is the one that applies
decode with MB_ERR_INVALID_CHARS so a wrong code page returns the text
untouched instead of emitting replacement characters, and pass text that
already decodes as UTF-8 through so a child emitting UTF-8 is never
decoded twice
the check drops an incomplete trailing sequence before validating, since
a streamed chunk can end in the middle of a multi-byte character
Oliver Simons [Tue, 4 Aug 2026 18:28:55 +0000 (20:28 +0200)]
sampler : remove "full-context windows" from history-based samplers (#26524)
* Resolve -1 to 1024 instead of ctx-len for samplers
Because of backend-sampling we initialize samplers before the complete
llama_context is there. Therefore, we cannot infer the resolved context
length yet at the time we construct the samplers.
* Shared default of 64 for history-based samplers, remove context_size
Aleksander Grygier [Tue, 4 Aug 2026 17:05:48 +0000 (19:05 +0200)]
ui: CWD for agent (#26518)
* server : extend file_glob_search for UI pickers
* ui : add per-conversation working directory with picker
* ui : add path navigation and search scope to cwd picker
Treat path-like queries (starting with / or ~) as directory navigation
instead of glob-matching the whole query: search the parent for the last
segment, and descend into an exactly-typed directory by listing its
children. Show the effective search scope in the footer and auto-search
on open so the current directory and its siblings appear immediately.
Assisted-by: Claude
* db : persist per-call tool cwd on tool result messages
* ui : abbreviate tool paths under home with a tilde
* ui : show the per-call cwd on exec shell rows
* ui : clarify the synthetic cwd message for the model
* ui : reuse the trailing cwd row on a repeated pick
* ui : don't jump when a cwd row is injected mid-chat
* chore: Formatting
* refactor: Cleanup comments
* ui : unify working directory naming and add a synthetic-message flag
* ui : render synthetic cwd rows without a scroll jump
* ui : decouple the working directory picker into utils and sub-components
* ui : escape glob metacharacters in picker search glob
* ui : simplify auto-scroll pin
* chore: Format
* fix: Use `SvelteMap`
* refactor: Post-review fixes
* ui: accept Windows roots in the working directory picker
recognize a drive root (C:) and a UNC share (//host/share) as path
navigation, alongside the POSIX root and ~, so a query like D:\repos
lists that directory instead of glob-matching it under the home dir
split below the root, so a bare drive resolves to its root rather than
to a drive-relative prefix
rewrite backslashes into forward slashes only when the query carries a
Windows root, since a backslash is a legal POSIX filename character
paths keep travelling with forward slashes, which is what the server
returns and what Windows accepts
This commit contains a suggestion to reduce some code duplication in
common_speculative_init when adding the enabled speculative decoding
configurations.
No tests were added but the existing server tests still passes with this
change:
```console
$ ./tests.sh unit/test_speculative.py -v -x
```
hcl [Tue, 4 Aug 2026 09:12:48 +0000 (17:12 +0800)]
gguf-py: validate n_dims and guard against uint64 overflow in reader (#25401)
The Python GGUF reader lacked two guards the C++ loader has:
- n_dims read as uint32 with no GGML_MAX_DIMS bound -> crafted file with
huge n_dims triggers oversized memmap read / OOM.
- np.prod(dims) on uint64 wraps silently -> a crafted dims triple can
overflow to a tiny element count, passing an undersized read through.
Add a GGML_MAX_DIMS check and compute the element count with Python ints.
Caleb DeLeeuw [Tue, 4 Aug 2026 07:34:30 +0000 (00:34 -0700)]
convert : import bytes_to_unicode from convert_slow_tokenizer (#26217)
bytes_to_unicode was removed from transformers.models.gpt2.tokenization_gpt2
in huggingface/transformers#40936, but it had already been copied into
transformers.convert_slow_tokenizer in huggingface/transformers#30334
(transformers 4.54.1), so import it directly from there.
Titaniumtown [Tue, 4 Aug 2026 02:08:05 +0000 (19:08 -0700)]
sycl: parallelize the non-contiguous concat kernel (#25852)
* sycl: parallelize the non-contiguous concat kernel
Launch geometry only: the non-contiguous concat kernel launched a single-lane
work-group (1, 1, 1), now it will launch a (1, 1, SYCL_CONCAT_BLOCK_SIZE) one.
SYCL_CONCAT_BLOCK_SIZE is defined in `ggml/src/ggml-sycl/presets.hpp`.
llama-bench (Arc Pro B70, Qwen3.6-27B-UD-Q4_K_XL, -fa on, q8_0 KV),
on top of upstream master: pp2048 920 -> 1006 t/s (+9.4%)
* sycl: cap non-contiguous concat block at ne0
* sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)
* Revert "sycl: make non-contiguous concat block width env-tunable (GGML_SYCL_CONCAT_BLOCK_SIZE)"
Ozymandias_EBON [Tue, 4 Aug 2026 02:07:23 +0000 (21:07 -0500)]
Extended SYCL oneDNN SDPA to non-FP16 KV caches (Q4_0–Q8_0 and FP32) (#25874)
* sycl: extend oneDNN SDPA to Q4_0-Q8_0 and F32 KV caches
Extends the oneDNN SDPA path (PR #25222) to handle non-F16 KV caches by
dequantizing or converting K/V to dense FP16 on-device before feeding
them into the SDPA graph. The fused systolic kernel then runs identically
to the native FP16 path.
Supported KV types:
- Q4_0, Q4_1, Q5_0, Q5_1, Q8_0: to_fp16_sycl / to_fp16_nc_sycl
- F32: cont_to_f16_sycl<float>
- BF16 and IQ types are excluded (no conversion kernel available)
Gate: non-F16 requires K >= 1024 and Q >= 32 (prefill only).
F16 KV runs at any length (existing behavior).
Also includes the stream sync fix (stream->wait_and_throw() unconditional,
PR #25741 by @malsbat) and removal of V_is_K_view aliasing (K and V are
always dequantized to separate buffers).
Co-Authored-By: Claude <redacted>
* docs: drop GGML_SYCL_FA_DEBUG from SYCL.md (not shipped in this PR) Co-Authored-By: Claude <redacted>
---------
Thiago Padilha [Mon, 3 Aug 2026 22:59:11 +0000 (19:59 -0300)]
chat : add new template for DeepSeek V4 Flash 0731 (#26398)
* common/chat: update DeepSeek V4 templates
Align the DeepSeek V4 templates with the official encoders while keeping parser behavior out of this change.
- Default drop_thinking for DeepSeek V4 history so prior thinking is omitted unless preserve_reasoning is requested or tools are present.
- Add structured output response-format instructions to the V4 templates and pass the schema into template rendering.
- Add a separate Flash 0731 template for the updated high and max reasoning effort mapping.
- Cover reasoning effort, drop_thinking, structured output prompts, preserved reasoning, continuations, and empty tool arguments in template rendering tests.
Official references:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.py
https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py
Pascal [Mon, 3 Aug 2026 16:58:15 +0000 (18:58 +0200)]
tests: add model resolution test on synthetic repo listings (#26172)
* tests: add model resolution test on synthetic repo listings
Include download.cpp and arg.cpp inside a namespace with hf_cache
monkey patched to serve hardcoded listings, so the resolution and the
model handler assembly are tested end-to-end through the real CLI
parsing, without modifying the tested code and without network access.
Covers the primary, shard, mmproj, sidecar and preset resolution on
layouts mimicking real vendor conventions, replays every case on
permutations of the listing to assert determinism, and asserts the
final wired paths for the spec type auto-selection, the -md precedence
and the fallback suppression.
* tests: keep model resolution checks active and let the handler resolve
Replace assert with a REQUIRE macro alive in Release builds, key the
fake hf_cache by repo id so the real handler init resolves every plan
itself, check the exact shard sets, restrict the permutation exception
to the order dependent picks, and cover dflash and eagle3.
* tests: fix model resolution build on fatal warnings CI and Windows
The namespaced copy of the sources leaves many static functions unused
in this TU, exempt it from the unused warnings. Pre-include the
windows headers so arg.cpp does not pull them inside the namespace.
Declare the renamed copies of the download.h functions, verbatim from
the header and renamed in sync by the macros, so missing declaration
and missing prototype warnings are satisfied on every toolchain.
* tests: fix winsock inclusion order for the model resolution test
WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does
not redefine the socket types afterwards.
* tests: link cpp-httplib to the model resolution test
The test compiles its own copy of download.cpp, which calls httplib
directly, and the private link of llama-common does not propagate the
symbols under lld-link.
* common_http_client
* common: finish the http client wrapper
Add the virtual Head, Get and Post methods and the passthrough
setters to the common_http_client skeleton, move follow_location
into the constructor, expose the underlying client for the ranged
pull path, and rename the missed common_http_client_init call sites.
* tests: rewrite model resolution on the http client stub
Replace the namespace inclusion of the sources by a plain TU: the
common_http_client factory returns a stub serving hardcoded HF API
responses, so the real hf_cache parsing, resolution and CLI handler
run against synthetic listings in an isolated cache directory.
Failures print the named case, the reordering and the actual versus
expected values, the assembly cases use the full command line as
context, and the empty result cases are checked once to keep the
logs short.
* tests: fix the model resolution on Windows and the builds without TLS
Assert the exact expected paths composed like the cache does instead of
suffix matching on forward slashes, set the environment portably, and
serve the stub through an http endpoint so the builds rejecting the
https scheme still reach it. Pause the log so the negative cases can be
replayed on every reordering.
* tests: make the model resolution failures self explanatory
Resume the paused log before the failure report so the CI shows why
the tested code bailed, and format the stub oids portably.
* common: hold the http client factory behind exported functions
The factory was an inline variable, and the Windows shared builds
export functions but not data, so the executable and the DLL each had
their own instance: the stub installed by the test was invisible to
the library, which reached for the real endpoint and resolved nothing.
Route the creation through functions compiled into the library and
format the stub oids portably.
* common: add the http client factory source missed in the previous commit
* common: typedef the http client factory callback
Address review from @ngxson
* tests: serve the model resolution repos over the loopback
Replace the client stub by a real httplib server bound to the
loopback, so no C++ object crosses the module boundary anymore and
the library exercises its own client and transport end to end. The
debug shared build on Windows crashed inside the stubbed path.
* common: add portable common_get_env and common_set_env helpers
Address review from @ngxson
* common: drop the http client factory left without a caller
The loopback server made the stub substitution unnecessary, the client
init builds the real client directly again.
* common: read the model endpoint through the env helpers
* nit: drop the stub leftovers from the model resolution test
* common: align common_set_env and isolate the test cache per run
The POSIX branch now behaves like _putenv_s, so the helper has a single
contract on every platform, and common_get_env already reads an unset and
an empty variable alike.
The model resolution test keys its cache directory on the loopback port,
where two concurrent runs on the same machine used to share one directory
and the initial cleanup of either wiped the other.
* tests: move the model resolution server into main
* tests: support the DSpark sidecar resolution
* common: revert the http client to the plain httplib client
Xuan-Son Nguyen [Mon, 3 Aug 2026 16:51:02 +0000 (18:51 +0200)]
server: add get_info tool (#26522)
* server: add get_info tool
* fix --rpc in docs
* server: harden get_info probe result handling
Report the OS as unknown when the probe process fails to spawn or times
out, so the diagnostic text from run() is never returned as an OS name.
Strip the probe output on both ends, which also drops the blank line
that ver prints before the version on Windows. Name the output and
timeout limits, and report an unreadable working directory as unknown
instead of an empty string.
* server: simplify get_info result handling
Drop the named limits and the working directory error branch, keeping
the probe result handling to a single expression.
AgoraPete [Mon, 3 Aug 2026 15:03:14 +0000 (17:03 +0200)]
ggml: use dynamic allocation for split graph inputs (#22789)
* ggml: use dynamic allocation for split graph inputs
Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically
allocated buffers in the backend scheduler. This fixes crashes when
loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on
multi-backend setups where graph splits exceed 30 input tensors.
- split->inputs: dynamic array with grow-on-demand
- sched->graph_inputs: dynamic array with grow-on-demand
- graph_size calculation now uses actual input count instead of fixed constant
Hongqiang Wang [Mon, 3 Aug 2026 14:36:19 +0000 (07:36 -0700)]
opencl: route large q6_K lm_head to the flat GEMV (#26427)
* add a direct size condition for `large` weights; the original
dimension condition is insufficient -- q6_K lm_head for gemma-4 E2B
has [1536, 262144], which is big enough to slowdown gemv_noshuffle but
does not satisfy the dimension condition (ne0 >= 2048)
Konrad Moren [Mon, 3 Aug 2026 12:26:09 +0000 (14:26 +0200)]
CUDA: Add backend sampler for penalties sampler (#25262)
* sampling: enhance penalty handling in common_sampler_init
- Set default value for penalty_last_n based on model context if not specified.
- Ensure penalty_last_n and n_prev are non-negative.
- Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.
- Implement backend initialization and application logic for penalties, including frequency and presence adjustments.
* tests: add backend penalties sampling tests and utility functions
- Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness.
- Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties.
- Added `test_backend_penalties_sampling` to validate backend penalties with various configurations.
- Enhanced the test suite for better coverage of penalty handling in sampling.
* sampling: add support for top-k penalties in backend sampling
* sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN.
* sampling: enhance penalty comparison tests with masking penalties logic
* add comments on padding
* sampling: add comments on modifications
* add the unit test to cover masked-out token as -INF
* validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values
* refactor: test functions to share logic and be less verbose
* add test to cover case where previously penalized token is not part of candidates
* remove comments
* remove redundant penalty_last_n initialization and validation in common_sampler_init
* add support for penalties in sampler chain with configurable positions
* add validation for penalty parameters and enhance tests for non-finite values
* add context parameter to common_sampler_init and set default for penalty_last_n
* add llama_n_ctx parameter to common_sampler_init for improved sampler initialization
* replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor
* add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64
* add test for backend penalties sampling after top-p with large history window
Thiago Padilha [Mon, 3 Aug 2026 04:33:37 +0000 (01:33 -0300)]
metal: implement DSv4 Lightning Indexer (#25893)
* metal: implement F16 Lightning Indexer
- Implement GGML_OP_LIGHTNING_INDEXER for 128-dimensional, 64-head inputs
with F32 queries and weights plus F16 keys and masks.
- Add tiled and tail kernels and test KV lengths around 8- and 64-element
boundaries.
Assisted-by: Codex
* metal: stage Lightning Indexer K tiles
- Stage and dequantize K in F16 threadgroup memory before simdgroup matrix loads.
- Zero-fill partial tiles and guard stores so all KV segments use the same numerical path.
- Support F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0 K caches.
llama-bench (--mmap 1, -fa on, -p 512, -n 128; d=0/10k/20k):
overview / pkg / ggml / sources / llama.cpp / log