From: Junmo Kim <redacted>
Date: Tue, 11 Aug 2026 04:20:17 +0000 (+0900)
Subject: model : fix SWA not being enabled for EXAONE 4.5 (#26848)
X-Git-Tag: upstream/0.0.10438~77
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=14e78ddef7a2061e7d5a31dce4eb7ee0bcdbc840;p=pkg%2Fggml%2Fsources%2Fllama.cpp

model : fix SWA not being enabled for EXAONE 4.5 (#26848)

* model : fix SWA not being enabled for EXAONE 4.5

load_arch_hparams tests `hparams.n_layer() == 64` before
LLM_KV_NEXTN_PREDICT_LAYERS has been read. n_layer() returns
n_layer_all - n_layer_nextn and n_layer_nextn defaults to 0, so a GGUF
carrying the MTP head (block_count=65, nextn=1) evaluates to 65 and the
whole SWA block is skipped. The model type switch further down in the
same function reads 64, because by then the key has been loaded.

n_swa is still filled in by the unconditional get_key below the block, so
llama_model_n_swa() reports 4096 and the logs look correct while only
swa_type stays LLAMA_SWA_TYPE_NONE.

This affects the official LGAI-EXAONE GGUF release as well. EXAONE 4.0 has
no MTP head, so block_count is 64 there and the check matches.

* model-loader : skip TENSOR_SKIP tensors in the metadata-only path

create_tensor asserts on a null buffer type when building from metadata
alone, but buft_for_tensor returns null by design for tensors marked
TENSOR_SKIP, which is how architectures with nextn/MTP layers mark theirs.
Those models cannot be constructed by llama_model_init_from_user at all.

The file-backed path below already returns nullptr for the same tensors, so
callers see the same thing either way.

* tests : cover exaone4 hparams ordering

Builds a synthetic exaone4 model with the layout the shipped EXAONE 4.5
GGUFs use (block_count 65 + nextn 1). The swa_type check is the one that
catches the ordering bug; the n_layer_nextn and n_layer() checks only tell
a broken fixture apart from a real regression.

Fails before the ordering fix with "swa_type is not STANDARD", passes after.

* Revert "tests : cover exaone4 hparams ordering"

This reverts commit d2f3bafeee591ad691396b2708de4baef3aaf602.

* Revert "model-loader : skip TENSOR_SKIP tensors in the metadata-only path"

This reverts commit aecb9bc0c7896b52afbc43921a1f572aa7b5e53c.
---

diff --git a/src/models/exaone4.cpp b/src/models/exaone4.cpp
index 863268abc..a06819a67 100644
--- a/src/models/exaone4.cpp
+++ b/src/models/exaone4.cpp
@@ -1,6 +1,9 @@
 #include "models.h"
 
 void llama_model_exaone4::load_arch_hparams(llama_model_loader & ml) {
+    ml.get_key(LLM_KV_NEXTN_PREDICT_LAYERS, hparams.n_layer_nextn, false);
+    GGML_ASSERT(hparams.n_layer_nextn < hparams.n_layer_all && "n_layer_nextn must be < n_layer");
+
     if (hparams.n_layer() == 64) {    // 32B
         hparams.swa_type = LLAMA_SWA_TYPE_STANDARD;
         hparams.n_swa = 4096;
@@ -15,9 +18,6 @@ void llama_model_exaone4::load_arch_hparams(llama_model_loader & ml) {
 
     ml.get_key(LLM_KV_ATTENTION_SLIDING_WINDOW,    hparams.n_swa, false);
     ml.get_key(LLM_KV_ATTENTION_LAYERNORM_RMS_EPS, hparams.f_norm_rms_eps);
-    ml.get_key(LLM_KV_NEXTN_PREDICT_LAYERS,        hparams.n_layer_nextn, false);
-
-    GGML_ASSERT(hparams.n_layer_nextn < hparams.n_layer_all && "n_layer_nextn must be < n_layer");
 
     switch (hparams.n_layer()) {
         case 30: type = LLM_TYPE_1_2B; break;