* Add preliminary MiniMax-M3 support
Text-only port that re-uses existing components: MiniMax-M2 style GQA with
per-head QK-norm and partial rotary, DeepSeek-V3 style leading-dense and
routed/shared experts, and swigluoai activation. Sparse attention is not
yet supported (dense fallback); vision tower and MTP heads are dropped.
* MiniMax-M3 vision tower (mmproj + clip graph)
* Delete m3_vision_ref.py
* Update clip.cpp
* MSA
* Update constants.py
* Update minimax.py
* Cache creation. Working withotu flash attention
* Added flash attention for sparse layers
* Decomposed slow cpu OP into GPU + CPU ops. Massive speedup over long ctx
* Rewrote indexer op to be cuda native. Modified flash attention to match per group block picking
* Implement sparse attention calc out of stock ops.
* Fix a cache allocation and cont issue
* Fixed -fa auto crash, flagged debug spots
* Delete vocab.json
* Delete model.safetensors.index.json
* Delete generation_config.json
* Delete Minimax directory
* Handled multi stream case to fall back on Dense Attention
* Development scaffolding cleanup. No functional change to the decode or
4-way paths. Full debug harness remains at <
8136a9c68ed7a5eb009aa67bba3fda8062f4648f> for reproducing the
selection-parity validation.
* Remove redundant comment from minimax-m3.cpp
* Changed 3 Gelu Ops for vision into Gelu_erf ops
* Assert that n_kv is multiple of 128
* Rename MSA index tensors to indexer convention
Note: All GGUFs generated before this change will need to be regenerated.
* Fix incorrect Assert
* Review driven changes (#3)
* Remove comment from conversion minimax.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* Remove whitespaces from constants.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* Tighten comment in minimax.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* inherit MiniMax-M3 from MiniMax-M2
* drop dead text_config fallbacks
* Add indexer writer methods
* Reuse LLM_FFN_SWIGLU_OAI_MOE
* Remove duplicate indexer setters, add only block_size/local_blocks, follow value naming convention
* Fix conversion error /gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* Update gguf-py/gguf/gguf_writer.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* Update gguf-py/gguf/tensor_mapping.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* Update conversion/minimax.py
Co-authored-by: Sigbjørn Skjæret <redacted>
* Remove whitespace in src/llama-kv-cache.cpp
Co-authored-by: Sigbjørn Skjæret <redacted>
* Remove Whitespace in Update src/llama-model.h
Co-authored-by: Sigbjørn Skjæret <redacted>
* Remove whitespace in src/llama-hparams.h
Co-authored-by: Sigbjørn Skjæret <redacted>
* Update minimax_m3.cpp
Rewrite code comment based on feedback and to better reflect the actual architecture, and reuse existing build_vit
* Rename minimax_m3.cpp to minimax-m3.cpp
* Update CMakeLists.txt
* Remove debug code from clip.cpp
* Update clip.cpp
* Update comments in tools/mtmd/models/minimax-m3.cpp
* Permute Q/K at conversion, drop precomputed sin/cos
* Log cache size on launch, block ctx shift, support prompt caching
Log indexer cache size on launch
Disallow ctx shift
Support prompt caching
* Update minimax-m3.cpp
* Optimize implementation, add multi stream support.
Fully rewrote minimax-m3.cpp for speed and buffer size gains:
Unified the 4-way + decode, 1 FA call per layer instead of 4, with the groups mapped onto ne[3]
Custom CPU op now emits block-level mask, expanded on GPU, which causes CPU to GPU transfer to shrinks at prefill
Decode: ~25 nodes/layer vs ~50, no per-group concats/conts
Unified selection semantics, so both regimes rank bs + local bias (position-anchored local force), which means prefill/decode can no longer disagree on selection
can_reuse on the MSA bias input. Graph reuse at decode restored (was rebuilding the full graph every token)
In-place mask adds, shrinking compute buffer ~6.8 to ~4.2 GiB at ub2048/62k
Multi-stream: MSA now runs with -np N when kv_unified=false. Decode stays batched across streams (still 1 FA call), prefill loops per stream. dense fallback only for --kv-unified + multi-seq
Measured effect on expert offload bound setup: decode 6.2(4WAY)–7.15(MSA_decode) -> 7.7~7.8 t/s, flat from 5k to 60k+. prefill around 10% faster. buffer about 20% smaller, multi-user support.
* set default cache type to F32
* Fix potential DSA double indexer cache allocation bug, only allocate in-cache k_idx for archs that opt in
* remove F16 downcasts in MSA attention, force F32 indexer score accum
* Add Minimax eos to llama vocab
* Guard edge case where idx cache can become stale after a tail trim
* Update llama-kv-cache.h
* Update llama-kv-cache.cpp
* Update llama-kv-cache.cpp
* Update llama-kv-cache.h
* Change resize Pad to none, resize alg to Bicubic Pillow
* Review driven changes
* Update llama-kv-cache.cpp
* rm unrotated pos_t
* fused rope w + pad
* rename merge --> merger for consistency
* add review skill for mtmd
* graph should use hparams n_merge
* fix lint
---------
Co-authored-by: Daniel Han <redacted>
Co-authored-by: Sigbjørn Skjæret <redacted>
Co-authored-by: Xuan Son Nguyen <redacted>
"LlavaForConditionalGeneration": "llava",
"MERaLiON2ForConditionalGeneration": "ultravox",
"MiMoV2ForCausalLM": "mimo",
+ "MiniMaxM3SparseForConditionalGeneration": "minimax",
"MiniCPMV4_6ForConditionalGeneration": "minicpm",
"Mistral3ForConditionalGeneration": "llava",
"NemotronH_Nano_VL_V2": "nemotron",
if TYPE_CHECKING:
from torch import Tensor
-from .base import ModelBase, TextModel, gguf
+from .base import ModelBase, TextModel, MmprojModel, gguf
@ModelBase.register("MiniMaxM2ForCausalLM")
data_torch = data_torch + 1.0
yield from super().modify_tensors(data_torch, name, bid)
+
+
+@ModelBase.register("MiniMaxM3SparseForConditionalGeneration", "MiniMaxM3VLForConditionalGeneration")
+class MiniMaxM3VisionModel(MmprojModel):
+ @classmethod
+ def filter_tensors(cls, item):
+ name, gen = item
+ # keep only the vision-side tensors; text / mtp / sparse-index are dropped
+ if not name.startswith(("vision_tower.", "multi_modal_projector.", "patch_merge_mlp.")):
+ return None
+ return super().filter_tensors((name, gen))
+
+ def set_gguf_parameters(self):
+ super().set_gguf_parameters()
+ assert self.hparams_vision is not None
+
+ self.gguf_writer.add_clip_projector_type(gguf.VisionProjectorType.MINIMAXM3)
+ self.gguf_writer.add_vision_use_gelu(True)
+
+ # the ViT carries its own LayerNorm eps (text tower uses a different one)
+ self.gguf_writer.add_vision_attention_layernorm_eps(
+ self.hparams_vision.get("layer_norm_eps", 1e-5)
+ )
+
+ comp = self.hparams_vision.get("img_token_compression_config", {})
+ merge_size = comp.get("spatial_merge_size", 2)
+ self.gguf_writer.add_vision_spatial_merge_size(int(merge_size))
+
+ def modify_tensors(self, data_torch, name, bid):
+ assert self.hparams_vision is not None
+
+ # Conv3d patch embed -> Conv2d slices
+ if name == "vision_tower.vision_model.embeddings.patch_embedding.weight":
+ if data_torch.ndim != 5:
+ raise ValueError(f"unexpected patch_embedding rank {data_torch.ndim} for {name}")
+ kt = data_torch.shape[2]
+ base = gguf.TENSOR_NAMES[gguf.MODEL_TENSOR.V_ENC_EMBD_PATCH]
+ for t in range(kt):
+ suffix = ".weight" if t == 0 else f".weight.{t}"
+ yield (base + suffix, data_torch[:, :, t, ...])
+ return
+
+ # Permute ViT q/k. HF [Ta Ha Wa | Tb Hb Wb | pad] reorder to [Ta Tb | Ha Hb | Wa Wb | pad].
+ for new_name, tensor in super().modify_tensors(data_torch, name, bid):
+ if ".attn_q." in new_name or ".attn_k." in new_name:
+ tensor = self._permute_vit_qk(tensor, new_name)
+ yield new_name, tensor
+
+ def _permute_vit_qk(self, t: "Tensor", new_name: str) -> "Tensor":
+ assert self.hparams_vision is not None
+ n_head = self.hparams_vision["num_attention_heads"]
+ d_head = t.shape[0] // n_head
+ axis_dim = 2 * ((2 * (d_head // 2) // 3) // 2)
+ ah = axis_dim // 2
+ half = 3 * ah
+ perm = []
+ perm += list(range(0, ah))
+ perm += list(range(half, half + ah))
+ perm += list(range(ah, 2 * ah))
+ perm += list(range(half + ah, half + 2 * ah))
+ perm += list(range(2 * ah, 3 * ah))
+ perm += list(range(half + 2 * ah, half + 3 * ah))
+ perm += list(range(2 * half, d_head))
+
+ assert axis_dim % 2 == 0
+ assert 3 * axis_dim <= d_head
+ assert len(perm) == d_head
+ assert sorted(perm) == list(range(d_head)), "perm is not a bijection of d_head"
+ assert t.shape[0] == n_head * d_head, f"{new_name}: {t.shape[0]} != {n_head}*{d_head}"
+ assert d_head == 80
+
+ idx = torch.tensor(perm, dtype=torch.long)
+ if t.ndim == 2:
+ return t.reshape(n_head, d_head, t.shape[1])[:, idx, :].reshape(t.shape)
+ return t.reshape(n_head, d_head)[:, idx].reshape(t.shape)
V_MM_UP = auto() # cogvlm
V_MM_DOWN = auto() # cogvlm
V_MM_GATE = auto() # cogvlm
+ V_MM_MERGER_FC1 = auto() # minimax-m3 (patch-merge MLP)
+ V_MM_MERGER_FC2 = auto() # minimax-m3 (patch-merge MLP)
V_TOK_BOI = auto() # cogvlm
V_TOK_EOI = auto() # cogvlm
V_TOK_IMG_BEGIN = auto() # hunyuanvl
MODEL_TENSOR.V_MM_UP: "mm.up",
MODEL_TENSOR.V_MM_DOWN: "mm.down",
MODEL_TENSOR.V_MM_GATE: "mm.gate",
+ MODEL_TENSOR.V_MM_MERGER_FC1: "mm.merger.fc1",
+ MODEL_TENSOR.V_MM_MERGER_FC2: "mm.merger.fc2",
MODEL_TENSOR.V_TOK_BOI: "v.boi",
MODEL_TENSOR.V_TOK_EOI: "v.eoi",
MODEL_TENSOR.V_MM_PRE_NORM: "mm.pre_norm",
MODEL_TENSOR.V_RESMPL_QUERY,
MODEL_TENSOR.V_TOK_EMBD_IMG_BREAK,
MODEL_TENSOR.V_MM_PATCH_MERGER,
+ MODEL_TENSOR.V_MM_MERGER_FC1,
+ MODEL_TENSOR.V_MM_MERGER_FC2,
MODEL_TENSOR.V_DS_NORM,
MODEL_TENSOR.V_DS_FC1,
MODEL_TENSOR.V_DS_FC2,
YOUTUVL = "youtuvl"
NEMOTRON_V2_VL = "nemotron_v2_vl"
HUNYUANVL = "hunyuanvl"
+ MINIMAXM3 = "minimax_m3"
MINICPMV4_6 = "minicpmv4_6"
GRANITE_SPEECH = "granite_speech" # audio
MIMOVL = "mimovl"
"visual.downsample", # glm4v
),
+ MODEL_TENSOR.V_MM_MERGER_FC1: (
+ "patch_merge_mlp.linear_1", # minimax-m3
+ ),
+
+ MODEL_TENSOR.V_MM_MERGER_FC2: (
+ "patch_merge_mlp.linear_2", # minimax-m3
+ ),
+
MODEL_TENSOR.V_DS_NORM: (
"model.visual.deepstack_merger_list.{bid}.norm", # deepstack in qwen3vl
),
- Security: don't trust client-supplied headers (e.g. `X-Forwarded-For`) or add footguns; things like IP allowlisting belong at a reverse proxy unless there's a trusted-proxy design.
- Wire new behavior into the existing request/response and checkpoint paths correctly; watch for resource leaks across requests.
+## Multimodal (`tools/mtmd/`)
+
+- Tensor names must be prefixed by `v.`, `a.`, `mm.` or `a.mm.` (legacy naming doesn't follow this convention - this is expected, but new code should follow it).
+- Do not use explicit sin/cos for RoPE; use `ggml_rope_ext` instead, see `HOWTO-add-model.md`. If it can't express the needed behavior, that's a design discussion, not a PR.
+- New GGML ops must not be introduced in the same PR, you must push it as a separate PR.
+- In most cases, `build_vit` should be enough to build the transformer graph for vision models. Do not add a loop to build the transformer graph manually, unless you have a very good reason to do so. If you do, please explain why in the PR description.
+- If you need a dedicated preprocessor, there is a high chance that it can be a derived class from one of the existing preprocessors. Check carefully before adding a new preprocessor class.
+- If the model need a new public API in `mtmd.h`, open a discussion first.
+
## General (always)
Enforce the `AGENTS.md` / `CONTRIBUTING.md` coding and naming guidelines on every changed line - this is a distinct pass from checking that the code works, and matters just as much for review speed:
models/paddleocr.cpp
models/pixtral.cpp
models/qwen2vl.cpp
+ models/minimax-m3.cpp
models/qwen3vl.cpp
models/mimovl.cpp
models/qwen3a.cpp
#define TN_MM_SOFT_EMB_N "mm.soft_emb_norm.weight" // gemma3
#define TN_MM_PROJECTOR "mm.model.fc.%s" // idefics3, deepseekocr
#define TN_MM_PATCH_MERGER "mm.patch_merger.%s" // mistral small 3.1, glm4v
+#define TN_MM_MERGER_FC1 "mm.merger.fc1.%s" // minimax-m3 patch-merge MLP
+#define TN_MM_MERGER_FC2 "mm.merger.fc2.%s"
#define TN_TOK_IMG_BREAK "v.token_embd.img_break" // pixtral
#define TN_TOK_GLM_BOI "adapter.boi" // glm-edge (these embeddings are not in text model)
#define TN_TOK_GLM_EOI "adapter.eoi" // glm-edge (these embeddings are not in text model)
PROJECTOR_TYPE_MINICPMV4_6,
PROJECTOR_TYPE_GRANITE_SPEECH,
PROJECTOR_TYPE_MIMOVL,
+ PROJECTOR_TYPE_MINIMAX_M3,
PROJECTOR_TYPE_GRANITE4_VISION,
PROJECTOR_TYPE_UNKNOWN,
};
{ PROJECTOR_TYPE_MINICPMV4_6, "minicpmv4_6"},
{ PROJECTOR_TYPE_GRANITE_SPEECH, "granite_speech"},
{ PROJECTOR_TYPE_MIMOVL, "mimovl"},
+ { PROJECTOR_TYPE_MINIMAX_M3, "minimax_m3"},
{ PROJECTOR_TYPE_GRANITE4_VISION, "granite4_vision"},
};
ggml_tensor * mm_0_b = nullptr;
ggml_tensor * mm_2_w = nullptr;
ggml_tensor * mm_2_b = nullptr;
+ ggml_tensor * mm_merger_fc1_w = nullptr; // minimax-m3
+ ggml_tensor * mm_merger_fc1_b = nullptr;
+ ggml_tensor * mm_merger_fc2_w = nullptr;
+ ggml_tensor * mm_merger_fc2_b = nullptr;
ggml_tensor * image_newline = nullptr;
ggml_tensor * view_seperator = nullptr;
{
builder = std::make_unique<clip_graph_mimovl>(ctx, img);
} break;
+ case PROJECTOR_TYPE_MINIMAX_M3:
+ {
+ builder = std::make_unique<clip_graph_minimax_m3>(ctx, img);
+ } break;
case PROJECTOR_TYPE_STEP3VL:
{
builder = std::make_unique<clip_graph_step3vl>(ctx, img);
LOG_WRN("%s: more info: https://github.com/ggml-org/llama.cpp/issues/16842\n\n", __func__);
}
} break;
+ case PROJECTOR_TYPE_MINIMAX_M3:
+ {
+ hparams.n_merge = 2; // spatial_merge_size
+ hparams.image_resize_algo = RESIZE_ALGO_BICUBIC_PILLOW;
+ hparams.image_resize_pad = PAD_NONE;
+ get_u32(KEY_SPATIAL_MERGE_SIZE, hparams.n_merge, false);
+ hparams.rope_theta = 10000.0f; // vision_config.rope_theta
+ // MiniMax-M3: max_pixels 451584 (=672^2) -> 576 merged tokens (image_seq_length)
+ hparams.set_limit_image_tokens(8, 576);
+ hparams.set_warmup_n_tokens(16*16);
+ } break;
case PROJECTOR_TYPE_MIMOVL:
{
hparams.n_merge = 2; // spatial_merge_size
model.mm_1_w = get_tensor(string_format(TN_LLAVA_PROJ, 2, "weight"));
model.mm_1_b = get_tensor(string_format(TN_LLAVA_PROJ, 2, "bias"), false);
} break;
+ case PROJECTOR_TYPE_MINIMAX_M3:
+ {
+ // per-patch MLP: mm.1 -> gelu -> mm.2
+ model.mm_1_w = get_tensor(string_format(TN_LLAVA_PROJ, 1, "weight"));
+ model.mm_1_b = get_tensor(string_format(TN_LLAVA_PROJ, 1, "bias"));
+ model.mm_2_w = get_tensor(string_format(TN_LLAVA_PROJ, 2, "weight"));
+ model.mm_2_b = get_tensor(string_format(TN_LLAVA_PROJ, 2, "bias"));
+ // 2x2 merge MLP: mm.merge.fc1 -> gelu -> mm.merge.fc2
+ model.mm_merger_fc1_w = get_tensor(string_format(TN_MM_MERGER_FC1, "weight"));
+ model.mm_merger_fc1_b = get_tensor(string_format(TN_MM_MERGER_FC1, "bias"));
+ model.mm_merger_fc2_w = get_tensor(string_format(TN_MM_MERGER_FC2, "weight"));
+ model.mm_merger_fc2_b = get_tensor(string_format(TN_MM_MERGER_FC2, "bias"));
+ } break;
case PROJECTOR_TYPE_STEP3VL:
{
model.mm_0_w = get_tensor(string_format(TN_LLAVA_PROJ, 0, "weight"));
case PROJECTOR_TYPE_QWEN3VL:
case PROJECTOR_TYPE_EXAONE4_5:
case PROJECTOR_TYPE_MIMOVL:
+ case PROJECTOR_TYPE_MINIMAX_M3:
case PROJECTOR_TYPE_GLM4V:
case PROJECTOR_TYPE_YOUTUVL:
{
set_input_i32("positions", positions);
} break;
+ case PROJECTOR_TYPE_MINIMAX_M3:
+ {
+ const int n_merge = hparams.n_merge;
+ const int gh = image_size_height / patch_size;
+ const int gw = image_size_width / patch_size;
+ std::vector<int32_t> pos_h, pos_w;
+ pos_h.reserve(gh * gw);
+ pos_w.reserve(gh * gw);
+ for (int bh = 0; bh < gh / n_merge; bh++)
+ for (int bw = 0; bw < gw / n_merge; bw++)
+ for (int mh = 0; mh < n_merge; mh++)
+ for (int mw = 0; mw < n_merge; mw++) {
+ pos_h.push_back(bh * n_merge + mh);
+ pos_w.push_back(bw * n_merge + mw);
+ }
+ set_input_i32("minimax_pos_h", pos_h);
+ set_input_i32("minimax_pos_w", pos_w);
+ } break;
case PROJECTOR_TYPE_DOTS_OCR:
{
const int pw = image_size_width / patch_size;
return ctx->model.mm_ffn_down_w->ne[1];
case PROJECTOR_TYPE_GLM_EDGE:
return ctx->model.mm_model_mlp_3_w->ne[1];
+ case PROJECTOR_TYPE_MINIMAX_M3:
+ return ctx->model.mm_merger_fc2_b->ne[0];
case PROJECTOR_TYPE_QWEN2VL:
case PROJECTOR_TYPE_QWEN25VL:
case PROJECTOR_TYPE_EXAONE4_5:
--- /dev/null
+#include "models.h"
+
+ggml_tensor * clip_graph_minimax_m3::apply_rope(
+ ggml_tensor * x, ggml_tensor * pos_h, ggml_tensor * pos_w) {
+ const int64_t Hn = x->ne[1];
+ const int64_t P = x->ne[2];
+ const size_t es = ggml_element_size(x);
+ const int dh = (int) x->ne[0];
+ const int axd = 2 * ((2 * (dh / 2) / 3) / 2);
+
+ GGML_ASSERT(x->nb[0] == es);
+ GGML_ASSERT(3 * axd <= dh);
+
+ const float th = hparams.rope_theta;
+
+ // layout of x is [t, h, w, pad]
+ // t is unrotated, h and w are rotated, pad is unrotated
+ // note: everything from n_dims onward untouched, so w and pad are rotated in one call.
+ auto sl = [&](int off, int n) {
+ return ggml_cont(ctx0, ggml_view_3d(ctx0, x, n, Hn, P, x->nb[1], x->nb[2], (size_t) off * es));
+ };
+ ggml_tensor * t = sl(0, axd);
+ ggml_tensor * h = sl(axd, axd);
+ ggml_tensor * w = sl(2 * axd, dh - 2 * axd); // w + pad
+
+ h = ggml_rope_ext(ctx0, h, pos_h, nullptr, axd, GGML_ROPE_TYPE_NEOX, 0, th, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f);
+ w = ggml_rope_ext(ctx0, w, pos_w, nullptr, axd, GGML_ROPE_TYPE_NEOX, 0, th, 1.0f, 0.0f, 1.0f, 0.0f, 0.0f);
+ return ggml_concat(ctx0, ggml_concat(ctx0, t, h, 0), w, 0);
+}
+
+ggml_cgraph * clip_graph_minimax_m3::build() {
+ GGML_ASSERT(model.patch_bias == nullptr);
+ GGML_ASSERT(model.class_embedding == nullptr);
+ GGML_ASSERT(model.patch_embeddings_0 && model.patch_embeddings_1);
+ GGML_ASSERT(model.mm_1_w && model.mm_2_w);
+ GGML_ASSERT(model.mm_merger_fc1_w && model.mm_merger_fc2_w);
+
+ const int batch_size = 1;
+ const int n_pos = n_patches;
+ const int merge = hparams.n_merge;
+
+ // patch embedding
+ ggml_tensor * inp_raw = build_inp_raw();
+ ggml_tensor * inp = ggml_add(ctx0,
+ ggml_conv_2d(ctx0, model.patch_embeddings_0, inp_raw, patch_size, patch_size, 0, 0, 1, 1),
+ ggml_conv_2d(ctx0, model.patch_embeddings_1, inp_raw, patch_size, patch_size, 0, 0, 1, 1));
+
+ // spatial merge
+ {
+ inp = ggml_permute(ctx0, inp, 1, 2, 0, 3);
+ inp = ggml_cont_4d(ctx0, inp, n_embd * merge, n_patches_x / merge, n_patches_y, batch_size);
+ inp = ggml_reshape_4d(ctx0, inp, n_embd * merge, n_patches_x / merge, merge, batch_size * (n_patches_y / merge));
+ inp = ggml_permute(ctx0, inp, 0, 2, 1, 3);
+ inp = ggml_cont_3d(ctx0, inp, n_embd, n_patches_x * n_patches_y, batch_size);
+ }
+
+ // t (time axis) is always 0 for now, so we leave it unrotated
+ ggml_tensor * pos_h = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_pos);
+ ggml_set_name(pos_h, "minimax_pos_h"); ggml_set_input(pos_h);
+ ggml_tensor * pos_w = ggml_new_tensor_1d(ctx0, GGML_TYPE_I32, n_pos);
+ ggml_set_name(pos_w, "minimax_pos_w"); ggml_set_input(pos_w);
+
+ ggml_tensor * inpL = build_vit(
+ inp, n_pos, NORM_TYPE_NORMAL, FFN_GELU_ERF, nullptr,
+ [&](ggml_tensor * c, const clip_layer &) {
+ return apply_rope(c, pos_h, pos_w);
+ });
+
+ // projector
+ ggml_tensor * emb = inpL;
+ emb = build_ffn(emb, model.mm_1_w, model.mm_1_b,
+ nullptr, nullptr,
+ model.mm_2_w, model.mm_2_b, FFN_GELU_ERF, -1);
+
+ const int64_t proj = emb->ne[0];
+ emb = ggml_reshape_2d(ctx0, emb, proj * merge * merge, n_pos / (merge * merge));
+
+ emb = build_ffn(emb, model.mm_merger_fc1_w, model.mm_merger_fc1_b,
+ nullptr, nullptr,
+ model.mm_merger_fc2_w, model.mm_merger_fc2_b, FFN_GELU_ERF, -1);
+
+ ggml_build_forward_expand(gf, emb);
+ return gf;
+}
ggml_cgraph * build() override;
};
+struct clip_graph_minimax_m3 : clip_graph {
+ clip_graph_minimax_m3(clip_ctx * ctx, const clip_image_f32 & img) : clip_graph(ctx, img) {}
+ ggml_cgraph * build() override;
+ ggml_tensor * apply_rope(ggml_tensor * x, ggml_tensor * pos_h, ggml_tensor * pos_w);
+};
+
struct clip_graph_mimovl : clip_graph {
clip_graph_mimovl(clip_ctx * ctx, const clip_image_f32 & img) : clip_graph(ctx, img) {}
ggml_cgraph * build() override;
img_end = "<|vision_end|>";
image_preproc = std::make_unique<mtmd_image_preprocessor_dyn_size>(ctx_v);
} break;
+ case PROJECTOR_TYPE_MINIMAX_M3:
+ {
+ // ]<]start of image[>[ ... (image embeddings) ... ]<]end of image[>[
+ img_beg = "]<]start of image[>[";
+ img_end = "]<]end of image[>[";
+ image_preproc = std::make_unique<mtmd_image_preprocessor_dyn_size>(ctx_v);
+ } break;
case PROJECTOR_TYPE_YOUTUVL:
{
// <|vision_start|> ... (image embeddings) ... <|vision_end|>
overview / pkg / ggml / sources / llama.cpp / commitdiff