]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
clip : Add Qwen2.5VL support (#12402)
authorHimariO <redacted>
Sun, 27 Apr 2025 08:10:34 +0000 (16:10 +0800)
committerGitHub <redacted>
Sun, 27 Apr 2025 08:10:34 +0000 (10:10 +0200)
commitca2bb89eac2097ab4620448737e58af8452e444b
tree29df1ed10e78fbc11146e997dd65f9f9b33dbf3a
parent2d451c80590b9ac250322769ac13d3b4870dbcf7
clip : Add Qwen2.5VL support (#12402)

* implment vision model architecture, gguf convertor

* handle window attention inputs

* add debug utils

* fix few incorrect tensor memory layout

* move position id remap out of ggml to avoid int32 cuda operations

* cleaning up

* ignore transformers Qwen2_5_xxx type check

* remove not so often use `qwen2vl-cli` debug functions

* remove commented-out code blocks

* fix attn weight scaling after rebase

* add `PROJECTOR_TYPE_QWEN2_5_VL`

* remove `KEY_USE_GLU_MLP`, `KEY_USE_RMS_NORM`

* replace `KEY_FULLATTN_BLK_IDX` with `KEY_WIN_ATTN_PATTERN`

* remove `attn_window_size` from gguf

* fix model conversion

* clean up

* fix merging problem

* add test

---------

Co-authored-by: Xuan Son Nguyen <redacted>
convert_hf_to_gguf.py
examples/llava/clip-impl.h
examples/llava/clip.cpp
examples/llava/qwen2_vl_surgery.py
examples/llava/qwen2vl-cli.cpp
examples/llava/tests.sh