]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
examples: support LLaVA v1.5 (multimodal model) (#3436)
authorM. Yusuf Sarıgöz <redacted>
Thu, 12 Oct 2023 15:23:18 +0000 (18:23 +0300)
committerGitHub <redacted>
Thu, 12 Oct 2023 15:23:18 +0000 (18:23 +0300)
commit370359e5baf619f3a8d461023143d1494b1e8fde
treeacfd94911cdb83780f7afc3a703b8abb31aa00e2
parent9e24cc6e2e589d405bd1720c400f5b0b9d0ca3ee
examples: support LLaVA v1.5 (multimodal model) (#3436)

* WIP: start implementing LLaVA

* rm scratch buf for now, will revert after cleanup

* LLaVA image encoder is working. will combine with llama

* Add llava inference code, but it's buggy. debugging

* LLaVA is working e2e, needs to optimize memory allocation + cleanup

* Use ggml_allocr + rm unnecessary code

* fix: crlf -> lf

* fix: new line at EoF

* fix: trailing whitespace

* Add readme

* Update readme

* Some cleanup

* Are you happy editorconfig?

* rm unused batch image preprocessing

* rm unused import

* fix: rm designated initializers

* introduce pad-to-square mode for non-square images

* are you happy editorconfig?

* gitignore /llava

* Handle cases where image file does not exist

* add llava target to Makefile

* add support for 13b model variant

* Maybe seed is unlucky?

* Check if apples are compared to apples

* are you happy editorconfig?

* Use temperature = 0.1 by default

* command line: use gpt_params_parse()

* minor

* handle default n_predict

* fix typo

* llava : code formatting, rename files, fix compile warnings

* do not use Wno-cast-qual for MSVC

---------

Co-authored-by: Georgi Gerganov <redacted>
15 files changed:
.gitignore
Makefile
common/common.cpp
common/common.h
common/stb_image.h [new file with mode: 0644]
examples/CMakeLists.txt
examples/llava/CMakeLists.txt [new file with mode: 0644]
examples/llava/README.md [new file with mode: 0644]
examples/llava/clip.cpp [new file with mode: 0644]
examples/llava/clip.h [new file with mode: 0644]
examples/llava/convert-image-encoder-to-gguf.py [new file with mode: 0644]
examples/llava/llava-surgery.py [new file with mode: 0644]
examples/llava/llava-utils.h [new file with mode: 0644]
examples/llava/llava.cpp [new file with mode: 0644]
ggml.c