]> git.djapps.eu Git - pkg/ggml/sources/ggml/commit
ggml : add RPC backend (llama/6829)
authorRadoslav Gerganov <redacted>
Tue, 14 May 2024 11:27:19 +0000 (14:27 +0300)
committerGeorgi Gerganov <redacted>
Tue, 14 May 2024 16:13:20 +0000 (19:13 +0300)
commit48bb90e37f55a5f7aa8939b70f5569cd3ce2306f
treedabf02626d76d313bc9f021c995dcb64baaba3af
parentcc3c140a72f42d265606ccab39bf5f588789a959
ggml : add RPC backend (llama/6829)

* ggml : add RPC backend

The RPC backend proxies all operations to a remote server which runs a
regular backend (CPU, CUDA, Metal, etc).

* set TCP_NODELAY

* add CI workflows

* Address review comments

* fix warning

* implement llama_max_devices() for RPC

* Address review comments

* Address review comments

* wrap sockfd into a struct

* implement get_alignment and get_max_size

* add get_device_memory

* fix warning

* win32 support

* add README

* readme : trim trailing whitespace

* Address review comments

* win32 fix

* Address review comments

* fix compile warnings on macos
src/ggml-rpc.cpp [new file with mode: 0644]
src/ggml-rpc.h [new file with mode: 0644]