]> git.djapps.eu Git - pkg/ggml/sources/whisper.cpp/commit
ggml : add RPC backend (llama/6829)
authorRadoslav Gerganov <redacted>
Tue, 14 May 2024 11:27:19 +0000 (14:27 +0300)
committerGeorgi Gerganov <redacted>
Tue, 14 May 2024 16:16:29 +0000 (19:16 +0300)
commitc451080c8b0e2080f2ca887047ef381b94523e14
treec0a5786ed1c2db18940c496e76b0e3e9457d4c0a
parent8e7c22fbdbc27c1c72abd192720e330e7f6361a9
ggml : add RPC backend (llama/6829)

* ggml : add RPC backend

The RPC backend proxies all operations to a remote server which runs a
regular backend (CPU, CUDA, Metal, etc).

* set TCP_NODELAY

* add CI workflows

* Address review comments

* fix warning

* implement llama_max_devices() for RPC

* Address review comments

* Address review comments

* wrap sockfd into a struct

* implement get_alignment and get_max_size

* add get_device_memory

* fix warning

* win32 support

* add README

* readme : trim trailing whitespace

* Address review comments

* win32 fix

* Address review comments

* fix compile warnings on macos
ggml-rpc.cpp [new file with mode: 0644]
ggml-rpc.h [new file with mode: 0644]