]> git.djapps.eu Git - pkg/ggml/sources/ggml/commit
overview / pkg / ggml / sources / ggml / commit
summary | shortlog | log | commit | commitdiff | tree
(parent: 7734363)
rpc : do not wait for response when sending RPC_CMD_SET_TENSOR (llama/12943)
authorRadoslav Gerganov <redacted>
Fri, 25 Apr 2025 07:08:08 +0000 (10:08 +0300)
committerGeorgi Gerganov <redacted>
Thu, 1 May 2025 07:39:34 +0000 (10:39 +0300)
commitc9207d32fd7fe8d2e8f3dd5cb3f3eca7029ffd43
tree13ccc142f107c51c51fd4df1b8150d0c67a3543btree
parent7734363ac74a14d7f2c8033c552d9d655c5dfc00commit | diff
rpc : do not wait for response when sending RPC_CMD_SET_TENSOR (llama/12943)

RPC_CMD_SET_TENSOR always returns an empty response and we send this 4
times per token. We can improve TG speed if we don't wait for this empty
response.

The performance impact of this change depends on the network latency.
include/ggml-rpc.h diff | blob | history
src/ggml-rpc/ggml-rpc.cpp diff | blob | history
Packaging of ggml-org/ggml