]>
git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
server: add auto-sleep after N seconds of idle (#18228)
* implement sleeping at queue level
* implement server-context suspend
* add test
* add docs
* optimization: add fast path
* make sure to free llama_init
* nits
* fix use-after-free
* allow /models to be accessed during sleeping, fix use-after-free
* don't allow accessing /models during sleep, it is not thread-safe
* fix data race on accessing props and model_meta
* small clean up
* trailing whitespace
* rm outdated comments
12 files changed:
overview / pkg / ggml / sources / llama.cpp / commit