]>
git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
server: refactor + correctness fixes for metrics (#26920)
* server: refactor metrics
* move most fields to server_slot_stats
* cont
* rm result_timings
* tie stats to batch
* cont
* nits: move place in code
* exclude first generated token
* more accurate batch metrics tracking
* n_predict --> n_gen
* metrics_on_prediction
* metrics_flush_idle
* metrics: seperate cache/processed prompt tokens
* refactor server_task_result_metrics
* add test
* nits
* fix flush before reset()
* cont
* rm dead code
* nits