]> git.djapps.eu Git - pkg/ggml/sources/llama.cpp/commit
batched-bench : handle empty `-npl` (#8839)
authorBrian Cunnie <redacted>
Sun, 4 Aug 2024 10:55:03 +0000 (03:55 -0700)
committerGitHub <redacted>
Sun, 4 Aug 2024 10:55:03 +0000 (13:55 +0300)
commitecf6b7f23e664afd7ff856ec39034240ce438daa
tree7ed839be99f5825ecf7efb5d202a22937dcd1e37
parent01aae2b4975b57a265ce8194928fd87f2d71027e
batched-bench : handle empty `-npl` (#8839)

* [example] batched-bench "segmentation fault"

When `llama-batched-bench` is invoked _without_ setting `-npl`, "number
of parallel prompts", it segfaults.

The segfault is caused by invoking `max_element()` on a zero-length
vector, `n_pl`

This commit addresses that by first checking to see if the number of
parallel prompts is zero, and if so sets the maximum sequence size to 1;
otherwise, sets it to the original, the result of `max_element()`.

Fixes, when running `lldb build/bin/llama-batched-bench -- -m models/Meta-Llama-3-8B.gguf`

```
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x000000010000366c llama-batched-bench`main(argc=3, argv=0x000000016fdff268) at batched-bench.cpp:72:28
   69       llama_context_params ctx_params = llama_context_params_from_gpt_params(params);
   70
   71       // ensure enough sequences are available
-> 72       ctx_params.n_seq_max = *std::max_element(n_pl.begin(), n_pl.end());
```

* Update examples/batched-bench/batched-bench.cpp

Co-authored-by: compilade <redacted>
---------

Co-authored-by: Georgi Gerganov <redacted>
Co-authored-by: compilade <redacted>
examples/batched-bench/batched-bench.cpp