From: tarcey <redacted>
Date: Thu, 30 Nov 2023 21:40:23 +0000 (+0100)
Subject: llama : fix typical sampling (#4261)
X-Git-Tag: upstream/0.0.4488~2908
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=954e22858c5cea1dc03e9172d3879402af2b5990;p=pkg%2Fggml%2Fsources%2Fllama.cpp

llama : fix typical sampling (#4261)

Typical sampling was broken because after copying new_candidates into canditates, the "sorted" bool is left at "true", but the new data is no longer sorted according to probability. Patch to set "sorted" to false.

Test: Generating with temp=0.0001 (approx. argmax)  should generate the same sequence at typical>=1.0 and typical=0.9999 (approx. disabled, but enters the typical sampling codepath).
---

diff --git a/llama.cpp b/llama.cpp
index cb544228..4af45061 100644
--- a/llama.cpp
+++ b/llama.cpp
@@ -7027,6 +7027,7 @@ void llama_sample_typical(struct llama_context * ctx, llama_token_data_array * c
     // Replace the data in candidates with the new_candidates data
     std::copy(new_candidates.begin(), new_candidates.end(), candidates->data);
     candidates->size = new_candidates.size();
+    candidates->sorted = false;
 
     if (ctx) {
         ctx->t_sample_us += ggml_time_us() - t_start_sample_us;