From: YavorGIvanov <redacted>
Date: Fri, 8 Sep 2023 13:17:44 +0000 (+0300)
Subject: Fix SAM example mask output with latest ggml
X-Git-Tag: upstream/0.0.1642~1247
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=ca21b435983c77cc4c008d7fd145151e6cb3b0cd;p=pkg%2Fggml%2Fsources%2Fggml

Fix SAM example mask output with latest ggml

- I am not sure why this inplace removal causes the output to turn
  correct again. I spend some time debugging and trying different
things, but my assumption is that some dependency is not properly
propagated and the allocator doesn't know about some tensor and
therefore decided to free it and overwrite its memory
- I also added commented out build_forward_expand, which also fixes
the output, but I am still not sure why
- Additionally I am still trying to figure out why the
  ggml_allocr_alloc(..) calls after the ggml_conv_transpose_2d_p0 are
needed
- I guess I have to spend some time debugging the ggml allocator and
  figure out what wrong is happening in this operations. Probably
something wrong in the operation implementation that I am unable to
notice.

Fixes #510.
---

diff --git a/examples/sam/main.cpp b/examples/sam/main.cpp
index a3469021..e15cbefd 100644
--- a/examples/sam/main.cpp
+++ b/examples/sam/main.cpp
@@ -1653,7 +1653,7 @@ bool sam_decode_mask(
                 struct ggml_tensor * q_0 = ggml_add(ctx0, queries, tokens);
 
                 struct ggml_tensor * self_attn = sam_decode_mask_transformer_attn(tfm_layer.self_attn, q_0, q_0, queries, ctx0, model);
-                queries = ggml_add_inplace(ctx0, queries, self_attn);
+                queries = ggml_add(ctx0, queries, self_attn);
             }
 
             queries = ggml_norm(ctx0, queries, hparams.eps_decoder_transformer);
@@ -1730,6 +1730,7 @@ bool sam_decode_mask(
     // ref: https://github.com/facebookresearch/segment-anything/blob/6fdee8f2727f4506cfbbe553e23b895e27956588/segment_anything/modeling/mask_decoder.py#L136
     keys = ggml_cont(ctx0, ggml_transpose(ctx0, keys));
     keys = ggml_view_4d(ctx0, keys, srcNE[0], srcNE[1], srcNE[2], srcNE[3], srcNE[0]*keys->nb[0], keys->nb[1], keys->nb[2], 0);
+    // ggml_build_forward_expand(gf, keys);
     struct ggml_tensor * upscaled_embedding = {};
     {
         // ConvTranspose2d