From: Galunid <redacted>
Date: Sat, 25 Nov 2023 21:45:02 +0000 (+0100)
Subject: scripts : Use mmap in torch load (#4202)
X-Git-Tag: upstream/0.0.4488~2922
X-Git-Url: https://git.djapps.eu/?a=commitdiff_plain;h=1ddb52ec38f9931925a587f45a23b1c37152c028;p=pkg%2Fggml%2Fsources%2Fllama.cpp

scripts : Use mmap in torch load (#4202)

* Use mmap in torch load, prefer .bin files when loading

* Revert .bin > .safetensors preference
---

diff --git a/convert-hf-to-gguf.py b/convert-hf-to-gguf.py
index 147d5717..53ce76c7 100755
--- a/convert-hf-to-gguf.py
+++ b/convert-hf-to-gguf.py
@@ -59,7 +59,7 @@ class Model:
                 from safetensors import safe_open
                 ctx = cast(ContextManager[Any], safe_open(self.dir_model / part_name, framework="pt", device="cpu"))
             else:
-                ctx = contextlib.nullcontext(torch.load(self.dir_model / part_name, map_location="cpu"))
+                ctx = contextlib.nullcontext(torch.load(str(self.dir_model / part_name), map_location="cpu", mmap=True, weights_only=True))
 
             with ctx as model_part:
                 for name in model_part.keys():