]> git.djapps.eu Git - pkg/ggml/sources/whisper.cpp/commit
cli : merge tokens split across UTF-8 boundaries in JSON output (#3751)
authortexasich <redacted>
Tue, 26 May 2026 04:23:41 +0000 (23:23 -0500)
committerGitHub <redacted>
Tue, 26 May 2026 04:23:41 +0000 (06:23 +0200)
commit27101c01dcac1676e2b6422256233cd0f1f9ae28
treefee3c6e4a1cdee179eb31091c72d51a10ade755c
parente0fd1f6787a5bd4a4957dd97c5b64df882ee7b0c
cli : merge tokens split across UTF-8 boundaries in JSON output (#3751)

* cli : merge tokens split across UTF-8 boundaries in JSON output

When a multi-byte UTF-8 codepoint (most commonly a CJK character, 3 bytes)
is split across multiple whisper tokens, the -ojf/--output-json-full
writer emitted each token's partial bytes as its own JSON string, producing
invalid UTF-8 that chokes downstream parsers.

Merge adjacent tokens in output_json whenever the accumulated text still
ends on an incomplete UTF-8 sequence. The merged entry keeps the first
token's id/p/t_dtw and extends t1 to the last absorbed token, which
matches how segment text is assembled elsewhere.

Refs #1798

* fix: address review — add braces for consistency, use full issue URL

- Add braces to if/else chain for codebase consistency
- Use full URL for issue #1798 reference

Review: @danbev

---------

Co-authored-by: texasich <redacted>
Co-authored-by: texasich <redacted>
examples/cli/cli.cpp