]> git.djapps.eu Git - pkg/ggml/sources/whisper.cpp/commit
tests : add a new benchmark test for long-form audio (#3185)
authorFujimoto Seiji <redacted>
Wed, 28 May 2025 05:08:44 +0000 (14:08 +0900)
committerGitHub <redacted>
Wed, 28 May 2025 05:08:44 +0000 (07:08 +0200)
commitb9d27b13581d721b33d880a18853f5594e6d77a2
treea9c4bedbf667c0fbf99142fc15b24ea5522539d8
parent0ed00d9d30e8c984936ff9ed9a4fcd475d6d82e5
tests : add a new benchmark test for long-form audio (#3185)

* tests : add a new benchmark test for long-form audio

Based on "Earnings-21" corpus by Del Rio et al.

    Earnings-21: A Practical Benchmark for ASR in the Wild (2021)
    https://arxiv.org/abs/2104.11348

This dataset contains 39 hours of long-form speech, sourced from public
earning calls. Each recording contains roughly 50 minutes of English
dialogues between multiple speakers (2-20 persons).

This benchmark suite should allow us to evaluate the performance of
whisper.cpp on long-form audio data.

Signed-off-by: Fujimoto Seiji <redacted>
* tests : apply PR feedback to 'earnings21/README.md'

Based on feedback from Daniel Bevenius.

 - Simplify how to download & prepare a Silero VAD model.
 - Fix typo: inferece -> inference

Signed-off-by: Fujimoto Seiji <redacted>
* tests : avoid crashing on non-UTF-8 characters

Based on feedback from Daniel Bevenius.

Add 'errors' parameter to open() in order to avoid unhandled
exception on invalid UTF-8 bytes.

Signed-off-by: Fujimoto Seiji <redacted>
* tests : try to interpret the hypothesis as Windows-1252

Based on the discussion in PR#3185.

Evidently Whisper.cpp can represent a quotation mark as '0x93', which
implifies Windows-1252 (Microsoft's ASCII excention), and cannot be
decoded by UTF-8.

Add an explicit decoding loop to address the issue.

Signed-off-by: Fujimoto Seiji <redacted>
---------

Signed-off-by: Fujimoto Seiji <redacted>
tests/earnings21/.gitignore [new file with mode: 0644]
tests/earnings21/Makefile [new file with mode: 0644]
tests/earnings21/README.md [new file with mode: 0644]
tests/earnings21/eval.mk [new file with mode: 0644]
tests/earnings21/eval.py [new file with mode: 0644]
tests/earnings21/normalizers/LICENSE [new file with mode: 0644]
tests/earnings21/normalizers/__init__.py [new file with mode: 0644]
tests/earnings21/normalizers/basic.py [new file with mode: 0644]
tests/earnings21/normalizers/english.json [new file with mode: 0644]
tests/earnings21/normalizers/english.py [new file with mode: 0644]
tests/earnings21/requirements.txt [new file with mode: 0644]