The audio was identical. The benchmark numbers changed.
Repeating a benchmark can reveal measurement instability without changing the model output. This five-profile Chatterbox Turbo rerun preserved all thirty comparison WAVs byte for byte.
Key finding
All 30 comparison WAV hashes matched the earlier run, while timings and four ASR transcripts changed.
Edited finding from the pinned experiment record. The report and figure metadata preserve the original workload and limits.
The boundary
This is a repeat of the same three sentences and one voice, not new linguistic coverage. CPU/CUDA recognizer environments differ.
Data
Reviewed aggregate results with explicit units, sources and qualification limits. Different cohorts remain separate.
Identical audio, new timing conditions
M5 · battery · Low Power Mode off · five warm repeats of one sentence
- Original FP322.861
- Community affine 6-bit1.535
- oQ61.489
- oQ51.445
- oQ41.389
warm median seconds
All 30 comparison WAV hashes match the earlier run. This is not additional prompt coverage.
View data & source
| Configuration | Value |
|---|---|
| Original FP32 | 2.861 |
| Community affine 6-bit | 1.535 |
| oQ6 | 1.489 |
| oQ5 | 1.445 |
| oQ4 | 1.389 |
Origin: reviewed report table. Revision 7e6fdbe3b1d1.
Write-ups
Report chatterbox-fidelity/retest-2026-10-09/README.md
A useful repeat
Repeating a benchmark can reveal measurement instability without changing the model output. This five-profile Chatterbox Turbo rerun preserved all thirty comparison WAVs byte for byte.
Faster timings, same speech
With Low Power Mode off, warm medians were 2.861 seconds for the original, 1.535 for community 6-bit, 1.489 for oQ6, 1.445 for oQ5 and 1.389 for oQ4. The controlled similarity and storage results remained attached to the same audio.
Four ASR transcripts changed between the earlier CUDA recognizer and this CPU environment. Consequently, some reported WER differences reflect the evaluator, rather than a new speech waveform. Power and background-resource conditions also changed, preventing a clean causal claim about the timing improvement.
The lesson
Freeze the generated artifacts as well as the text. Waveform hashes help distinguish a changed system output from a changed measurement. Community 6-bit remained the provisional fidelity/speed candidate, and oQ6 the closest of the oQ variants, with the same small-corpus limits.
Evidence and publication boundary
This is an edited account of the recorded experiment. The linked archive is private; the methods, aggregate findings and limits are included above. Reference recordings, raw conversations, authentication material and workstation configuration are excluded.
More in Speech & codecs
MOSS on a 3090: Q2 failed, AWQ4 stayed closer
Every profile began with the pinned BF16 generator. The comparison reused fixed reference codes and an FP32 decoder for the main quality measurements, keeping codec changes separate from generator quantization.
AWQ4 stayed close to the BF16 reference on WavLM and character errors; the sensitivity Q2 candidate was rejected.
Drama3: BF16 decoded the same codes about three times faster
The experiment replayed saved Spanish and English codes representing 21.5481 and 15.8824 seconds of audio. It compared FP32, BF16, mixed BF16 and two INT8 decoder variants on one RTX 3090.
BF16 reduced median decoder time from 390.07 to 129.54 ms in Spanish and from 292.42 to 96.88 ms in English.
Keeping FP32 and moving Supertonic to Triton
Rather than reduce the CPU graph precision again, this follow-up used NVIDIA Triton’s ONNX backend with CUDA execution and the unchanged FP32 graphs. The paired comparison retained both speed and acoustic checks.
The paired GPU test cut median first-audio time to about 0.35 seconds while preserving the measured ASR results.