Newsletter
Lab Notes
Free write-ups from the lab, with the question, the result and its limits. Open the linked experiment for graphs and downloadable measurements.
A benchmark harness that keeps dry runs out of the results
The new harness layer provides four kinds of runs: streamed API throughput, GGUF quant comparison, task evaluation through lm-eval, and command-level memory/load measurement. It uses standard-library Python for the core, with optional external model and…
MOSS on a 3090: Q2 failed, AWQ4 stayed closer
Every profile began with the pinned BF16 generator. The comparison reused fixed reference codes and an FP32 decoder for the main quality measurements, keeping codec changes separate from generator quantization.
Drama3: BF16 decoded the same codes about three times faster
The experiment replayed saved Spanish and English codes representing 21.5481 and 15.8824 seconds of audio. It compared FP32, BF16, mixed BF16 and two INT8 decoder variants on one RTX 3090.
Keeping FP32 and moving Supertonic to Triton
Rather than reduce the CPU graph precision again, this follow-up used NVIDIA Triton’s ONNX backend with CUDA execution and the unchanged FP32 graphs. The paired comparison retained both speed and acoustic checks.
Longer Spanish phrases exposed the voice-transfer cost
Six phrases per language were synthesized with the production Supertonic stack, converted, then scored with an independent Parakeet recognizer. The experiment used public demo narrator references and did not alter the production application.
More diffusion steps did not reliably improve the voice match
The complete indexed collection contained fifty-one converted variants. Each was independently embedded against its exact selected narrator and source waveform. Twenty-nine used four steps, eleven eight and eleven twelve.
Supertonic: half the storage did not mean faster CPU speech
Three variants ran the same twelve phrases per language at eight synthesis steps, twice each: FP32, FP16 with the duration component retained, and a mixed INT8 graph. The latter quantized only the verified vector-estimator path; a directory label was not…
Voice resemblance improved before call readiness did
Seed-VC2 converted existing Supertonic speech with four, eight and twelve diffusion steps. The selected public audiobook references were thirty seconds long, but the upstream wrapper used only the first twenty-five seconds.
A faithful decoder can still be the slower pipeline
The complete community 6-bit MLX route had a 1.359-second warm median and RTF 0.225. The ordinary CoreML/ONNX Python adapter measured 4.974 seconds and RTF 0.749. Their payloads were 0.524 and 3.331 GiB respectively, with process RSS peaks of 1.181 and 4.963…
A local LatAm conversion failed the content gate
A local MLX conversion of the LatAm package was compared with the existing multilingual Q8 and an original FP32 route. All active weights were loaded strictly, but that alone did not establish correct speech.
Holding speech tokens fixed made the Q8 advantage clearer
This follow-up applied the Turbo method to Multilingual FP32, Q8 and Q6. Thirty-six full generations and thirty-six controlled decoder outputs formed the primary cohort; warm timing and FP32 self-repeats were retained separately.
The decoder swap passed its word-error gate
Both routes used the same community 6-bit MLX T3 generator. The hybrid replaced only the audio stage with the unchanged published ONNX conditional decoder. No CoreML execution occurred in this hybrid.
MOSS: the best similarity score did not mean the fewest word errors
Nine generator profiles were tested against BF16 plus the same Argentina LoRA. The English study retained ninety complete generations, controlled decoder runs and a timing recheck. A separate Spanish extension used one generation per phrase.
Q6 saved space; predicted MOS did not settle naturalness
Q6 and Q8 quantized the same 651 main matrices and 37 external tokenizer matrices in execution. Unsupported tensors stayed FP32. The Q6 checkpoint was produced directly from the original FP32 source.
The audio was identical. The benchmark numbers changed.
Repeating a benchmark can reveal measurement instability without changing the model output. This five-profile Chatterbox Turbo rerun preserved all thirty comparison WAVs byte for byte.
What six bits preserved in Chatterbox Turbo
Five checkpoints were compared in two cohorts. Complete generations allow speech-token choices and pacing to change. Controlled decoder runs reuse the original tokens and conditioning, allowing a narrower acoustic comparison.
An unfinished gauntlet cannot name a winner
This campaign compared Qwen UD-Q4_K_XL with MTP2 against Bonsai through coding harnesses. It stopped at the 70 GiB free-space guard, retaining the incomplete evidence rather than risking the workstation.
Native AWQ saved idle memory and cost speech time
This follow-up separates native AutoRound AWQ from the earlier rejected emulations. It measures whether reduced residency also improves the speech request.
Spanish Chatterbox: smaller weights, uneven speech
The original BF16 baseline had recorded WER 0.42 and real-time factor 0.76 in this small Spanish screen. A dynamic, emulated FP8 path recorded WER 0.24 and an isolated RTF around 1.05. The user favored that candidate during listening, which is a personal…
The fastest kernel was not the qualified speech stack
The fitting setup combined HQ4, SDPA, the INT8 GPU codec, 0.8-second chunks, a 200-character budget and a 256-step cap. The BF16 comparison peaked at about 22.8 GiB, beyond the RTX 3060.
144 recoveries, one carefully bounded task
The earlier recovery experiment proved that the pieces connected. This follow-up asks whether adding recent conversation history helps a worker resume after an independent reviewer has repaired the code.
A context setting is not a context test
A server can accept a large context configuration without demonstrating useful work at that length. I separated configured capacity, an actual request, and a successful result before making another speed claim.
Making MOSS fit: the codec and the chunk budget
The HQ4 generator occupied about 6.24 GiB and the codec about 3.31 GiB. A one-shot path reached roughly 13 GiB and ran out of memory; 0.8-second streaming chunks kept the resident stack near 9.55 GiB on the 12 GB RTX 3060.
A faster speech encoder with the same 50 transcripts
Could an RTX 3060 transcribe the same speech with less GPU memory by accelerating the encoder while retaining the existing decoder?
Why a busy speech GPU felt slower
A fast isolated encoder does not guarantee a responsive shared service. I tested ASR, embeddings and speech synthesis together on the RTX 3060 and retained failures and health checks.
A smaller speech model earns its place on the iPhone
The model that feels fastest after loading may still make you wait for its first words. This paired iPhone experiment compared English and Spanish transcription, kept a suspended timing sample out of both engines, and weighed accuracy against first-use…
A coding agent that knows when to ask for help
A coding agent can keep repeating a failed approach. This experiment gave a local Qwen worker an independent reviewer, then checked whether it could recover without weakening the final acceptance tests.
Corrective data meets a fresh gameplay gate
Corrective trajectories were added to a fresh same-host training comparison on Ubuntu CPU. The initial and repeated results stay separate, including the control’s changed weights, so the result is not presented as a direct improvement over the earlier Mac…
When the retention gate says stop
Adding new behavior is only useful if the model keeps the behavior it already has. This training experiment put a public retention floor in front of promotion and required a rollback when a candidate damaged the parent.
A second dish, a new frozen test
A first-dish policy did not automatically know how to reach the second dish. The new paired frozen test rejected the first candidate, then evaluated the corrected candidate against the same unchanged baseline.
Eight clears are still not a promotion
A full-kitchen clear needed survival and next-stage readiness as well as reaching the objective. The Mac protocols measured partial success, but the strict ten-of-ten gate kept both candidates from promotion.
Learning to play from pixels
The visual policy had to act from owned frames and recent controls rather than hidden emulator state. The first-dish baseline established a small replayable milestone; the later curricula ask harder questions without rewriting that first result.
The reusable RGBA learning pipeline
The reusable pipeline joins owned RGBA frames, bounded controls, teacher data, learned policies and immutable replay evidence. Pluto is the exercised target; the notebook does not turn planned support for other games into demonstrated gameplay.
What counts as a successful episode?
An episode can reach a goal and still fail its acceptance contract. These evaluator fixes tightened survival, terminal-state and replay checks while keeping historical outcomes attached to their original gates.
Finding a local model for the inbox
Inbox work needs accurate structured answers and reasonable response times. The benchmark ran the same 50 tasks across local model, runtime and hardware configurations, with thinking on and off, to make that tradeoff visible.
Teaching a 4B model to read tmux status
The tmux status problem is narrow enough to ask whether a small trained classifier can handle it locally. The project followed teacher-labeled examples through QLoRA, merged weights, quantization and a lazy service, while retaining the limits of that teacher…
An image editor that respects the pixels you keep
A masked edit should leave the protected part of an image alone. The studio project checked a real object removal, compared decoded pixels outside the mask, and traced the edit through a working tool flow.
Character LoRAs: identity, motion, and continuity
Character adapters need more than completed training. This project records the initial identity and motion runs, later action probes, continuity work and identity corrections, while leaving private reference media outside the article.
Fifteen pictures, one photographic-style adapter
The style-adapter experiment started with a large mixed image folder and a much smaller selected dataset. Deduplication, actual selected images and training provenance matter more than calling the result a single-person identity adapter.
Making the keyboard lock key really sleep the phone
The keyboard lock key left the phone displaying lock-screen pixels. The device experiment traced the keyboard-specific handler and power transition, checked the display state directly, and left reboot persistence as a separate follow-up.
Masked cleanup before a fresh LoRA
Cleanup happened before a fresh adapter was trained. The experiment saved lossless copies, verified unchanged pixels outside each mask, and documented that the resulting dataset still contained mixed subjects.
Moving speech decode onto the GPU
Moving the final audio decode onto the GPU shortened both that stage and total render time. The deployment A/B also records the extra GPU memory, so the speed improvement can be weighed against its cost.
Putting a watermark gate in front of training
A binary watermark screen became a gate in the training workflow. The project checked known positive controls and recorded the excluded image without treating a model label as infallible visual proof.
A complete image stack on one 24GB GPU
An image stack that fits on one GPU still has to produce useful pictures. This project measured the full pipeline and paired FP8 with opt-in INT8 across eight fixed scenes, keeping changed visual details beside the timing gain.
Long speech, bounded memory
Long input needed a bounded rendering path. The live service split an eight-sentence request into chunks and checked the resulting audio, while documenting the separate CLI run that remained deferred.
Round-trip tests expose a broken encoder
A round-trip test challenged the assumption that a codec was healthy because its decoder worked. The record compares the encoder-plus-decoder path and qualifies the new FP8 implementation on the named clips.
The clone bug was in the reference encoder
The clone quality problem was narrowed to the reference encoder. Changing that stage to FP32 recovered the measured target similarity while the output decoder stayed INT8, leaving accent and listening questions open.
The faster image pipeline that produced nothing
The two-GPU path appeared faster until the outputs were inspected. Every image was invalid, so the experiment records a failure and an unresolved sharding suspect rather than promoting the shorter timing.
The same story, six precision combinations
The same text made it possible to compare generator and codec precision separately. The six combinations exposed a small decoder similarity effect alongside a much more audible difference in pacing.
Balancing GPU experts when the model will not fit
The quant was larger than the three GPUs could hold on their own. The experiment tested explicit GPU expert splits with CPU offload and measured the decode effect under the same large-context workload.
Pipeline parallelism, INT4, and a 262K context
Pipeline parallelism and INT4 offered a different serving route for the large model. This record qualifies its long context, single-stream decode and two-stream aggregate throughput, with vision and MTP disabled.
A speech service that sleeps between requests
A speech service need not hold all of its resources between requests. The lazy-server experiment checked wake-up, warm rendering, reap and re-wake, while preserving the known short-prompt silence behavior.
A quantization job goes silent
A full-spec quantization job stopped making progress after only six blocks. The incident record follows the monitor, idle GPU and checked failure explanations without inventing a confirmed cause.
AWQ speech: fast enough, with a memory catch
AWQ speech delivered a useful render rate, but fitting a smaller GPU was still a separate question. The benchmark kept prompt health, pooled real-time factor, decode speed and peak memory in the same record.
Do you really need the bigger embedding model?
The larger embedding model asks for more memory and compute. This paired bilingual retrieval experiment checked whether the extra cost bought a detectable quality improvement on the actual evaluation set.
FP32, BF16, INT8: separating codec drift from speech
Codec precision can change storage, waveform detail and reference encoding in different ways. The A/B/C experiment separated those effects using pure speech and clone cases, rather than letting transcription parity stand in for every kind of fidelity.
Keeping the audio-sensitive layers intact
The HQ4 preview tried to save memory while protecting layers that mattered to audio. Text-token fidelity, codec behavior and staged-memory probes were checked separately, with calibration and physical-device limits still open.
Staging the speech stack under a 12GB envelope
The speech stack exceeded the target memory envelope before staging. This hardening experiment verified protected layers and quantized projections, then measured staged memory on the hardware actually available.
Three draft tokens hit the sweet spot
More draft tokens can add work as well as save it. The sweep compared several windows on the same short-prose workload, then kept the longer prompt and concurrent-stream measurements separate.
From a model download to a working local service
A downloaded model does not yet make a useful service. This deployment notebook follows formats, cache settings, routing and resource constraints; its dated experiments explain which configuration produced each result.
The EXL3 quantization journey
The EXL3 work began with official BF16 weights and followed the conversion into a deployable artifact. The journey keeps the intermediate checks and remaining work visible rather than declaring success from a running process.
A model proxy earns its promotion
A proxy must keep its serving contracts while managing several local model families. This promotion record follows the merge through regression checks and separates new live validation from code-reviewed operational notes.
The best checkpoint was not the last one
Training longer did not make every speech checkpoint better. The fixed Spanish transcription screen compared the saved steps, found an earlier best checkpoint, and retained the distinction between word accuracy and how a voice sounds.
Rebuilding calibration for an agentic quant
Calibration was rebuilt around agentic work rather than assumed adequate from a generic matrix. The hard task, new quantization artifacts and subsequent draft-decoding probes stayed separate evidence gates.
A draft head, a rollback fix, and a midnight incident
Draft decoding exposed a rollback issue, and the fix needed a fresh comparison at different sampling settings. The later disk incident is retained because it invalidated the scheduled long-context validation.
Making room for a 200K context
Getting a large model to load is the first hurdle; completing a nearly full context is another. This experiment qualified a specific three-GPU profile with a warm decode measurement and a real long-context request.
DFlash2 on a single RTX 3090
Speculative decoding needs to earn its speedup on more than an easy completion. This single-GPU DFlash2 record separates predictable prose from a realistic prompt and records the effect of draft acceptance.
Less thinking, fewer tokens: a template A/B
A chat template can change how much a model writes, even when the weights stay the same. The paired test measured accuracy, completion tokens, total tokens and wall time across stock and Sharp templates.
The real cost of a five-second video
Five seconds of generated video can take many minutes of compute. The H3 archive groups real 20-step runs by resolution and keeps quantization, placement, mode and complete decode checks attached to each timing.
Free Lab Notes
Follow the next experiment
Get new findings in your feed reader. All published articles and measurement exports are free to read.
Follow via RSS