Skip to content

Test transcription shielding while retaining clean audio - #7682

Draft
ComputelessComputer wants to merge 2 commits into
mainfrom
chore/transcription-shield-experiment
Draft

ComputelessComputer wants to merge 2 commits into
mainfrom
chore/transcription-shield-experiment

Conversation

@ComputelessComputer

@ComputelessComputer ComputelessComputer commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

Add an isolated local experiment with model and codec comparisons, listening reports, and measurement checks. ANLG-393.


Summary by cubic

Implements ANLG-393 with an isolated, offline transcription-shield experiment. It fits a perturbation to one complete utterance while retaining the clean audio, then compares transcription and codec effects through reproducible artifacts.

Details

  • Adds WER, audio validation, SNR and length checks, deterministic noise control, safe WAV writing, and escaped HTML report generation.
  • Makes noise-control comparisons reproducible with a fixed seed and by rejecting shuffled perturbations that would clip instead of changing their distribution.
  • Compares tiny.en, base.en, or small.en models before and after a 24 kbps Opus round trip.
  • Runs with uv, requires ffmpeg, downloads Whisper models locally, and refuses existing output directories.
  • Adds tests for metric edge cases, clipping, WAV round trips, deterministic perturbations, and report escaping.
  • Limits the experiment to one utterance and Whisper-based listening and measurement; it does not evaluate real-time calls or human intelligibility.

Written for commit 8190a5b. Summary will update on new commits.

Review in cubic

@ComputelessComputer
ComputelessComputer marked this pull request as draft September 17, 2026 14:08

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread scripts/test_transcription_shield.py
Comment thread scripts/transcription_shield.py Outdated
@ComputelessComputer
ComputelessComputer force-pushed the chore/transcription-shield-experiment branch 2 times, most recently from 0500354 to 525260c Compare September 18, 2026 02:24
Add an isolated local experiment with model and codec comparisons, listening reports, and measurement checks. ANLG-393.
Declare the test environment and reject shuffled perturbations that would clip instead of changing their distribution. ANLG-393.
@ComputelessComputer
ComputelessComputer force-pushed the chore/transcription-shield-experiment branch from 47b538d to 8190a5b Compare September 18, 2026 04:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant