Speak Savvy

Problem

Self-review of a recorded practice answer doesn’t work without instrumentation. A speaker replaying their own audio will not reliably notice a filler-word rate of 6/min, a 40 WPM pace spike three sentences in, or a flattened pitch range — these are exactly the signals a human ear tunes out in real time because it’s tracking content, not delivery. A concrete case: a 44-second STAR-format answer that scores 5/5 on Fluency and Relevance can still score 1/5 on Structure because it skips the Task section entirely — a defect invisible without a parser checking for section presence, not just listening for smoothness. Professional coaching catches this kind of thing but costs per session; unassisted practice catches none of it.

Approach

The design bet is that the delivery half of the feedback — filler rate, pace, pitch range, voice quality — should be scored deterministically, never through a model, so the same recording always produces the same numbers; content-judgment feedback (relevance, structure, coaching prose) is where a model earns its keep. The two are architecturally separated: ScoringRouter routes delivery metrics to a fixed .deterministic tier that never touches a model, computed entirely on-device via Accelerate/vDSP and Apple’s Speech framework. Audio bytes never leave the device — transcription runs on-device (SpeechAnalyzer on iOS 26, SFSpeechRecognizer as fallback) and pitch/volume/voice-quality extraction runs on-device (native YIN, RMS, CPP). Content-judgment scoring is tiered: Apple’s on-device Foundation Models (~3B) when offline, Groq (default) or OpenAI (fallback) through an authenticated Cloudflare Worker proxy when online — the transcript and derived metrics, not raw audio, are what reach the cloud tier, and only when that tier is active. What was deliberately not built: a single cloud-only pipeline (rejected because delivery metrics don’t need model judgment and paying LLM latency/cost for a WPM calculation is wasted spend) and a fully offline-only pipeline (rejected because the on-device 3B model is measurably weaker on nuanced content feedback than Groq/OpenAI, and most sessions happen with connectivity available). Also not built: cross-device sync or a backend account — Sign in with Apple establishes identity only, session history is local-only via SwiftData, trading continuity for a minimal backend surface (the Worker proxy exists to hide LLM keys, not to store user data).

Architecture

Two pipelines run off the same recording: a deterministic delivery-metrics path that never touches a model, and a tiered content-judgment path that picks on-device Apple Foundation Models, Groq, or OpenAI depending on connectivity and drill type. ScoringRouter is the seam — it maps each scoring workload (deliveryMetrics, semanticCoaching, structureExtraction) to an ordered list of tiers (deterministic → onDeviceLLM → cloudSmall → cloudLarge) and only tiers whose capability flag is actually true get included, so the routing decision is explicit and testable rather than implicit in call-site branching.

flowchart TD
    A[AVAudioEngine tap] --> B["AudioAnalyzer: YIN pitch / RMS / CPP<br/>Accelerate+vDSP, on-device, deterministic"]
    A --> C["SpeechAnalyzer (iOS 26, on-device)<br/>falls back to SFSpeechRecognizer"]
    C -->|transcript + word timestamps + confidence| D["SpeechMetricsEngine<br/>fuses ASR + DSP -> SpeechMetrics"]
    B --> D
    D -->|WPM, fillers/min, pitch/volume/CPP stats -<br/>never sent to a model| K
    D --> E{"AIRouter: tier decision<br/>online + Apple Intelligence availability"}
    E -->|offline / degraded| F["Apple Foundation Models (~3B)<br/>on-device LanguageModelSession"]
    E -->|enhanced, online| G["Cloudflare Worker proxy<br/>speaksavvy.ai/api/llm/*"]
    G -->|default| H["Groq: Llama 3.3 70B (prose)<br/>GPT-OSS 20B/120B (strict JSON)"]
    G -.fallback.-> I["OpenAI: gpt-4o-mini / gpt-4o"]
    H --> J["CloudStructureExtractor:<br/>grounds each LLM segment against the<br/>actual transcript text, deterministically"]
    I --> J
    F --> K["CoachFeedbackParser -> ScoreRubric"]
    J --> K
    K --> L[("SwiftData: Session,<br/>ScoreRubric, SpeechMetrics JSON")]
    L --> M[Score breakdown + vocal delivery UI]
    E -.Live Voice, opt-in.-> N["OpenAI Realtime gpt-realtime-2.1-mini<br/>WebSocket via Worker relay"]
    E -.coach playback.-> O["OpenAI TTS gpt-4o-mini-tts<br/>-> AVSpeechSynthesizer offline"]

Audio bytes never cross the network in this diagram. The transcript and derived metrics reach the Worker proxy only on the enhanced tier, and the proxy holds the OpenAI/Groq keys server-side — the app never ships an LLM provider key, only a rotatable bearer token.

Models — which, where, why:

Stage	Model / tier	Where it runs	Why there
Delivery metrics (pitch, volume, voice quality, activity)	None — deterministic YIN, RMS, CPP via Accelerate/vDSP	`AudioAnalyzer`, on-device, real-time buffer callback	`ScoringRouter` routes `deliveryMetrics` to the `.deterministic` tier only — these numbers must reproduce exactly on replay, and a model adds cost/latency with no accuracy benefit for a signal-processing calculation
Transcription	Apple SpeechAnalyzer (iOS 26, on-device); falls back to SFSpeechRecognizer on older assets	`SpeechService`	On-device ASR keeps raw audio off the network unconditionally, regardless of coaching tier
Filler/hedge detection	None — hand-rolled lexicon set-match (`fillerPatterns`, `multiWordFillers`) over ASR tokens	`SpeechMetricsEngine`	Deterministic word-list match, not the Natural Language framework or an ML classifier — same transcript, same filler count, every time
Offline coaching + rubric	Apple Foundation Models (on-device, ~3B, `LanguageModelSession`, guided generation on iOS 26)	`CoachService.getOfflineDrillFeedback`, when offline or the cloud call degrades	Zero-network, zero-marginal-cost coaching tier; the PRD’s “on-device-first” stance means a degraded proxy call falls back here rather than erroring
Cloud coaching — conversational turn (prose, latency-sensitive)	Groq Llama 3.3 70B Versatile	`CoachService`, `TaskRouter.conversationTurn`	Groq’s throughput (documented ~394 tok/s in-repo) matters most for the interactive turn, where no structured schema is required
Cloud coaching — drill rubric / structured scoring (strict JSON)	Groq GPT-OSS 20B (default), escalating to GPT-OSS 120B or OpenAI gpt-4o when structure confidence is low	`CoachService`, `TaskRouter.drillRubric` / `.drillCoaching` / `.quickPractice`	Groq-first per the PRD’s cost stance; GPT-OSS supports strict JSON Schema so the load-bearing rubric fields (scores, structure map) can’t silently drift out of contract
Cloud coaching fallback provider	OpenAI gpt-4o-mini (default), gpt-4o (quality escalation)	`CoachService`, when Groq is unreachable or the user sets `ProviderPreference.preferOpenAI`	Fallback provider — “Groq is the default cloud tier; OpenAI is the fallback when Groq quality disappoints” (PRD)
Structure (STAR/PREP) validation	None — `CloudStructureExtractor` grounds each LLM-proposed segment by locating it verbatim in the transcript before accepting it	Runs on the cloud rubric response, on-device	No framework verdict from the model is trusted un-validated; a segment the model claims exists but isn’t findable in the transcript is dropped
Live Voice speech-to-speech (opt-in drill mode)	OpenAI Realtime, `gpt-realtime-2.1-mini`	`RealtimeVoiceService`, WebSocket via the Worker relay (`wss://.../api/realtime/ws`)	Cloud-only real-time speech-to-speech; gated behind explicit mic-to-OpenAI consent and an online check, kept separate from the default turn-based STT→LLM→TTS loop
Text-to-speech (coach voice)	OpenAI `gpt-4o-mini-tts` (cloud, `enhanced` tier); AVSpeechSynthesizer (on-device, `offline` tier)	`TTSService`	`gpt-4o-mini-tts` supports tone/pace/emphasis direction that older `tts-1` models don’t; AVSpeechSynthesizer covers the offline tier with zero network dependency

Tools & infra — which, why:

SwiftData — local persistence for Session, UserSettings, Recording, ConversationTurn, Achievement; SpeechMetrics and ScoreRubric are JSON-encoded fields on Session. No server-side app database — single-device by design.
Cloudflare Workers (speaksavvy-site) — serves the static marketing site and, under /api/llm/*, an authenticated LLM proxy that holds the OpenAI and Groq API keys server-side and injects them per request; the app authenticates with a rotatable bearer token, never an LLM key.
Cloudflare KV (LLM_RESPONSE_CACHE) — response caching at the proxy layer.
Cloudflare Durable Objects (RateLimiter) — per-token rate limiting on the proxy, since keys are shared server-side across all app installs.
Cloudflare D1 (shared rosslabs-waitlist database) — used only by the marketing site’s early-access waitlist, tagged products='speaksavvy'; not app or session data, and not owned by this repo (schema lives in ross-labs-astro).
Keychain — stores the proxy bearer token on-device (via Config.local.xcconfig at build time); legacy API-key fields exist only for a SwiftData migration path and are unused at runtime.
Sign in with Apple — establishes a local identity at onboarding; there is no backend account and no cross-device sync.
Accelerate / vDSP — real-time on-device DSP backing the YIN pitch, RMS volume, and CPP voice-quality extraction, with pre-allocated FFT buffers and an os_unfair_lock (not a DispatchQueue or actor, to avoid context-switch cost on the audio thread).
XcodeGen — regenerates SpeakSavvy.xcodeproj from project.yml; required after adding any new Swift file.

How it works

Capture — AVAudioEngine taps the mic into a PCM buffer while the recording UI is live; the tap callback feeds two parallel consumers.
On-device audio DSP — AudioAnalyzer runs YIN pitch detection, RMS volume, and CPP (cepstral peak prominence) voice-quality analysis on the raw buffer via Accelerate/vDSP, thread-safe via os_unfair_lock. This branch is fixed to the .deterministic ScoringRouter tier and never reaches a model.
On-device transcription — SpeechService runs Apple’s SpeechAnalyzer (iOS 26, on-device) with per-word timestamps and confidence, falling back to SFSpeechRecognizer if SpeechAnalyzer’s on-device assets aren’t installed.
Metric fusion — SpeechMetricsEngine combines the ASR transcript and the DSP output into SpeechMetrics: words-per-minute, fillers-per-minute (matched against a hand-rolled lexicon — um, uh, basically, actually, multi-word patterns — not the Natural Language framework), pause count, pitch standard deviation in semitones, volume standard deviation, CPP mean, and a pronunciation score derived from ASR confidence.
Tier decision — AIRouter resolves offline vs enhanced from network reachability, TierOverrideManager (admin/user override), and live Apple Intelligence availability; .enhanced degrades to .offline mid-call if the cloud request fails in a degradable way (timeout, 5xx, 429-exhausted, proxy 401) and the on-device model is usable.
Offline content coaching — when .offline, CoachService builds a drill-specific system prompt and runs Apple’s on-device Foundation Models (~3B) via LanguageModelSession with guided generation, producing the rubric and coaching prose with zero network calls.
Cloud content coaching — when .enhanced, CoachService posts to the Cloudflare Worker proxy at speaksavvy.ai/api/llm/*, authenticated with a bearer token (not an LLM key); TaskRouter picks the model per call type — Groq Llama 3.3 70B for unstructured conversation turns, Groq GPT-OSS 20B/120B for strict-JSON drill scoring, OpenAI gpt-4o-mini/gpt-4o as the fallback provider and for quality escalation when the small rung’s structure confidence is low.
Structure validation — for drills with a structural rubric (PREP, STAR), the cloud model proposes transcript segments; CloudStructureExtractor grounds each one by locating it verbatim in the actual transcript before accepting it into the StructureMap — a segment the model claims but can’t be found is dropped, so no ungrounded model claim reaches the score.
Composite scoring — CoachFeedbackParser turns the model (or on-device) response into a ScoreRubric: seven dimensions (Structure, Relevance, Fluency, Pace, Delivery, Engagement, Conciseness) scored 1–5, plus a Pronunciation figure derived from ASR confidence, independent of the LLM call.
Optional Live Voice — a consent-gated, opt-in drill mode instead uses OpenAI Realtime (gpt-realtime-2.1-mini) for turn-free speech-to-speech over a WebSocket relayed through the same Worker; cloud-only, so it’s unavailable offline and falls back to the turn-based loop.
Coach playback — text feedback is spoken back via OpenAI TTS (gpt-4o-mini-tts, cloud) on the enhanced tier or AVSpeechSynthesizer (on-device) on the offline tier.
Persistence — SessionPersister writes the session, transcript, SpeechMetrics, and ScoreRubric (JSON-encoded) to SwiftData. Nothing leaves the device at this step — the cloud call, if any, already happened upstream in step 7.

Tech stack

Swift, SwiftUI, SwiftData for local persistence. Apple’s Speech framework (SpeechAnalyzer on iOS 26, SFSpeechRecognizer fallback) for transcription; AVFoundation for capture; Accelerate/vDSP for on-device YIN pitch, RMS volume, and CPP voice-quality extraction. Apple Foundation Models (~3B, on-device) for offline coaching. Groq (default cloud tier: Llama 3.3 70B, GPT-OSS 20B/120B) and OpenAI (fallback: gpt-4o-mini/gpt-4o; also Realtime for opt-in Live Voice and gpt-4o-mini-tts for coach playback) for online coaching, reached through a Cloudflare Workers proxy that holds provider keys server-side, backed by Cloudflare KV (response cache) and a Durable Object (rate limiting). Sign in with Apple for local identity. No app backend database — SwiftData is the only datastore for user/session data.

Results

Worked example from a captured session (Interview Prep drill, STAR prompt): the response transcribed to 22 words over 44 seconds, scored 165 WPM (off-target, amber) and 0.0 fillers/min (under threshold). The five-dimension breakdown came out Structure 1, Relevance 5, Fluency 5, Pace 2, Conciseness 2 — content was fine but the answer both ran short and skipped a required STAR section, which the structural parser caught and the fluency/relevance scores did not. A separate PREP-drill capture on the same day produced a vocal-delivery read of Pitch Variety 2.9 ST (Flat), Volume Range 0.06 (Moderate), Voice Quality 211.1 (Clear), Speech Activity 78% (Steady).

⚠️ no benchmark yet — no measured word-error rate for the Speech framework transcription on this content, no comparison of filler/pace detection against human-coach labeling, and no aggregate accuracy numbers across sessions. The figures above are a single worked example, not a validated accuracy claim.

Lessons

Delivery metrics (filler rate, pace, pitch, volume, voice quality) were routed to a fixed deterministic tier — Speech framework transcription plus native YIN/RMS/CPP — that never touches a model, because those numbers need to reproduce exactly on replay and a model adds cost/latency without an accuracy benefit for a signal-processing calculation. Content-judgment feedback was not built on-device-only: it’s tiered through ScoringRouter, with Apple’s on-device Foundation Models (~3B) as the offline/degraded path and Groq/OpenAI as the default online path — the trade-off is that transcript and derived metrics (never raw audio) reach the cloud tier when it’s active, in exchange for coaching quality the on-device 3B model doesn’t yet match.
SwiftData was chosen over Core Data because sessions, drills, and per-metric history are simple structured records; the macro-based model definitions keep the persistence layer thin without giving up migrations.
Local-first with no accounts was chosen over sync because the app needed to work offline (a rehearsal use case, e.g. on a plane) — the cost is no cross-device history, the benefit is zero backend and zero data-disclosure surface.

Open challenges

Actively in progress, stated plainly rather than hidden:

On-device vs cloud coaching. Content-judgment coaching runs tiered — Apple’s on-device Foundation Models (~3B) offline, Groq (default) or OpenAI (fallback) online — but which tier gives the best coaching at acceptable latency is not settled. Active work is a head-to-head on the same drills: on-device 3B vs Groq (Llama 3.3 70B) vs OpenAI, scored for coaching usefulness and turnaround, to decide where each tier should actually be the default rather than assuming cloud always wins.
Turn-based interaction. The Live Voice path (OpenAI Realtime, opt-in) is the least-settled surface: real-time conversational coaching trades the deterministic-scoring guarantees the rest of the app relies on for a latency-sensitive back-and-forth. Testing how much of the turn-based loop can stay on-device and where a cloud realtime model earns its round-trip is open.