Live voice fluidity, and a crew you can feel getting better

0. What the clip actually is

It is not a voice demo. Nobody in it talks to an assistant.

Source: @brycent_, 2:08, "VCs just invested $250m into an ai assistant that's in beta. Is this the next big thing or Larp?"

From the transcript and keyframes: a talking-head explainer about Instinct, a personal-AI startup that raised $250M at $2.5B four months in, still in closed beta. Its founder, Noah Shinn, co-authored the Reflexion paper (on screen: the actual PDF, "Reflexion: Language Agents with Verbal Reinforcement Learning"). The clip's central slide is a two-line animation:

LLM: try → fail → (try again from scratch)

REFLEXION: try → fail → write down what went wrong → try again

The narrator's thesis, verbatim: agents "can learn from their mistakes without retraining or changing the model weights … by writing down what went wrong, remembering the lesson, and then using it on the next attempt." He connects this to recursive self-improvement and argues that is what the $250M is actually buying.

The one product anecdote in the whole clip is the bar you are really pointing at:

"One user even told Instinct to go find a booking for the 70-millimetre Odyssey film and it did so without coming back to him with any problems."

Other frames: Shinn's X profile, an Instinct 1Password partnership post, an "invite link isn't valid / join the waitlist" page, stock robot-learning-to-walk footage.

Part A  ·  Live-voice fluidity in Crew

A0. The precondition, now fixed: the spoken turn could never close Fixed, unproven on device

B65's number-one defect was that voice turns were never dispatched – mic works, words appear, waveform animates, crew never replies. The identical thing was reported on B50, B52, B53 and B65.

fix/crew-b66-voice-endpoint-stall (commit aae952e, crew-ios-native) fixes it, and the root cause is worth knowing because it shapes everything below.

One predicate was answering two unrelated questions. rmsClearsLocalSpeechGate (0.0015) is deliberately set near the noise floor to answer "may these bytes leave the device" – ordinary room tone clears it, since real speech measures 0.03–0.15. That same predicate was also read as "is the user still talking", and on that reading it cancelled the 2-second endpoint countdown. Capture frames arrive every ~85ms, so the countdown was re-cancelled roughly 24 times before it could ever fire. Build 53 shortening it from 12s to 2s could not help: the timer was not too long, it was being cancelled.

The fix makes server VAD the only speech authority for the endpoint; a local frame may pre-empt it only while the server has an utterance open. Barge-in and mid-utterance rescue keep their latency advantage; the fatal case goes away.

A1. Keep the wire on the realtime model. Astra cannot do voice. Measured

Proven live against the API, not read from docs:

wss://…/v1/realtime?model=gpt-6-astra      → invalid_model: "not supported in realtime mode"
wss://…/v1/realtime?model=gpt-realtime-2.1 → session.created, output_modalities=['audio']

gpt-6-astra is text + image in, text out. No audio, either direction. Crew is already on gpt-realtime-2.1, the current recommended realtime model, so there is no upgrade to capture by chasing Astra onto the voice wire. That path is closed.

Two traps to respect:

• POST /v1/realtime/client_secrets mints a token for any string, including a deliberately fake model id, and echoes back output_modalities: ["audio"]. A successful mint is a false green. Validation only happens on the actual connect.
• The Realtime beta API shape is dead. Any code still sending OpenAI-Beta: realtime=v1 fails with beta_api_shape_disabled on every model, including working ones.

A2. The real finding: on iOS the voice is a verbatim reader, and the web client is already ahead of it

This is the load-bearing fact, and it is not what was expected before reading the code.

iOS registers zero tools. RealtimeVoiceSession.mintBody(for:) sends only instructions + turn_detection. No tools on the session at all. Separately, the TTS render call (responseCreateBody) is clamped with "tools": [], "tool_choice": "none" and "Read the authorized Crew text verbatim. Do not add, omit, translate, paraphrase, explain, or answer it."

The gap is the other one. iOS voice is transcription-only: it transcribes, dispatches text to /api/voice-dispatch/v1, and speaks back a signed, verbatim-matched authorization. The realtime model never holds a turn, never decides, never thinks.

The web client already does what iOS does not. crew-mvp/index.html:15863 registers five functions on the session, including do_task, which hands the request straight to the crew brain. So the delegation pattern you are asking about already exists in Crew, on web.

And the server side is already built for it. crew-api/api/voice-rt-token.js:124,165 accepts up to 24 client-supplied tools and sets session.tools + tool_choice: 'auto' at mint time. iOS simply never sends any.

So the honest framing:

CREW ON iOS   speech → transcript → /api/voice-dispatch → agent.js brain → verbatim TTS
              chained pipeline. the user waits for the whole brain before hearing anything.

CREW ON WEB   speech → realtime model holds the turn, has do_task → hands to the brain
              already the right shape.

GPT-LIVE      speech → realtime model holds the turn
                         ├─ easy → answers now
                         └─ hard → async-delegates to frontier model
              the user never waits in silence.

iOS is one mint-body change away from the web client's shape, not an architecture away.

A3. Barge-in is already built. Do not rebuild it. Exists

Worth stating plainly because it is the usual first recommendation and it is already done, correctly, including the part everyone skips.

RealtimeVoiceSession.swift:2633-2690 implements interruptOutputForBargeIn(), which sends response.cancel and conversation.item.truncate with a real audio_end_ms offset. That third step is the one that matters: without it the model believes it said a sentence the user never heard and every later turn is subtly wrong. Crew does it right, and the B66 fix was carefully written to preserve it.

Barge-in is not the gap. The missing session tools on iOS are.

A4. The change that actually buys the clip's feel: send iOS one tool at mint time Proposal

Promote the iOS realtime model from reader to turn-holder, minimally, without giving up the verbatim guarantee:

1. Add one tool to mintBody(for:): hand_to_crew(job). Not twenty-one. One. The realtime model's only decision is "is this a job for the crew?" – it never does the work. The server already accepts and installs it; web already proves the shape with do_task.
2. It fires mid-utterance, the moment intent is clear, and returns a job id in ~0ms. The crew starts working while the user is still finishing the sentence.
3. Keep the two response modes separate. Conversational turns (acknowledging, filler, "which Friday?") may use the session's tool. Result delivery keeps today's render clamp exactly as it is – tools: [], tool_choice: "none", verbatim instructions, signed authorization with SHA256 text match. The crew's actual output stays word-for-word authored by the crew. The law survives; only the conversational scaffolding around it is new.
4. A mid-flight addition amends the running job, never spawns a second one. One utterance, one job. Two jobs for one sentence is the tell that it is not really listening.
5. Set it at session creation, never via session.update (see the trap in A2).

A5. Spoken filler that is true Net-new

A delegated frontier turn is a reasoning round trip inside a live conversation. Something has to fill it or the call goes silent.

Today there is no filler path at all. No hold-audio, no "one moment", no thinking sound. The only slow-turn behaviour in RealtimeVoiceSession.swift is a timeout and a stall-drop, and create_response is false in production so the model never spontaneously fills. A long turn is simply silence. This is net-new work and it is the difference between "it's thinking" and "it's broken".

The filler must be generated from the real dispatch, not a canned stall phrase. Crew's law is that home reports work and never manufactures a prompt; the same applies to the voice.

• Good: "Ben's pulling the Friday times now." – names the creature, names the real action, sourced from the actual job row.
• Banned: "Hmm, let me think about that…", "Great question!", any generic AI throat-clearing.

Gate delegation to turns that genuinely need it. Firing it every turn makes the crew sound slower, not smarter. Log which turns delegated and why. That log is an input to Part B.

A6. The brain can already be Astra Built

crew-api-wt-astra commit fb4fbe3 makes CREW_OPENAI_MODEL=gpt-6-astra a working switch via a Responses adapter (api/_openai_responses.js). This was necessary, not cosmetic: measured against the prod key, gpt-6-astra cannot use function tools on /v1/chat/completions at all – every parameter combination 400s, and OpenAI's own error says to use /v1/responses. Crew's brain is a 21-tool loop, so chat/completions could only serve Astra with the tools stripped.

So the frontier half of the delegation is built. What is missing is the front half: the voice model that can hold a turn while it runs.

A7. Voice speaks one line; the village delivers the result

When the crew returns, the creature delivers the artifact in the village as text, and the voice says exactly one line that it landed ("Booked. It's on the shelf."). No generic AI voice reading a result aloud. This is unchanged from today and should stay unchanged.

A8. The acceptance test, in your own terms

Not "does voice work". The bar the clip sets:

Ask for something real, out loud, in one natural sentence. Interrupt it once mid-answer. Get the thing done with zero clarifying questions.

If the crew came back and asked something, that turn failed the bar – and, per Part B, that failure is a row that gets written down.

Part B  ·  A crew you can feel getting better

The reference is an internal effectiveness harness. The Crew-native version is not a harness and not a dashboard. It is the thing the clip's slide describes, built into a village:

try → fail → write down what went wrong → try again, better.

Crew has an enormous advantage here that a chat product does not: there is already a village and there are already creatures. The improvement does not need a reporting surface invented for it. The creatures are the readout.

B0. Start here: Crew already built this and never plugged it in Built, idle

Before designing anything, the single most important fact.

crew-ios-native/Sources/Crew/Models/CrewSpecialistLifecycle.swift is a 593-line per-creature promotion/demotion evidence reducer that already exists. It has:

• CrewSpecialistPermanencePolicy.standard: a creature becomes permanent on 2 distinct work items, 2 distinct artifacts, 2 related reusable evidence rows. Not vibes, thresholds.
• CrewSpecialistValueEvidence { evidenceID, relationshipKey, workID, artifactID, valueSummary, reusable, observedAt, retraction } – including retraction, so evidence can be withdrawn.
• Residency states temporary | permanent | merged, decisions promoted / demoted / merged, event-sourced with optimistic concurrency.
• Companions: CrewMindContinuity.swift (649 lines), CrewInterviewArchitectureStore.swift, CrewSpecialistPermanenceReceipt.swift.

This is exactly the pattern the B65 audit named as the shape behind 26 of 89 defects:

A state machine is designed correctly, unit-tested as a pure function, and then never wired to its real inputs.

So Part B is not "design a new system". It is: wire the one that already exists, give it honest inputs, and show it in the village. The hard modelling work is done and it is good. That reframes this from a quarter of work to a wiring job plus a design job.

Note also CrewSpecialistLifecycle already gets the two hardest honesty rules right: evidence must be distinct (two artifacts, not one artifact counted twice) and retractable (a creature can be demoted). Do not redesign around it.

B1. The mechanism: every job leaves a receipt, every miss leaves a lesson Proposal

Two records, and only two.

The receipt

Written for every job, automatically, from real events. Crew is closer to this than it looks: crew_event rows of kind agent_run already carry { ask, title, result, tool_calls, usage } plus user_id and created_at (crew-api/api/agent.js:2792). What is missing is an outcome and an owner.

Do not add a new table for this. Crew's stated policy is to reuse crew_event rather than create tables (new tables carry Supabase auto-grant risk). A new kind plus a richer body is the cheap, in-grain move.

The lesson

Written only when a job is corrected or abandoned. A short plain-language note, keyed to (creature, job kind, user), injected into the next same-kind job's context. This is Reflexion, and it is the whole product claim: the crew improves without anyone retraining anything.

The lesson is not a log line. It is a sentence the user could read and agree with:

"Fares wants times, not options. Book the first one that fits and tell him."

Nearest existing thing: crew_skills (migrations/001_autonomy_intent.sql:53) already stores name / trigger / steps / embedding / use_count / last_used_at, retrieved by a match_crew_skills() RPC. It counts usage and never success, which is precisely the gap. A lesson store is crew_skills plus an outcome, and it can reuse the same embedding-retrieval path.

B1b. The template already in the repo

Crew has built exactly one honest closed-loop measurement system, and its header states the design principle better than I could (migrations/003_crew_growth_metrics.sql, Edward, 2026-07-01):

"the 002 model tracks whether a post POSTED but NOT whether it WORKED. Without performance capture you cannot tell a good hook from a bad one, so every next hook is a blind guess."

Its shape is the shape to copy: a variant tag on the work (hook_pattern), outcome capture as one row per (item, capture time) so you get a trend rather than a last-value, and a rollup view grouped by the variant (crew_growth_hook_rollup). Swap post → job and hook_pattern → creature + job_kind and the design is done.

Crew's agent_run today is in the 002 state: it records that the crew acted, never whether it worked.

B2. What makes it honest (this is the part that usually rots)

Five rules. Each one exists because its absence produces a vanity metric.

1. No creature scores itself. Every counter increments from a user action or a wall clock. Never from the model's judgement of its own work. A model grading its own homework is the highest-severity failure mode Crew has already been bitten by.
2. Silence is not success. A result the user never opened is unseen, not accepted. The default state of every job is unresolved, not good.
3. Competence can go down. If the number can only rise, it is a progress bar, not a measurement. A bad week must be visible as a bad week.
4. Show the denominator, and hide small numbers. "8 of 10 jobs landed", never "80%". Below a floor (say 5 jobs of a kind) the creature shows "still learning" and no number at all. A percentage computed from three attempts is a lie with a decimal point.
5. Report the miss with the win. Any weekly moment that names a win must name a miss and the lesson written from it. A surface that can only deliver good news gets ignored within three weeks.

Sixth, structural: track per job kind, never globally. A creature is good at booking things, not "good". A global score lets a strong kind hide a weak one, and it is exactly the number that turns out to be about the wrong thing.

B3. How the user feels it (village language, not a dashboard)

Ranked by how much the user actually feels it.

1. Fewer questions is the headline

It is the thing the user physically experiences, it is the clip's own bar, and it needs no explanation:

"Last month I asked you four things before booking this. This time, none."

2. The creature earns its place, and you see it

This is already modelled: a creature starts temporary and becomes permanent on real evidence (2 distinct work items, 2 distinct artifacts). That promotion is the showcase. A temporary creature is visibly a visitor in the village; a permanent one has a place and the tool for its kind. Competence shows in the sprite and its idle behaviour, never in a badge with a number on it. The user reads it the way you read that a colleague has got good at something: without being told.

And because the model supports demoted and evidence retraction, a creature that stops earning its keep can visibly go back to being a visitor. That is what makes the promotion mean anything.

3. The notebook

Tap a creature, see its little book. Real lessons, in plain sentences, dated, each one traceable to a real correction the user made. This is deliberately the anti-vanity surface: it is a list of things the creature got wrong and what it now does instead. It is also the most convincing artifact in the product, because the user recognises their own corrections in it.

4. "Your crew this week"

A short Sunday moment in the village, ~15 seconds, a scene rather than a report: the creatures gathered, one line each. What got done. What got faster. One lesson learned. One thing missed. Warm, light, brief, skippable.

5. Never a leaderboard

They are a crew, not competitors. No ranking creatures against each other, ever. The comparison is always this creature now vs this creature a month ago.

B4. Why this is the same loop as Part A

They close on each other, which is why they should ship as one thing:

voice turn → job receipt → lesson written on a miss
    ↑                              ↓
    └──── fewer clarifying questions next time ─────┘

Every voice turn produces a receipt. Receipts produce lessons. Lessons enter the next turn's context. The measurable result is questions_asked trending to zero – which is precisely what the user hears as the crew getting better, and precisely what the clip's one product anecdote describes.

That is the honest version of "recursive self-improvement": not a claim, a counter.

What to do next

1. Prove fix/crew-b66-voice-endpoint-stall on a real phone in a normal room. It is committed (already merged into feat/crew-b66) but its Swift unit test is unrun and the Simulator structurally cannot reproduce the defect. Nothing else here is real until a spoken turn reaches a creature on device.
2. Do not rebuild barge-in. It is already correct, truncate included (A3).
3. Send one tool, hand_to_crew, in the iOS mint body (A4) – at session creation, never session.update. Copy the web client's do_task shape. Keep the verbatim render clamp untouched. This is the single change that converts iOS from a chained pipeline into the clip's architecture.
4. Add a true filler path (A5). None exists; today a slow turn is silence.
5. Add creature_id + job_kind + outcome to the run record (B1). Nothing is attributable to an agent today, which is the one gap that blocks all of Part B. Reuse crew_event, do not add a table.
6. Wire the permanence-receipt endpoint so CrewSpecialistLifecycle finally receives evidence (B0). The reducer is built, tested and idle. This is wiring, not design.
7. Then design the notebook, the promotion moment and "your crew this week" through Codex, mockup-first, per standing direction.

Grounding