Bilingual voice meets visual context: the next Thor checkpoint

MS-08 / Conversation

Stability means keeping the right context.

After CP09-I, the next work addressed malformed reasoning markers, speculative transcription that no longer matched the spoken input, and history rollover that could remove the context needed for a follow-up. The repaired working set retains complete exchanges. A bounded two-exchange, 1,800-character carryover improved prewarming without treating the model cache as permanent memory.

A 50-turn Qwen-only probe kept the five block medians for time to first token around 228–231 ms. Its semantic checks passed 24 of 25 automatically; manual review found that the remaining answer was correct and the keyword scorer was too narrow. This supports the tested context policy over that run. It does not demonstrate lifelong memory.

Long speech now stays one Core turn across bounded transcription chunks. Complete 6.6- and 9.8-second questions were exercised without being misclassified as extended input solely because of their duration. Short follow-ups such as “Why?” and their Spanish equivalents also received explicit handling.

Several alternative approaches were rejected. Speculative decoding increased tokens per second but did not meet the net first-response improvement gate, so it did not replace the physical baseline. Story-specific long-input handling failed activation tests and gave way to a general model of the user’s conversational floor.

CP09-S made generation deterministic and kept interrupted assistant prefixes out of the next model prompt while preserving the canonical conversation record. CP09-W then added explicit spoken switching between English and Spain Spanish in the same session, including after a confirmed interruption. Core recognises a finite command grammar and gives a local acknowledgement without asking Qwen to decide the language.

Timing / Keep the measurements separate

No new blanket latency claim.

The ordinary-response reference below comes from CP09-S. The later CP09-W sample mixes Qwen answers with four fast local language acknowledgements, so its median is not a fair headline for general conversation.

The standing ≤1.3-second P95 voice target has not passed. The previous 1.3705-second median in J-012 remains a dated eight-response result, not the current headline or a promise that every reply is faster.

MS-09 / Capture and coexistence

Seeing continuously without starving the conversation.

The C920 could fall to roughly 15 FPS when the dynamic-framerate control reset. Reapplying and reading back that control restored about 30.92 FPS before and after a USB reconnect. A separate kernel-clock test measured 29.89 FPS and a latest-frame age of 24.216 ms, then demonstrated the transition from current to stale. That capture-clock result does not yet prove the freshness of every semantic image sent to Qwen.

The perception path now connects camera capture, RT-DETR detection, NvDCF tracking and typed scene metadata. Core controls how much work is admitted. Fixed detector schedules at 30, 15 and 10 FPS hurt voice performance; the adaptive candidate changes its interval from 2 to 32 during generation and back to 2 afterwards, while camera capture and tracking stay near 30 FPS.

Moving NvDCF tracking from CUDA to PVA reduced contention in the isolated comparison. Median Qwen request-to-onset was 712.2 ms with PVA, compared with 844.9 ms using CUDA and 696.9 ms without perception. The PVA diagnostic also processed 3,590 frames at 29.91 FPS with no invalid metadata. These are coexistence diagnostics, not end-of-speech-to-audible-answer results or a passed full-duplex gate.

Current-image grounding

The resident Qwen model can receive the camera image.

The current-image bridge captures a JPEG from the same camera path and hands it through a bounded in-memory media broker. An attempted separate Qwen3-VL 8B FP8 service failed on a DeepGEMM compatibility issue with Thor before an image request ran. That was a runtime failure, not a model-quality comparison.

The next test used the resident Qwen3.8 model’s native multimodal interface. At 256×256, it incorrectly treated a person in a background collage as another physical person in the room. A full 640×480 image corrected that scene interpretation in the retained physical test. A 720p capture experiment followed. The follow-up also strengthened grounding instructions, so this result does not isolate resolution alone. Broad visual accuracy and voice/vision response timing remain unqualified.

Recent video / Results and next candidate

Receiving video is not the same as understanding motion.

Two-second/four-frame and four-second/eight-frame clips reached the model and produced text, but both failed the intended motion test. First text arrived at about 751 ms and 1,245 ms respectively. Later endpoint-image and resolution diagnostics also failed the motion rubric.

The fixture itself had problems: its first selected frame was about 540 ms after the motion cue, and camera-versus-operator left/right wording was ambiguous. The evidence therefore cannot establish that the model is incapable of temporal reasoning. A calibrated motion fixture was implemented to separate a bad test from a model limitation.

The window is temporary visual processing, not permanent memory. Raw images and clips are handled in RAM without named media files in this path. Core remains responsible for admission, freshness and cleanup. The candidate attaches context during an active session; it does not authorise indefinite retention.

Where the project stands

Retain the working baselines. Qualify the combined system.

• Keep CP09-W as the frozen explicit bilingual voice baseline and CP05 as the retained voice/perception rollback point.
• Run the 15-second candidate on Thor with calibrated actions, grounded answers and measured end-to-end timing.
• Close the remaining overlap, cancellation, recovery and P95 latency gates. A shutdown/provenance cleanup failure during resumed WAIT/CONTINUE playback remains recorded.
• Continue governed memory, relationships and internet work through their own acceptance gates before Alpha or powered embodiment.

MS-08 and MS-09 both remain in progress and blocked at their formal acceptance gates. Core, cognition, visual processing and speech run locally; speech recognition also uses the resident Cohere Transcribe service locally on Thor.