LAB RECORD / J-007 / LOCAL VOICE
From a prepared robotics computer to a working conversational prototype.
The current head is built around an NVIDIA Jetson Orin Nano Super with a Logitech C920 camera, a Seeed Studio ReSpeaker XVF3800 four-microphone array, and Creative Pebble V3 speakers. Those devices had already been brought online individually. This stage made the audio hardware and local AI stack participate in one continuous conversational system.
The result is the first working local Evopien voice loop: microphone input, speech recognition, local Qwen reasoning, streamed speech synthesis, hands-free turn-taking, and real conversational interruption now operate together on the Jetson.
Local conversation loop
The complete path now runs locally on the Jetson.
Speech recognition is handled by whisper.cpp, Qwen3.5-4B runs locally through llama.cpp, and Piper generates spoken output. Qwen is the current language and reasoning component; it is not the Evopien identity.
ReSpeaker microphone
|
v
local speech recognition
|
v
Evopien Core
|
v
Qwen3.5-4B
|
v
streamed local speech synthesis
|
v
Pebble V3 speakers- Speech input
- ReSpeaker XVF3800 microphone array with local capture
- Speech recognition
- whisper.cpp multilingual base model, currently locked to English for Alpha stability
- Reasoning
- Qwen3.5-4B Q4_K_M through llama.cpp with GPU offload
- Speech synthesis
- Persistent local Piper service using the en_GB-alan-medium voice
- Playback
- Creative Pebble V3 through the PipeWire/PulseAudio output path
- Core boundary
- Identity and relationship/session state remain separate from the replaceable language model
Hands-free interaction
The conversation no longer depends on push-to-talk.
Evopien listens for speech, detects pauses, and estimates whether the person has actually finished the thought before submitting the turn to the model.
- Speech begins naturally
The user speaks without pressing a key or holding a push-to-talk control.
- Pauses are interpreted
A short silence can close a completed sentence while an incomplete thought can remain open.
- Thinking holds extend the turn
Phrases such as “give me a moment” or “let me think” can temporarily extend the listening window rather than triggering a premature answer.
- The resulting turn enters Core
Only after the turn is considered complete does the current transcript proceed to the local model path.
Barge-in / interruption
Evopien can be stopped while it is speaking.
The microphones continue listening during playback. Clear human speech can stop the current audio, cancel the active language-model generation, and preserve the interruption audio for transcription instead of discarding it.
Evopien: [speaking]
User: Stop.
Evopien: [silent]Current Piper/paplay speech stops when a valid barge-in is detected.
The active llama.cpp HTTP stream closes and the abandoned generation task is cancelled.
The interruption itself remains available for transcription and classification.
Stop, wait, stand by, and similar control intentions can end speech without causing another unnecessary answer.
A later “go on” can continue the interrupted subject using the temporary session context.
The next utterance can immediately replace the interrupted topic with a new request.
Streaming instead of waiting
Qwen, TTS, and playback now overlap.
The earliest version waited for the complete model answer before generating speech. The current runtime streams model output and sends sentence or clause-sized chunks to Piper while Qwen continues generating the rest of the answer.
Qwen generation
+
speech synthesis
+
audio playbackBest observed speech-end-to-first-audio measurements have reached roughly 1.7-1.9 seconds. The result varies with end-of-turn detection, transcription, and response content, so it is a development measurement rather than a fixed latency guarantee.
Working context, not long-term memory
The voice loop remembers the active conversation only temporarily.
The runtime currently keeps eight messages, approximately four exchanges, so it can follow the active subject and recover after interruption. That context disappears when the voice loop exits and is deliberately not presented as Evopien's governed persistent memory.
- Current context
- Eight temporary messages in the live voice process
- Purpose
- Follow the active topic, simple references, and interruption-to-resume behaviour
- Persistence
- None; the working context disappears when the voice loop exits
- Future memory
- Language-model proposals will require deterministic classification, permission, retention, and audit rules before storage
- Privacy target
- Unauthorised private context must be blocked before it is released to any model
- Claim boundary
- Temporary chat history is not being described as long-term relationship memory
8 GB Jetson optimization
The same 4B model now leaves room for the next perception layer.
The Jetson uses unified memory, so the operating system, GPU model, speech stack, and future vision processing all compete for the same 8 GB pool. The first complete voice configuration worked but left too little headroom for vision.
Qwen3.5-4B Q4_K_M remains the current local model.
Reduced from the earlier 2048-token voice configuration for the current Alpha workload.
Reduced from 128 / 128 to lower runtime memory pressure.
Improved from less than 1 GB available in the earlier full-voice configuration.
Physical memory headroom remained during the highest observed test in this configuration.
Observed typical local generation range during current development testing.
Current demonstrated state
What is actually working now.
This is the current development boundary rather than a claim about the finished humanoid.
The persistent Evopien identity remains separate from the current model.
The current Core resolves the creator user and relationship/session context.
The local llama.cpp provider sits behind the existing model abstraction.
Local text reasoning runs through llama.cpp with GPU offload.
whisper.cpp and Piper provide the current speech recognition and speech output path.
Adaptive pauses and explicit thinking holds have been demonstrated.
Clause-sized output overlaps generation, synthesis, and playback.
Playback and abandoned model generation can be cancelled by valid barge-in speech.
The live loop keeps a small working history for conversational continuity.
The Core repository retained nine passing automated tests at this milestone.
Known limit
The camera works. Live conversational sight does not yet.
The Logitech C920 already captures video correctly, and Qwen's multimodal capability has been tested independently. The current voice runtime does not yet capture a current camera frame and route it into the live spoken turn.
Evopien therefore cannot yet truthfully answer “what do you see?” from a current frame during normal conversation. Governed persistent relationship memory, complete multi-user privacy separation, a full memory audit trail, continuous scene understanding, face recognition, ESP32 facial movement, servo/body control, autonomous physical action, and a complete humanoid body are also future milestones rather than current claims.