The Problem with Raw Voice Input
Voice AI assistants, from smart speakers to in-car systems, often struggle with maintaining a coherent conversation. A common pitfall lies in how they handle the raw, uncurated stream of audio and text data. Developers often build demo applications that simply append every piece of data to a single transcript. This includes partial speech recognition results, the final user utterance, the AI's response, and even text sent for speech synthesis. This complete, unedited log then becomes the prompt for the next AI turn.
While it seems intuitive to keep all data for maximum context, this approach introduces subtle but significant problems. Partial recognition results might be incorrect or incomplete. An AI's response might be interrupted before it's finished, or a late-arriving callback might pertain to an abandoned turn. When this raw, sometimes inaccurate, data is fed back into the model, the AI has no way of knowing what was truly said or intended. It can lead to the AI hallucinating conversation history, making nonsensical responses, or losing track of the user's actual intent.
The AI model cannot reliably repair this damage because it only sees the history your application presents. It lacks the critical ability to distinguish between provisional, incomplete data and committed, finalized information. This is akin to trying to reconstruct a conversation based on overheard snippets and discarded notes – the full picture is lost.
Committing the Conversation: A Better Approach
The practical solution is to treat conversation history as committed application state, not as a raw log of every generated string. This means actively filtering and curating the data before it becomes part of the AI's permanent memory. Instead of blindly appending everything, developers must implement rules to determine what constitutes a finalized part of the conversation.
This approach requires building a boundary layer that intelligently manages the conversational context. This boundary acts as a gatekeeper, ensuring that only validated and complete pieces of information are added to the AI's understanding of the ongoing dialogue. The goal is to create an AI companion that sounds convincing because it's working with accurate, finalized conversational turns, not a jumble of provisional data.
Implementing a Filtered Context
To achieve this, a small TypeScript boundary can be built that applies several key rules. These rules ensure that only confirmed user utterances and complete AI responses contribute to the persistent context. Let's break down the essential filtering principles:
1. Provisional User Speech
Partial user speech recognition results should be treated as provisional. This means they are temporary and subject to change or cancellation. If the user continues speaking and a final, complete utterance is recognized, the provisional result should be discarded. If the user stops speaking mid-utterance and no final recognition is available, that partial segment should not be committed to the conversation history. This prevents the AI from acting on potentially incorrect or incomplete user input.
2. Finalized User Utterances
Only when a full user utterance is recognized and confirmed should it be appended to the conversation history. This 'committed' utterance becomes a stable piece of context that the AI can reliably act upon. This ensures that the AI's understanding is based on what the user actually completed saying.
3. Complete AI Responses
Similarly, the AI's responses must also be finalized. If the AI's response generation is interrupted, or if the text-to-speech (TTS) process is cut short, that incomplete response should not be added to the conversation history. The AI should only 'commit' a response once it is fully generated and ready for playback. This prevents the AI from referencing parts of its own response that were never actually delivered to the user.
4. Handling Abandoned Turns
Late callbacks or responses that arrive after the user has already moved on or abandoned a turn should be ignored. For instance, if a user asks a question, then asks a follow-up before the AI answers the first, the AI's response to the first question should be discarded if it arrives too late. The system needs a mechanism to track the current conversational turn and invalidate responses that are no longer relevant.
The Impact on AI Companions
By implementing these filtering rules, developers can significantly improve the coherence and reliability of AI voice companions. Instead of an AI that sounds confused or makes bizarre leaps in logic, users will interact with a companion that remembers what was actually said and responded to. This creates a more natural, intuitive, and trustworthy user experience.
This approach shifts the responsibility of context management from the LLM to the application layer. The LLM remains a powerful engine for understanding and generating language, but it's provided with clean, validated data. This is crucial because LLMs, by their nature, cannot inherently distinguish between a user's final intention and a transient, incomplete recognition artifact. They operate on the data they are given.
The challenge for developers is to build robust state management into their voice applications. This involves careful handling of asynchronous operations, speech recognition confidence scores, and timing mechanisms to ensure that only committed turns are added to the conversational context. The reward is an AI companion that feels more intelligent and responsive, capable of maintaining a more convincing and continuous dialogue.
The surprising detail here is not the complexity of the LLM itself, but the subtle yet critical importance of how its input context is managed. A few lines of carefully written code at the application boundary can prevent the AI from 'hallucinating' conversation history, a common frustration for users of current voice assistants.
What's Next?
The next step for developers is to integrate these filtering mechanisms into their voice AI architectures. This might involve creating dedicated modules or services that handle conversation state, ensuring that provisional data is clearly marked and only finalized data is committed. For users, this means looking forward to AI companions that feel more present and less prone to nonsensical conversational errors, offering a genuinely improved interactive experience.