Voice Pipelines for Agents: On-Device STT and TTS

Voice interaction adds a dimension that text alone can't match. When you're driving, cooking, or walking the dog, typing a message to your agent isn't practical. But most voice implementations either depend on cloud APIs (latency, privacy, cost) or require GPU hardware that a small self-hosted box doesn't have. The sweet spot is on-device inference with models small enough to run on CPU alongside everything else.

This article covers the voice pipeline architecture in humux: how a Telegram voice note becomes a transcription, passes through the agent, and comes back as a spoken reply — all without leaving the machine.

The Pipeline End to End

A voice round-trip has six stages. Each stage is a function call, and the whole thing runs sequentially in a single async task:

1. Download — Telegram sends voice notes as OGG/Opus files. The bot API gives us a file ID; we download it to a temp path.
2. Transcode to WAV — faster-whisper expects 16kHz mono WAV. A single ffmpeg call handles this.
3. Transcribe (STT) — faster-whisper runs CTranslate2-optimized Whisper inference on CPU.
4. Agent turn — The transcribed text goes into the normal message pipeline, as if the user had typed it.
5. Synthesize (TTS) — If the agent's response contains a [[voice]] marker, the text is routed through Kokoro or edge-tts.
6. Send — The resulting WAV is transcoded to OGG/Opus and sent back as a Telegram voice note.

The core transcription function is straightforward:

from faster_whisper import WhisperModel

model = WhisperModel("base", device="cpu", compute_type="int8")

async def transcribe(audio_path: Path) -> str:
    wav_path = audio_path.with_suffix(".wav")
    await run_command(
        "ffmpeg", "-i", str(audio_path),
        "-ar", "16000", "-ac", "1", "-f", "wav",
        str(wav_path)
    )
    segments, info = model.transcribe(
        str(wav_path), beam_size=3, language=None
    )
    text = " ".join(s.text.strip() for s in segments)
    wav_path.unlink(missing_ok=True)
    return text

Setting language=None lets Whisper auto-detect. The info object returns the detected language and probability, which can be logged or used downstream. The int8 compute type halves memory usage on CPU with negligible quality loss.

Choosing a Whisper Model

faster-whisper ships CTranslate2-converted models in several sizes. The tradeoff is simple: bigger models transcribe more accurately but use more RAM and CPU time. Here are real numbers from a 4-core x86 CPU (Intel N100, the kind of thing you'd run a home server on):

base is the default in humux. It handles accented English, Italian, German, and Spanish well enough for conversational messages. tiny works if you're tight on RAM or only deal with clean, single-speaker English. small is worth it for multilingual heavy use or noisy environments, but 500 MB of resident memory is significant when you're also running an embedding model and TTS in the same container.

TTS: Cloud vs. Offline

humux supports two TTS backends, selectable per agent:

edge-tts calls Microsoft's Edge speech service. Zero local resources, excellent voice quality across dozens of languages, but requires internet access. Since the audio is generated server-side, your agent's responses are sent to Microsoft. For personal use this may be fine; for privacy-sensitive deployments, it's a non-starter.

Kokoro 82M runs entirely on-device. At 82 million parameters, it's small enough to run on CPU in under a second per sentence. The quality is remarkably good for its size — natural prosody, no robotic artifacts — though it currently covers fewer languages than edge-tts. It loads ~200 MB into RAM and stays resident.

async def synthesize(text: str, voice_id: str, backend: str) -> Path:
    out = tmp_path / f"{uuid4().hex}.wav"
    if backend == "kokoro":
        audio = kokoro_model.generate(text, voice=voice_id)
        sf.write(str(out), audio, samplerate=24000)
    else:
        communicate = edge_tts.Communicate(text, voice_id)
        await communicate.save(str(out))
    return out

async def to_ogg(wav_path: Path) -> Path:
    ogg_path = wav_path.with_suffix(".ogg")
    await run_command(
        "ffmpeg", "-i", str(wav_path),
        "-c:a", "libopus", "-b:a", "48k",
        str(ogg_path)
    )
    return ogg_path

The Voice Marker Protocol

Not every response should be spoken. A three-paragraph answer with code snippets makes no sense as audio. The agent decides by including a [[voice]] marker in its response. The output post-processor strips the marker and routes through TTS:

VOICE_MARKER = "[[voice]]"

async def process_response(text: str, agent: Agent, chat_id: int):
    if VOICE_MARKER in text:
        spoken = text.replace(VOICE_MARKER, "").strip()
        wav = await synthesize(spoken, agent.voice_id, agent.tts_backend)
        ogg = await to_ogg(wav)
        await bot.send_voice(chat_id, ogg)
        wav.unlink(missing_ok=True)
        ogg.unlink(missing_ok=True)
    else:
        await bot.send_message(chat_id, text)

The agent's system prompt instructs it: "When the user sent a voice message and your reply is short and conversational, include [[voice]] at the start of your response." This keeps the decision with the model, where context about message length and content type lives. The rule is a heuristic, not a hard gate — the agent learns quickly which responses work as voice and which don't.

Per-Agent Voice Selection

Each agent in humux has a voice_id and tts_backend field in its configuration. A household might have one agent using a deep male Kokoro voice and another using a female edge-tts voice. This is configured in the admin UI or directly in the agent's YAML:

agents:
  jarvis:
    tts_backend: kokoro
    voice_id: af_heart
    whisper_model: base
  friday:
    tts_backend: edge-tts
    voice_id: en-US-AriaNeural
    whisper_model: base

The Whisper model is also per-agent, though in practice most deployments use the same model everywhere. Splitting it only makes sense if one agent handles primarily non-English input and you want to give it the larger small model for better multilingual accuracy.

Multilingual Handling

Whisper auto-detects language from the first 30 seconds of audio. In practice, for messages under a minute, this just works — it identifies the language, transcribes in that language, and returns the text. No language configuration needed.

TTS is where it gets interesting. Kokoro voices are language-specific: an English voice can't synthesize Italian. The pipeline detects the language of the outgoing text (a simple heuristic based on character ranges and common words, falling back to the detected STT language) and picks the matching voice variant. If no matching voice exists, it falls back to text output instead of generating garbled audio.

End-to-End Performance

Total round-trip for a typical voice interaction (10-second message, one-sentence reply) on a 4-core CPU:

Under five seconds feels conversational. The agent turn (the actual thinking time) dominates for longer responses. The voice infrastructure itself — STT plus TTS plus transcoding — adds about 2.5 seconds of overhead, which is roughly what a human takes to start replying in a real conversation.

The key decision in this architecture is keeping both models resident in memory. Loading Whisper on demand would add 3-4 seconds of cold-start time. At 150 MB (base) + 200 MB (Kokoro), the combined footprint is 350 MB — significant on a 2 GB VPS, comfortable on a 4 GB one. If memory is tight, edge-tts eliminates the Kokoro cost entirely, and you can drop to tiny whisper for another 75 MB savings.