How I Replaced My Cloud AI Assistant with a Self-Hosted Agent

I used to spread my AI usage across four or five different services. A chat app for general questions. A coding assistant in my editor. A separate tool for email summarization. Another for scheduling. Each one had its own subscription, its own context window, and none of them knew what the others were doing.

The turning point wasn't a privacy scare or a billing shock, though both played a part. It was the realization that I was spending more time switching between assistants than I was saving by using them. I'd ask one to summarize a thread, then manually paste the summary into another so it could draft a reply. I had five tools and zero integration.

The setup

I run humux on a small home server — an Intel NUC with 32GB RAM. The entire stack is one Docker container: the agent runtime, the admin UI, voice processing, and the database. No sidecars, no Redis, no Kubernetes. Total resource usage at idle is about 400MB RAM and negligible CPU.

services:
  humux:
    image: ghcr.io/mattmezza/humux:latest
    volumes:
      - ./data:/app/data
      - ./config.yml:/app/config.yml
    ports:
      - "8080:8080"
    restart: unless-stopped

That's the whole deployment. The config file defines my agents, their tools, and which LLM provider to use. I started with Anthropic's API for the reasoning model and added a local Ollama instance later for tasks where latency matters more than capability.

What I replaced

I consolidated five separate tools into three humux agents:

• forge — my coding agent. Reviews PRs, edits files, runs tests, commits changes. Replaced the editor-integrated coding assistant and the separate code review tool.
• atlas — my productivity agent. Handles email triage, calendar management, morning briefings, and contact lookups. Replaced the email summarizer and the scheduling assistant.
• sage — my research agent. Web searches, document analysis, general questions. Replaced the chat-based AI assistant.

The key difference: all three share the same memory database. When I tell sage about a project decision, forge remembers it during the next code review. When atlas reads an email about a deadline, sage can reference it when I ask about the project timeline. Context flows across agents because they share the same infrastructure.

What broke (at first)

The migration wasn't seamless. Three things caught me off guard:

Email credentials were the hardest part. Not technically — the vault handles encryption fine — but getting app passwords from Google, configuring IMAP with the right OAuth scopes, and testing CalDAV endpoints was a full afternoon of work. Cloud services hide this complexity; self-hosting exposes it. The Bitwarden import helped for passwords I already had stored.

Skill authoring took iteration. humux uses markdown skill files to teach agents how to use CLI tools. My first attempt at an email skill was too verbose — the model got confused by the wall of options. I rewrote it to show just the three most common patterns (send, search, list) and added edge cases only when the agent asked for help. The lesson: skills work best when they're concise.

Memory needed tuning. Out of the box, the agent remembered too much. Every offhand comment became a memory entry. I learned to configure the importance threshold and lean on the memory hygiene features (deduplication, reflection filtering) to keep the memory store useful rather than noisy.

What I gained

Three months in, the advantages are clear:

• One interface. Everything goes through Telegram or the CLI. I don't switch apps. I don't copy-paste between tools. I say "summarize my unread emails and check if any conflict with today's calendar" and one agent handles both.
• Real context continuity. The agents remember what I told them last week. I don't re-explain project context every session. The memory system isn't perfect, but it's better than starting every conversation from scratch.
• Cost control. I pay for the LLM API tokens I use, nothing else. No subscriptions. My typical usage runs about $30/month in API costs across all three agents. That's less than two of the five subscriptions I was paying before.
• No data export anxiety. My conversations, memories, and credentials are in a SQLite database on my server. If I want to switch LLM providers, I change one line in the config. If I want to stop, my data is already mine.

What I'd do differently

If I were starting over:

• Start with one agent, not three. I configured all three agents on day one. I should have started with just the coding agent, gotten it working well, then added the others. Each agent's skills and personality take time to dial in.
• Set up the vault first. I initially put API keys in the config file directly and migrated them to the vault later. Starting with the vault avoids secrets ever appearing in plain text on disk.
• Write skills from real usage. I pre-wrote skills for every CLI tool I might need. Most of them went unused. The better approach: use the agent without skills, notice when it fails or asks how to do something, then write a skill for that specific gap.

The migration took a weekend for the basics and about two weeks of incremental tuning to get everything working smoothly. It's not zero effort. But the result is an AI assistant that actually knows my workflow, runs on my hardware, and costs less than what I was paying before. That's the trade-off, and for me it was worth it.