An always-on assistant, driven from the phone's chat app (no app to build), that understands voice notes and runs entirely on owned hardware — no per-token bill, no audio or messages traveling to a third party. The trick: split it across two machines by strength.
This page describes a system running on my own hardware. The repository publishes the architecture and the decisions, not the implementation. The components that do ship code have their own repositories, linked below.
The always-on node holds state and connectivity; the GPU node provides on-demand capacity. If the GPU is asleep, the body still responds and queues work for when it wakes up.
24/7 services: agent orchestrator, chat gateways, work queues. Cheap to keep running forever; a stable host for everything that must never sleep.
Heavy compute: local LLM inference + speech-to-text. Only powers on when needed; does the work the mini-PC can't. They connect over a private VPN mesh.
A daemon-gateway exposes a local HTTP API with a token. The security edge is SSH: ssh mini-pc "send ...". Any tool — even an AI session — reads and sends messages without ever holding the credentials.
The note is queued on the always-on node, transcribed on the GPU with a local Whisper (CUDA), and the text returns to the chat tagged with who sent it. The audio never leaves home.
The orchestrator runs as a persistent service and uses the GPU's local LLM (via an OpenAI-compatible endpoint over the mesh) as its reasoning engine — persistent memory, tool use, accessible from chat.
If the GPU node is off, the always-on node still responds and queues. Nothing hard-fails because one machine was asleep. The cloud is an option, never a dependency.
| Concern | Choice |
|---|---|
| Always-on host | Linux mini-PC, systemd services (Restart=always) |
| Heavy compute | GPU workstation (24 GB), powered on-demand |
| Local LLM | Ollama (OpenAI-compatible API) |
| Speech-to-text | Whisper.cpp (CUDA build, large model) |
| Network | Tailscale mesh VPN — nodes reachable from anywhere, private |
| Chat control | Gateway daemon + local API with token; ssh as the trust boundary |
Separate always-on connectivity from on-demand capacity. Cheap to keep alive, powerful when it matters.
If the GPU is off, the always-on node responds and queues. Nothing hard-fails because one machine was asleep.
Audio, messages, and reasoning stay on owned hardware. The cloud is an option, not a dependency.
Services bind to localhost + token; remote access is only through the mesh + SSH, so secrets never travel.