Back

Setting up a personal AI OS

I set up a self-hosted AI assistant and read how other people run theirs. The model turned out to be the easy part. Most of the work, and most of what made it useful, was the wiring around it. These are my notes.

The stack

The setup I ended on is common: Ollama to run the local model, Open WebUI as the chat front end, n8n for automation, all in Docker behind a reverse proxy. I put the reverse proxy up before exposing anything. Getting the model running took an afternoon. Connecting it to things I actually do, through scheduled jobs and tool calls in n8n, took much longer, and that is the part that makes it an assistant instead of a chat box. My takeaway was to spend time on the plumbing, not on chasing a bigger model.

Files instead of a vector database

I assumed I needed a vector database with embeddings. I tried plain full-text search over files instead, and at single-user scale it worked better for me.

SQLite full-text search, or a simple index over a folder of markdown notes, was faster and cheaper than the vector setup. The main advantage was that when it returned the wrong thing I could open the file and see exactly what it had stored. I could not do that with a vector store. One build I looked at, Pepper, runs its whole assistant on full-text search over an Obsidian vault with no embeddings. I also kept a small file with basic facts (timezone, key people, a few defaults) loaded at the start of each session so it stopped asking me the same context every time. If you are serving one person, keep the store in a format you can read.

Scheduled agents

The setup started to feel like an OS once I let agents run on a schedule without me. The pattern I copied runs a few cron jobs: create the daily note in the morning, and late at night reconcile contradictions between notes and clean up loose ends, plus a weekly review. New information rewrites existing notes instead of adding new ones. The knowledge base maintaining itself overnight is the difference between a tool you operate and one that runs on its own.

Local versus cloud

Running everything locally has real limits:

  • It is slower. A modest response often took 10 to 15 seconds.
  • The smaller quantized models are less reliable. Structured JSON output that worked at full precision broke when I ran a heavily quantized model.
  • Local context is smaller than what a cloud model gives you.

I stopped trying to run everything locally. The setup that stuck is hybrid: the local model handles the easy majority of requests, and I send only the hard ones to a cloud model, with no raw personal data leaving the machine. That kept the cost and privacy benefit for most of the work without fighting the local model on the things it does badly.

What worked in the end

  1. A memory store I can read (markdown or SQLite), kept in git, and fix by hand.
  2. Scheduled agents doing maintenance overnight.
  3. The right small model per task, with a cloud fallback for the hard requests.
  4. Tool-calling so the assistant does things instead of only answering.
  5. Reachable from where I already work, not only a web tab.

References