Building a small RAG over my own notes

I’ve got about 800 pages of notes going back to 2020. Markdown files, mostly. Some in Obsidian, some in plain text, some in Apple Notes, some in code comments. I wanted to ask questions across all of them. So I built a small RAG.

The setup

• 2000 chunks of ~500 tokens each, with 50-token overlap
• Embeddings: I tried four. OpenAI’s text-embedding-3-small, Cohere’s embed-english-v3.0, a local bge-large-en-v1.5, and nomic-embed-text-v1.5
• Vector store: I started with a hosted one (Pinecone free tier). Moved to a local chromadb after a week because the network latency was annoying
• Retriever: top-5 chunks by cosine similarity, no re-ranking
• Generator: the cheap model with a 2000-token context window
• Interface: a CLI that takes a question and prints the answer plus the chunk sources

It took a weekend. The hard parts were not the ones I expected.

Chunking matters more than the model

I assumed the embedding model was the lever. It isn’t. The chunking strategy is. With 500-token chunks and 50-token overlap, my notes about “Proxmox networking” and “pfSense VLANs” got split across multiple chunks, and the embedding for each chunk had a different topic. When I asked “what’s the right way to set up VLANs for the lab,” the top-5 results included 3 chunks about pfSense, 1 chunk about Proxmox networking that happened to mention VLANs, and 1 chunk about my home network that was completely irrelevant.

I rewrote the chunker to be semantic-aware: it splits on heading boundaries and never splits inside a code block. The number of chunks went from 2400 to 1900. The retrieval quality went from “kinda works” to “actually works.” The model didn’t change. The chunker did.

The local embedding model is “good enough”

bge-large-en-v1.5 runs on my desktop GPU in about 80ms per query. The retrieval quality is within a few percent of text-embedding-3-small on my actual data. The hosting cost is zero. I switched to the local model and haven’t looked back.

Cohere was the best on paper but the hosted API was slower than I’d like and the local versions of their model were behind the API. OpenAI’s was fast and good but tied to my OpenAI bill.

The generator lies confidently

The model says things that aren’t in the retrieved chunks. I added a strict prompt: “answer using only the information in the chunks below. If the chunks don’t contain the answer, say so.” It still hallucinates about 1 in 20 queries. I added a guard that compares the model’s answer to the retrieved chunks using a simple keyword overlap and warns me if the overlap is below 30%. The warning is not 100% accurate but it catches the worst cases.

What I actually use it for

I ask it things like “what did I say about GPU passthrough last time” and “summarize my notes on the OPNsense firewall.” Both work. I don’t ask it anything I can’t verify by reading the source chunks. The CLI prints the source chunks so I can always check.

Total cost: a weekend and a coffee per month on the generator. The embedding model runs on my own hardware. The vector store is a single file on disk.

This isn’t a product. It’s a personal tool. But it’s the most useful thing I’ve built this year. If you have a pile of notes, build one.