Memory and knowledge: grounding agents with RAG

The model knows general things from training, but it does not know your prices, your policies, or last week's orders, and its training has a cutoff date. To be useful, an agent has to be grounded in your specific, current information. This lesson is about how.

Two kinds of memory

Short-term memory is the context window: the current conversation plus whatever you inject for this request. It is precise but small and temporary. Long-term memory is everything outside that, your documents, database rows, past conversations, that you store somewhere and pull from when relevant. The skill is deciding what to load into the limited context for each request.

Retrieval-augmented generation (RAG)

RAG is the standard way to give an agent your knowledge without retraining it. The idea: when a question comes in, find the few most relevant pieces of your own content and paste them into the context, then let the model answer from them. The model stays general, the facts come from you, and you can update the facts any time by changing your documents.

Embeddings, in plain terms

An embedding turns a piece of text into a point in space, positioned so that texts with similar meaning sit close together. "What time do you close?" and "opening hours" land near each other even though they share no words. That is why retrieval finds the right chunk by meaning, not keyword matching. Closeness is just distance between those points.

RAG vs fine-tuning

People often ask whether they should fine-tune (retrain) a model on their data. For most businesses the answer is no, use RAG. Fine-tuning is expensive, has to be redone whenever facts change, and is poor at facts anyway, it shapes style and behaviour, not knowledge. RAG is cheap, updates instantly when you edit a document, and lets the agent cite its source. Reach for fine-tuning only when you need a specific tone or format at scale, not to teach facts.