How a language model actually works

Before you can build an agent, you need an honest mental model of the thing underneath it: a large language model (LLM) such as GPT, Claude, or Gemini. Almost every mistake people make with AI comes from misunderstanding what the model is doing.

It predicts the next token

An LLM does one thing: given a stretch of text, it predicts the next chunk of text, one token at a time. A token is roughly a word fragment (about four characters of English). It does not look anything up, it does not reason in the human sense, it produces the most statistically likely continuation based on patterns learned from a huge amount of text.

This single fact explains its strengths and its failures. It writes fluent language because fluent language is highly predictable. It invents facts because a plausible-sounding continuation is not always a true one. We call those inventions hallucinations, and they are not bugs, they are the model doing exactly what it was built to do.

The context window is its working memory

The model has no memory between calls. Everything it knows in the moment is the text you send it, called the context. The maximum size of that text is the context window, measured in tokens. When people say an agent remembered a previous message, what they mean is the application re-sent that message inside the context on the next call.

Roles: system, user, assistant

A chat request is a list of messages, each with a role. The system message sets the rules and persona and is the single most important lever you have. The user messages are the human turns. The assistant messages are the model's previous replies. Getting the system message right is most of the work of building a reliable agent.

system:  You are the front desk for "Kopi Lima", a coffee shop in
         Jakarta. Answer only from the facts below. If a question is
         outside them, say a staff member will follow up. Never invent
         prices. Reply in the customer's language.
         FACTS: Open 07:00-22:00 daily. Espresso 25k, Latte 32k.
         Address: Jl. Melati 5. Delivery via GoFood only.
user:    bisa pesan untuk 20 orang besok pagi?
assistant: Untuk pesanan 20 gelas, tim kami akan follow up ya untuk
         konfirmasi stok dan waktu. Boleh saya minta nomor WhatsApp Anda?

Temperature controls randomness

Because the model picks from a distribution of likely next tokens, you can tune how adventurous it is. Temperature near 0 makes it pick the single most likely token almost every time, giving consistent, repeatable answers. Higher temperature adds variety, useful for brainstorming, risky for facts. For business agents that answer questions or follow rules, keep it low.

Prompting that works

A good prompt is not magic words, it is clear specification. Four parts cover most of it: the instruction (what to do), the context (the facts and data it may use), the constraints (what it must not do), and optionally examples of the input-output you expect. If the model behaves badly, the fix is almost always a clearer prompt, not a bigger model.