How Large Language Models Work: Explained Simply

The people who use LLMs the most seem to understand them the least

That isn't an insult. Millions of people use ChatGPT and Claude daily for debugging code, drafting emails and working through ideas, and most have a mental model that is pretty shallow: "it predicts the next word." Technically true. It's also so incomplete that it's close to misleading.

At the other end you have the academic papers, full of dense notation, matrix multiplication and gradient descent, all written for people who already know the material. I have a CS background. I still can't get through most of them in one sitting.

So this post is my attempt at the middle ground: how large language models work, explained so that a curious person without a machine learning degree can follow along. I'll simplify things because this isn't a textbook, but nothing here should be wrong. If you're completely new to AI, start with our beginner's guide to getting started with AI. If you know the basics and want to see how the big models compare in practice, read the GPT vs Claude vs Gemini comparison instead.

By the end you should have a rough picture of what happens between typing a question into ChatGPT and getting an answer back.

Transformers, the architecture under everything

Every major LLM family (OpenAI's GPT models, Anthropic's Claude, Google's Gemini, Meta's Llama, Mistral) runs on the Transformer architecture. It came out of a 2017 Google paper titled "Attention Is All You Need," and I'd put it among the most significant publications in the history of computer science. I don't think that's an overstatement.

Before transformers, language models were built on RNNs (Recurrent Neural Networks) and LSTMs (Long Short-Term Memory networks), which read text one word at a time, left to right. That was slow. They were also bad at long-range dependencies, so if a character was named on page one, the model had a hard time remembering the name by page ten.

Transformers solved this with self-attention: the model looks at the whole input at once and works out which words matter most to which other words. More on that in a bit.

A transformer has two halves: an encoder that reads and understands the input, and a decoder that generates output. The 2017 paper used both because it was doing machine translation. Modern LLMs like GPT-4 keep only the decoder, generating text one token at a time, with each prediction depending on everything that came before it.

The models are stacked in layers, usually dozens and sometimes more than a hundred. Each layer has a self-attention mechanism (the big innovation), a feed-forward neural network that processes each position separately, and some plumbing like layer normalization and residual connections that keeps training stable.

Turning words into numbers

Neural networks can't work with text directly. They need numbers, so before anything reaches the model the text is converted into tokens, which are numerical codes for chunks of text.

Splitting on spaces would be too crude. Instead, tokenizers use algorithms like BPE (Byte Pair Encoding) to break text into subword pieces, where common words get their own token and rare words get split up. "Understanding" might be one token, while "Unfathomable" might become "un", "fath", "om", "able." Numbers, code and non-English text usually get chopped into smaller pieces.

This has three practical effects:

• The model has a fixed vocabulary, usually 50,000 to 100,000 tokens. Everything it reads or writes comes from that set.
• The token count of your prompt decides how much of the model's context window you use up. For example, GPT-4o had a 128K token window and Claude 3.5 Sonnet had 200K, and newer models go well beyond that. That's how much text the model can "see" at once.
• Tokens drive cost. API pricing is almost always per token.

OpenAI has a free tokenizer at platform.openai.com/tokenizer if you want to see exactly how your text gets split.

# Example: Simple tokenization concept (not actual LLM tokenizer)
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
text = "Large language models are fascinating"
tokens = tokenizer.encode(text)
decoded = [tokenizer.decode([t]) for t in tokens]

print(f"Token IDs: {tokens}")
print(f"Decoded tokens: {decoded}")
# Token IDs: [16962, 3303, 4981, 389, 21426]
# Decoded tokens: ['Large', ' language', ' models', ' are', ' fascinating']

Self-attention: the part that makes it all work

Self-attention lets the model work out which words in a sentence are most relevant to each other. The intuition is simple, even if the math gets dense.

Take this sentence: "The cat sat on the mat because it was tired."

"It" refers to the cat. You know that instantly. A computer needs a method, and self-attention gives it one by computing a relevance score between "it" and every other word, which comes out high for "cat" and low for "mat." The model then builds a representation of "it" that leans heavily on the meaning of "cat."

Multi-head attention runs this several times in parallel through different "heads," and each head can pick up a different kind of relationship. One might learn syntax, like subject-verb agreement, while another handles meaning (what a pronoun refers to, say) and others focus on nearby words or track high-level themes. The outputs get combined into a rich representation of each token.

Stripped of the matrix math, a single attention step looks like this:

1. For each token, the model creates three vectors: Query (Q), Key (K), and Value (V)
2. The Query of one token is compared against the Keys of all other tokens, which produces the attention scores
3. The scores are normalized (softmax) so they add up to 1
4. Those scores are used to take a weighted average of the Value vectors

The analogy that works best for me is a library. Your Query is what you're looking for, the Keys are the index entries for every book, and the Values are the books' contents. You compare your search against every index entry, see which books are most relevant, and blend their contents in proportion to how relevant each one is.

How the model learns, and what it costs

Training an LLM is simple in concept and computationally insane in practice.

Pre-training comes first. The model is fed a massive pile of text (web pages, books, papers, code, forums and more), which for frontier models like GPT-4 or Claude means trillions of tokens. On paper the task is dead simple. Given a sequence, predict the next token. If the real next token is "dog" and the model guessed "cat," its parameters get nudged slightly so "dog" is more likely next time in a similar context.

This repeats billions of times. What the model picks up along the way is statistical patterns rather than memorized text: how language flows, what facts get stated, how reasoning typically goes, what code syntax looks like.

The scale is hard to wrap your head around:

Fine-tuning is next. A raw pre-trained model is a powerful text predictor and a terrible assistant, one that will happily generate toxic content, make up facts, or ignore what you asked, because it has no idea it's supposed to be helpful. Fine-tuning adjusts it using curated examples, thousands of prompt-response pairs that show the helpful, honest behaviour the developers want.

RLHF (Reinforcement Learning from Human Feedback) is the final polish, and it's why ChatGPT and Claude feel conversational instead of like a random text generator. The model generates several responses to the same prompt, and human raters rank them from best to worst. A separate "reward model" learns to predict those preferences. Then the LLM is trained further to maximize that model's score.

That's also why ChatGPT and Claude feel different even though the architectures underneath are similar. Their RLHF reflects different priorities. Anthropic (Claude's maker) puts the emphasis on safety and careful answers, while OpenAI has pushed harder on general capability and engagement. Some newer models use variations like DPO (Direct Preference Optimization) or RLAIF (RL from AI Feedback), which are more efficient and get similar results.

Temperature, top-p, and why the same question gives different answers

At each step the model produces a probability for every token in its vocabulary. Two settings control how it picks from those probabilities, and they matter a lot.

Temperature controls randomness. Say the model predicts "the" at 40%, "a" at 25%, "one" at 15%, "some" at 10%, and everything else at 10%. At temperature 0 it always picks "the," the highest-probability token, so the output is deterministic and repetitive. At temperature 1 it samples according to the real probabilities (a 40% chance of "the," 25% of "a," and so on), so the output is varied and more creative. Above 1 it gets chaotic. The distribution flattens further and unlikely tokens become more probable.

Top-p (nucleus sampling) is a smarter filter. Rather than reshaping the whole distribution, it limits the choice to the smallest set of tokens whose combined probability exceeds p. With top-p at 0.9, the model only considers the most likely tokens that together cover 90% of the probability mass. Wildly unlikely tokens are cut. There's still room for interesting variation.

Most chatbot deployments run temperature around 0.7 to 1.0 with top-p around 0.9 to 0.95. For code generation, a lower temperature (0.2 to 0.5) gives more reliable output. For creative writing, a higher one (0.8 to 1.2) produces more interesting results.

Context windows: how much the model can see at once

The context window is the maximum number of tokens the model can handle in a single conversation turn. Your input and the model's output both count toward it. Some well-known examples from the GPT-4o and Claude 3.5 generation (newer versions of these models have larger windows):

A bigger window means longer documents, longer conversations and more information available when the model writes a response. Gemini's 2M token window can take in entire novels or full codebases.

There's a catch. Models don't pay equal attention to everything in the window, and research shows they're better at using information near the beginning and the end while tending to lose track of things in the middle (people call this the "lost in the middle" problem).

When models confidently make things up

If you remember one thing from this post, make it this: LLMs hallucinate. They produce text that sounds confident and plausible and is factually wrong. They'll cite papers that don't exist, attribute quotes to people who never said them, and describe events that never happened.

The reason is that LLMs are statistical pattern matchers. They aren't knowledge databases. They generate whatever text is statistically likely given the patterns they absorbed in training, and if a plausible but wrong fact fits the pattern, the model has no way to tell it apart from a real one.

Some examples:

• Citing a research paper with a real-sounding title, believable author names and a specific DOI, none of which exist
• Getting a historical date wrong by several years and stating it with total confidence
• Writing code that calls an API function the library doesn't have
• Giving legal advice that cites statutes nobody ever wrote

A few habits help. Verify factual claims against real sources, and where you can, use RAG (Retrieval-Augmented Generation), where the model first pulls relevant documents from a database and then answers from them. Ask for citations and check them yourself. Test code before you trust it. And for anything where accuracy matters more than creativity, turn the temperature down.

LLMs vs search engines: different tools, different jobs

People mix these up a lot.

Use both. LLMs for reasoning, synthesis and drafting; search engines for fact-checking, recent information and primary sources.

The major model families

Model versions change every few months, so the notes below describe each family as of the GPT-4o, Claude 3.5 and Gemini 2.0 generation; check each provider for its current lineup.

GPT-4o from OpenAI became the most widely used model. It's strong at general tasks, coding and multimodal work (text, images and audio), and the "o" stands for "omni" because it handles multiple input and output types natively. You get it through ChatGPT and OpenAI's API.

Claude 3.5 Sonnet from Anthropic is known for strong reasoning, a 200K token context window, careful responses that handle grey areas well, and really good coding. It tends to be more cautious. The free tier on claude.ai is generous.

Gemini 2.0 is Google's flagship, with that 2M token context window. It's deeply integrated with Workspace (Docs, Gmail, Drive) and has strong multimodal capabilities. In the Gemini app, the Pro version went to paying Gemini Advanced subscribers.

Llama 3 from Meta is the most capable open-source model family. The 405B version, which arrived with the Llama 3.1 update (along with the 128K context window; the original Llama 3 had 8K), is competitive with GPT-4 on a lot of benchmarks. Open weights mean you can run it locally, fine-tune it and deploy it without API costs. The smaller 8B and 70B variants work on consumer hardware.

Running models on your own machine is one of the most exciting developments. Ollama, LM Studio and llama.cpp make it pretty easy to download and run open-source models locally.

# Running Llama 3 8B locally with Ollama
ollama pull llama3:8b
ollama run llama3:8b "Explain quantum computing in simple terms"

A laptop with 16GB RAM handles 7-8B parameter models comfortably. A desktop with 32GB RAM and a decent GPU manages 13-30B models. The full 70B or 405B versions need serious hardware, meaning multiple high-end GPUs with lots of VRAM. Local models are slower than cloud APIs, and the small ones are less capable than frontier models, but your data never leaves your machine, there's no API cost, and you can fine-tune for specific tasks. Fair trade, for a lot of use cases.

What LLMs can't do

The hype hides the real limits, so here they are.

They can't reason reliably about truly new problems. They pattern-match against what they saw in training. When a problem is unlike anything in the training data, the "reasoning" falls apart. Some of what looks like reasoning may just be matching against problems that look similar on the surface.

Math is unreliable. Large-number arithmetic, multi-step calculations and proofs are all shaky without external tools. That's why models increasingly hand math off to Python instead of computing it "in their heads."

There's no persistent memory. Each conversation starts fresh unless the platform adds a memory feature on top, and the model won't remember yesterday's chat unless that history is explicitly fed into the current context.

The base model has no real-time internet access either. Its knowledge is frozen at the training cutoff, and while some platforms bolt on search as a plugin, the core model isn't browsing the web.

They don't understand things the way we do. This one is more philosophical, but I think it matters. LLMs process statistical patterns in text, and whether that counts as "understanding" is a deep debate in AI research. What's clear is that it's different from human cognition. They have no embodied experience and no common-sense grounding in the physical world, and they struggle with causal reasoning in plenty of situations.

Where I think this is going (predictions in this field are almost always wrong)

A few directions seem fairly likely. Multimodality is becoming the default: models will natively handle text, images, audio, video and structured data, and the line between "language model" and "vision model" is already blurring. Image generation in particular is quite mature already, and our AI image generation guide for 2026 covers the current tools and techniques. Reasoning is improving through techniques like chain-of-thought and reasoning-focused training. Models are shrinking too, since distillation and quantization mean a phone in 2027 might run something as capable as a 2024 cloud model. And tool use is becoming central, with models learning to call calculators, code interpreters, web browsers and databases instead of doing everything through language.

Then there's the hard one: safety and alignment are still unsolved. As models get more capable, making sure they do what we want, as opposed to whatever they were statistically trained to produce, gets harder.

Nobody fully understands where this technology is heading. The people building these systems openly say they're surprised by what the models can do, which is exciting and a little unsettling. Knowing the mechanics (transformers, attention, training, hallucinations) won't tell you where it ends up. It will help you use these tools better, notice when they fail, and decide when to trust the output and when to double-check, and with things moving this fast I'd count that as a skill worth having.