What Is a Large Language Model?

Definition

ChatGPT doesn't look answers up. At each step, it predicts the next piece of text.

A Large Language Model (LLM) is a neural network, usually a Transformer, trained on huge amounts of text to predict the next token. Repeating that prediction, one token at a time, produces fluent text. "Large" refers to its billions of parameters and the amount of data used to train it.

In simple words

Your phone's keyboard suggests the next word after "Happy", probably "birthday". An LLM is the same idea, made vastly more powerful. It has read so much text that it can continue almost any sentence, which turns out to be enough to answer questions, write code and summarise reports.

Why it matters

LLMs are behind most of today's AI products. Knowing how they work tells you what to expect: why they make things up, why they can be wrong in a confident voice, why long inputs cost more, and why they don't know recent events.

How it works

Your text is split into tokens: common words or pieces of words, each with a number.
Each token becomes a vector, called an embedding, that captures something of its meaning.
Transformer layers use attention to let each token take context from the others.
The model outputs a probability for every possible next token.
One token is chosen, added to the text, and the process repeats.

Two stages of training

Stage	What happens
Pre-training	It reads huge amounts of text and learns to predict the next token. It's now a powerful autocomplete.
Post-training	People's examples and ratings (RLHF and related methods) teach it to follow instructions and be helpful.

The maths

What the model computes at each step

P (next token = k ∣ text so far) = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}

$z_{k}$: The score the network gives token k
$j$: Every token in the vocabulary

In code

The smallest possible language model: count which word follows which, then generate. A real LLM learns far richer patterns, but the job is the same.

tiny_lm.py

import random
from collections import Counter, defaultdict

text = "the cat sat on the mat and the cat saw the dog and the dog sat on the mat"
words = text.split()

follows = defaultdict(Counter)
for current, following in zip(words, words[1:]):
    follows[current][following] += 1

print("After 'the':", dict(follows["the"]))

random.seed(3)
word = "the"
output = [word]
for _ in range(7):
    options = follows[word]
    word = random.choices(list(options), weights=list(options.values()))[0]
    output.append(word)
print(" ".join(output))

Output

After 'the': {'cat': 2, 'mat': 2, 'dog': 2}
the cat saw the mat and the cat

Common mistakes

Believing an answer because it's well written. Fluency isn't accuracy.
Assuming it remembers you. Unless the product adds memory, each new chat starts fresh.
Thinking it counts letters or does arithmetic reliably. Token-based models can stumble at both.

Check yourself

1. What is an LLM fundamentally trained to do?

2. Why do LLMs sometimes state false facts confidently?

Summary

An LLM is a Transformer trained to predict the next token.
Generation is that prediction, repeated.
Pre-training gives knowledge, and post-training gives helpful behaviour.

Read it. Now try it yourself.

Open the interactive activity

Finished reading?

Sources: Stanford CS224N · Hugging Face LLM Course