Week 2 — Inside the Machine
Sign in
The Visible LLM · Lesson 2 of 6
Week 2 — Inside the Machine
Tokens, Embeddings, Attention, Transformers

Adventure 8

Where We Left Off
A Quick Week 1 Gut-Check
  • An LLM is a next-word-prediction machine — that's it
  • It picks one token at a time from a probability distribution
  • "Intelligence" is an emergent side effect of doing this at huge scale
  • If anything still feels fuzzy, pause here

Adventure 8

Today's Outcomes
What You'll Know in 3 Hours
This is the most technically dense week. Don't worry if some of it slips past on first contact — we'll revisit it all in Week 3.
Tokens
Explain how a sentence becomes tokens — and why it's not words
Embeddings
Describe embeddings as positions on a giant map of meaning
Attention
Walk through how attention lets the model "look back" at earlier words
Transformers
Sketch the inside of a Transformer block — at a level most adults can't

Adventure 8

Warm-Up
Let Me See What You Brought
Before the technical content — your 5 project ideas from last week.
For each one, three quick questions:
What's the input?
What does the user provide or type in?
What's the output?
What does the system return?
Who would actually use this?
Name a real person, not "anyone"

We are not eliminating anything yet. Just clarifying.

Adventure 8

Part 1 — Tokens
How Does "Apple" Get Inside the Machine?
If you had to teach a computer to read English from scratch — what would you have it look at?
Three obvious options, each with a serious problem:
Whole words
Vocab explodes; can't handle "ungoogleable"
Single characters
Too slow; the model has to relearn "the" every time
Something in between
This is what real LLMs do — and it's called a token

Adventure 8

Tokens Are Lego Blocks
The metaphor we'll use all day: a giant box of Lego blocks.
  • Some blocks are whole common words ("the", "cat", "and")
  • Some are pieces of words ("ing", "tion", "un")
  • Some are single characters or punctuation
  • The model snaps these blocks together to read your sentence

Adventure 8

The Tokenizer "Reads Greedy"
The algorithm that builds the token vocabulary is Byte-Pair Encoding (BPE).
01
Start with single characters
02
Find the most common pair
e.g. "t" + "h" = "th"
03
Merge it into a new token
04
Repeat thousands of times
End up with ~50K–200K reusable chunks

The model "discovers" its own Lego set by reading billions of documents.

Adventure 8

Knowledge Check — Try It Live
platform.openai.com/tokenizer
Open the OpenAI tokenizer in your browser. Type these in and watch what happens.
Start simple — predict before you look:
Input
Tokens
hello
1 token
Hello, world!
4 tokens
unbelievable
2 tokens
antidisestablishmentarianism
Predict first ❓

Adventure 8

Knowledge Check — Push It Harder
Mix Languages and See What Happens
Count the tokens for each — some results will surprise you:
Input
Language
Tokens
"hello"
English
1
"你好"
Chinese
?
"こんにちは"
Japanese
?
"مرحبا"
Arabic
?
"🎉🚀"
Emoji
?

Non-English languages often use more tokens per character. Same idea costs more on the API bill.

Adventure 8

Why ChatGPT Can't Count Letters
A famous puzzle: ask an LLM "How many R's are in 'strawberry'?"
Many say 2 instead of 3. This isn't stupidity — it's architectural.
  • The model sees straw + berry as two tokens
  • It never sees the individual letters
  • It's counting blocks, not characters

Adventure 8

Tokens Are How You Pay for AI
Every API call is billed by the token. This is where tokens stop being abstract.
Model
Input cost (per 1M tokens)
Claude Sonnet 4
$3
GPT-4o
$2.50
GPT-4o-mini
$0.15

A 500-page book ≈ 200K tokens ≈ $0.60 to feed to Claude once. Building a product? Token math = profit math.

Adventure 8

✓ Takeaway
Tokens — The Takeaways
Lock these in before we move to embeddings:
Sub-word chunks
Tokens are not words — they're pieces
50K–200K vocab
BPE builds it by merging common pairs
Non-English costs more
Same idea, more tokens, higher bill
No individual letters
The model never sees inside a token
Your API bill is a token bill
Token count = cost

Adventure 8

Part 2 — Embeddings
Now What? From Tokens to Numbers
The model has tokens with ID numbers — say [15496, 11, 995] for "Hello, world!"
But these are just IDs. They identify the token the way a jersey number identifies a player. They don't carry meaning.
How do we teach the model that "cat" is more like "dog" than like "rocket"?

Adventure 8

Imagine a Giant Map — Words Live on It
Picture a map — not 2D, a map with hundreds of dimensions.
Your brain can't visualize that. Mine can't either. So we'll use 2D for illustrations. But the principle holds:
  • Every token gets a coordinate on this map
  • Similar meanings cluster together
  • Distance = semantic similarity
  • This coordinate is called an embedding

Adventure 8

"You Shall Know a Word by the Company It Keeps"
A 1957 idea by linguist J.R. Firth — modern LLMs are its most successful implementation ever.
The model learns embeddings by reading billions of sentences and noticing context:
Which words appear near which
Co-occurrence patterns across billions of sentences
Similar neighbors → similar coordinates
Words that hang out together end up close on the map
No human labels anything
The data does the work entirely

Adventure 8

King − Man + Woman = ???
This genuinely shocked AI researchers around 2013.
Take the embedding for King. Subtract the embedding for Man. Add the embedding for Woman.
The closest point on the map is: Queen.
The model never learned this rule. The geometry of the map just... worked out that way.

This is called vector arithmetic — and it works for dozens of relationships.

Adventure 8

It Works for Many Relationships
King–Queen isn't a fluke. The same arithmetic works for dozens of relationships:
Arithmetic
Result
Paris − France + Germany
≈ Berlin
Walking − Walk + Swim
≈ Swimming
Bigger − Big + Small
≈ Smaller
The map captures not just meanings, but relationships between meanings.

Adventure 8

The Map Reflects the Data
Doctor − Man + Woman ≈ Nurse
This isn't random. It's a documented bias — and it matters for real products.
The map reflects whatever's in the training data
Patterns — good and bad — get encoded
Stereotypes in the corpus → stereotypes in the embeddings
Not intentional, but real and measurable
This is why bias-mitigation work exists
Real products have to actively correct for this

Adventure 8

✓ Takeaway
Embeddings — The Takeaways
Lock these in before we move to attention:
High-dimensional map
Modern LLMs use 1,000–12,000 dimensions per token
Similar meanings cluster
Distance = semantic similarity
Vector arithmetic works
King − Man + Woman = Queen — the geometry just works
Biases included
The map reflects the training data, flaws and all

Adventure 8

Part 3 — Attention
How Does AI Understand "She"?
"Mike gave the book to his sister because she wanted to read it."
Your brain resolved all of this instantly. Let's slow it down:
Word
Refers to
she
his sister — not Mike
it
the book — not the sister
his
Mike — not the book
How does the model do this?

Adventure 8

Attention = Pointing a Spotlight
As the model processes each word, it has a spotlight that illuminates earlier words by relevance.
🔆 Bright
"Pay attention to this word"
🔅 Dim
"Mostly ignore this word"
The model's "thinking" is just: redistribute attention across earlier words.

Adventure 8

The Model Has Multiple Spotlights
The model doesn't have ONE spotlight. It has many — typically 12 to 128.
  • Each spotlight specializes in a different kind of relationship
  • One tracks subjects → verbs
  • One tracks pronouns → their referents
  • One tracks adjectives → nouns
  • All run in parallel; results combine
This is called multi-head attention.

Adventure 8

Why This Beat Older Approaches
Earlier neural networks processed text sequentially — like reading with a finger, one word at a time.
1
Sequential (old)
Slow — can't parallelize. Forgets long-range references. Struggles with deeply nested sentences.
2
Attention (new)
Reads everything at once. Every word sees every other word. Parallel, fast, accurate over long range.

Adventure 8

The Hidden Cost
Every word looking at every other word means N×N comparisons.
Sentence length
Comparisons
100 words
10,000
1,000 words
1,000,000
100,000 words
10,000,000,000

This is why long context windows are expensive — and why every model has a hard limit.

Adventure 8

Knowledge Check — See It Live
BertViz Demo — Google Colab
Open a BertViz notebook in Google Colab. We're going to literally watch attention happen.
01
Pick a sentence
02
Run the model
03
Watch the notebook draw lines
Lines connect words showing which heads are looking where

Some heads are interpretable. Most are weird and beautiful.

Adventure 8

When Attention Fails
Attention isn't magic. It fails in predictable ways:
Long-range references
"The book that the man who lived next to the woman with the cat picked up..." — too much nesting
Distractors
Irrelevant nearby words steal attention from the relevant ones
Position confusion
When many similar words appear close together
Repetition loops
The model attends to its own recent output and gets stuck

Understanding these failures = better prompts in Week 3.

Adventure 8

✓ Takeaway
Attention — The Takeaways
Lock these in before we move to Transformers:
Spotlight on relevance
Attention points at earlier words that matter right now
Many heads in parallel
12–128 spotlights, each tracking a different relationship
Replaced sequential processing
Faster, better at long range
Cost grows N×N
Long context windows are expensive by design
Can fail
Long-range refs, distractors, repetition — know the weak spots

Adventure 8

Part 4 — Transformers
A Transformer Is Just Attention, Stacked
We've seen:
Tokens — Stage 1: input
Sentence → sub-word chunks with IDs
Embeddings — Stage 1: meaning
IDs → coordinates on a high-dimensional map
Attention — Stage 2: thinking
Every word points a spotlight at relevant earlier words
A Transformer stacks these into many layers — like a multi-floor factory where your sentence rides an elevator up, getting refined on each floor.

Adventure 8

One Floor of the Factory
Every floor of the Transformer has the same internal structure. Three sub-components, repeated identically:
1
Multi-head attention
The spotlights — all heads running in parallel
2
Feed-forward network
A small neural net applied per token
3
Layer norm + residual
Keeps things numerically stable

The magic isn't any one floor. It's what the model learns across all of them.

Adventure 8

Encoder vs Decoder
The original 2017 paper had both. Modern models split them apart and specialize.
Type
What it does
Used for
Encoder-only (BERT)
Read & understand
Search, classification, embeddings
Decoder-only (GPT, Claude)
Read & generate
ChatGPT, Claude, every chatbot
Encoder-decoder (T5)
Read one, generate another
Translation, summarization

★ For this course we focus on decoder-only — Claude, GPT-4, Gemini are all this shape.

Adventure 8

A Short History of Architecture Battles
Around 2018, both BERT and GPT-2 were released. The AI community split into two camps.
Camp 1 — BERT-style (2018–2020)
"Understanding is the hard part. Build encoders."
Camp 2 — GPT-style (2019–now)
"Just predict the next token. At scale, everything else falls out."
Camp 2 won. Decoder-only models scaled better. ChatGPT was the proof.

Adventure 8

The "Emergent Abilities" Surprise
A discovery that shocked researchers: when you make Transformers bigger, abilities emerge suddenly at certain scales.
Scale
Arithmetic ability
Below 10B parameters
Bad at arithmetic
Above ~50B parameters
Arithmetic works

The transition is sharp, not gradual. This is genuinely strange and not fully understood. It's why nobody can predict what GPT-6 will do.

Adventure 8

✓ Takeaway
Transformers — The Takeaways
You now know the fundamental architecture of every commercial AI system in 2025.
Stacked layers
A Transformer = many layers of attention + feed-forward
Decoder-only dominates
GPT, Claude — these power every chatbot you've used
80–120 layers deep
Modern frontier models
Emergent abilities
Abilities can appear suddenly at scale — still not fully understood

Most adults — including most software engineers — couldn't explain this. After 6 hours of class, you can.

Adventure 8

Hands-On Time
Three Filters — 5 Ideas → 2
Time to narrow. For each of your 5 ideas, run them through these three filters. Be honest.
1
The Input/Output Test
Can you describe both in one sentence?
2
The "Who Cares?" Test
Can you name a real person who'd use it?
3
The 3-Hour Test
Could a working prototype exist in 3 hours?
The goal: find the 2 ideas that survive all three filters.

Adventure 8

Run Each Idea Through the Filters
Let's work through each of your 5 ideas together, one at a time.
For each idea:
01
Read it aloud
02
Apply Filter 1, then Filter 2, then Filter 3
03
Mark your verdict
SURVIVES / KILLED / RESCUE WITH TWEAK

Take your time. This is the most important decision-making exercise of the course.

Adventure 8

Week 2 Recap
The 3-Hour Summary
Six concepts. Let's run through them one more time:
01
Tokens
Sub-word Lego blocks; 50K–200K vocab; BPE; cost = token count
02
Embeddings
Every token = coordinate on a high-D map; distance = similarity
03
Vector arithmetic
King − Man + Woman = Queen — the geometry just works
04
Attention
Model points a spotlight at relevant earlier words
05
Multi-head
Many spotlights in parallel; each tracks a different relationship
06
Transformer
Many layers of attention + FFN, stacked

Adventure 8

Next Week — Week 3
Generation & Prompts
Week 3 covers Stage 3: how the model actually picks the next word. This is the week that immediately upgrades how you use AI day-to-day.
  • 🎲 Autoregressive generation — one token at a time
  • 🌡️ Temperature — the creativity dial
  • 🎯 Top-k / Top-p — the cutoff dials
  • 🎭 System vs user prompts — what ChatGPT hides from you
  • 🛠️ 5 patterns that always work + 5 anti-patterns to avoid

This is also where we narrow your 2 ideas down to 1.

Adventure 8

Before Week 3
Between Now and Week 3
Three things to do before next class. None of them are hard.
🧠 Sit with your 2 ideas
Don't pick yet. Imagine building each. Imagine showing each to a friend in 6 months.
🔍 Play with platform.openai.com/tokenizer
Type 10 weird strings. Notice the patterns.
📖 Re-read the Attention section
It'll click harder the second time.
See you next week.

Adventure 8