Week 3 — Generation & Prompt Engineering
Sign in
Adventure II · Week 3
Week 3 — Generation & Prompt Engineering
You know what's inside the box. Now learn how to drive it.

Adventure 8

Last time on Adventure II
Where We Left Off
  • Week 1: An LLM is a next-word-prediction machine
  • Week 2: Inside the machine — tokens, attention, transformers
  • Week 3 (today): How YOU make it say what you want

Adventure 8

Learning outcomes
What You'll Know in 3 Hours
By the end of today, you can:
Explain randomness
Why an LLM gives different answers to the same prompt
Pick the right temperature
For any task — code vs creative writing
Use both prompt types
Tell the difference between system prompt and user prompt — and use both
Write reliable prompts
5 patterns that consistently get better output
Debug bad prompts
Diagnose a broken prompt and fix it in 2–3 iterations

Adventure 8

Before we begin
The Deal — 3 Ground Rules
1
You drive the API, not the chat app
We'll use the playground, not ChatGPT.com
2
Predict before you press run
Every experiment: predict what you'll see, then check
3
Iterate, don't beg
Bad output ≠ "ask nicer." Bad output = "change the prompt structure."

Adventure 8

Warm-up
Before We Start — Quick Calibration
Three questions to wake your brain up:
Question 1
If you ask ChatGPT the exact same question twice, do you get the same answer? Why or why not?
Question 2
What's the difference between "high temperature" and "low temperature" in an AI?
Question 3
Have you ever written a prompt that didn't work? What did you change?

Don't look anything up — just share your best guess. There are no wrong answers here yet.

Adventure 8

Part 1 — How generation actually works
Generation Is Not What You Think
The model doesn't pick a word. It picks a probability distribution.
  • Every step: model outputs a probability for EVERY token in its vocabulary (~100,000 tokens)
  • Then it samples one token from that distribution
  • Repeat until done

Adventure 8

Watch it in slow motion
Prompt: "The capital of France is"
The model's actual output — probabilities for the next token:
Token
Probability
Paris
0.87
the
0.04
a
0.02
located
0.01
... 99,996 more
tiny
Paris usually wins. Not always.

Adventure 8

Let's clear up some myths
Three Common Misconceptions
Most people believe at least one of these:
❌ "ChatGPT picks the best answer"
It picks ONE answer from a probability distribution — not the "objectively best" one
❌ "Same prompt = same answer"
Almost never — because of sampling, the dice roll again every time
❌ "Setting temperature=0 makes it deterministic"
Not quite — even then, tiny differences can creep in (more on this soon)

Adventure 8

📘 Concept card
Sampling
What: Picking one token from a probability distribution over all possible tokens.
Like: Spinning a weighted roulette wheel — bigger pockets are more likely but small ones can still hit.
Why it matters: This is where every bit of "AI creativity" comes from. It's literally random.
Every word the model writes is one spin of that wheel. Change the wheel shape → change the output.

Adventure 8

The first dial
Temperature — The Creativity Dial
A single number that reshapes the probability distribution before sampling.
Setting
Effect
T = 0
Highest-probability token always wins. Most predictable, most boring.
T = 0.7
Roughly the default. Some variety, still focused.
T = 1.0
Raw probabilities, no rescaling.
T = 1.5+
Flattens distribution. Low-prob tokens get a real chance. Things get weird.

Adventure 8

Watch Temperature Live
Same prompt, three temperatures:
T = 0
"The cat sat on the mat."
T = 0.7
"A small orange cat curled up on the windowsill."
T = 1.5
"Whiskers, those tiny diplomats of midnight, negotiate with the moon."

Adventure 8

When to use which setting
When to Set T Low, When High
Task
Temperature
Code, math, factual lookup
T ≈ 0
Summarization, translation
0.2 – 0.5
Brainstorming, writing
0.7 – 1.0
Push the model out of a rut
> 1.0

Rule of thumb: Random hurts → T low. Random helps → T high.

Adventure 8

Knowledge check
Quick Gut Check — Mid-Class Checkpoint
Two questions to answer out loud (not in your head):
Question 1
Why does a higher temperature give more "creative" output, mathematically?
Predict your answer before discussing.
Question 2
If you wrote a Python script that needs the same answer every run, what temperature would you set?
And would that actually guarantee identical output?

Adventure 8

The second dial
Top-k — The Hard Cutoff
Temperature reshapes the distribution. Top-k chops off the tail.
What it does
Top-k = N means: only consider the N most likely tokens. Throw the rest away.
Top-k = 1
Only the top token survives — equivalent to T = 0. No variety at all.
Top-k = 50
Top 50 tokens, plenty of variety. A common safe default.

Adventure 8

The third dial
Top-p (Nucleus Sampling) — The Adaptive Cutoff
The rule
Top-p = X means: keep just enough top tokens to cover X% of the probability mass
Example: Top-p = 0.9
Keep enough tokens to account for 90% of the probability mass. Discard the long tail.
Why it's smarter than top-k
The cutoff is dynamic: easy prompt → maybe 3 tokens. Open prompt → maybe 200.

Adventure 8

📘 Concept card
Top-p (Nucleus Sampling)
What: Keep the smallest set of top tokens covering X% of the probability mass; discard the long tail.
Like: At a buffet, only consider dishes everyone is actually picking. Ignore the empty trays.
Why it matters: Cleaner than top-k. Adapts to whether the model is "sure" or "unsure" about the next word.

Adventure 8

The 3 Dials — Cheat Sheet
Dial
What it does
Good default
Temperature
Rescales distribution before sampling
0.7
Top-k
Keep only top N tokens
50
Top-p
Keep enough tokens to cover X% mass
0.9

Most APIs let you set all three. They combine. You usually only need to tune one.

Adventure 8

Knowledge check
Knowledge Check 2 — Pick the Dials
For each task, what dial settings would you pick? Think before answering.
Generating SQL queries
What temperature? What top-p? Why?
Writing a kid's bedtime story
What temperature? What top-p? Why?
Extracting JSON from messy text
What temperature? What top-p? Why?
Brainstorming startup ideas
What temperature? What top-p? Why?

Adventure 8

A question you've probably had
Why Same Prompt, Different Answer?
You've noticed: ask ChatGPT the same thing twice, get different answers.
1
What you think
"The AI is being inconsistent" or "It's broken."
2
What's actually happening
That's not a bug. That's sampling. Every call: the dice roll again on the probability distribution.

✓ Takeaway: Non-determinism is a feature, not a bug. Learn to design around it.

Adventure 8

Going deeper
Even T=0 Isn't Truly Deterministic
Two surprising reasons why "temperature zero" still isn't perfectly reproducible:
Floating-point math on GPUs
Isn't perfectly reproducible across runs. Tiny rounding differences accumulate.
Cloud providers batch your requests
Your request is processed alongside others — batch composition affects results.

Need exact same output every time? Run a local model with fixed seeds. Otherwise: design around non-determinism.

Adventure 8

Your turn
Try It Yourself
01
Open the playground
OpenAI or Anthropic playground — not the chat interface
02
Set temperature to 0
Find the temperature slider and drag it all the way down
03
Run the same prompt 5 times
Don't change anything between runs
04
Check: identical or just similar?
If they're not identical, you just observed floating-point non-determinism. Welcome to the real world.

Adventure 8

Part 2 — Talking to the model
Two Channels Every API Has
Modern LLM APIs accept two kinds of input:
  • System prompt — persistent instructions for the whole conversation
  • User prompt — what the user types each turn
The system prompt is hidden in ChatGPT. Visible in the API.

Adventure 8

See it in action
Example — The Pirate System Prompt
System: You are a pirate. Speak only in pirate English.

        Every response must include the word "matey."



User:   What is photosynthesis?



Output: Arrr, matey! Plants be doin' a magical thing called

        photosynthesis — they take sunlight, water, and a bit

        o' the breeze and turn it into food.

The system prompt persists across turns. The user prompt is just this turn's message.

Adventure 8

📘 Concept card
System Prompt
What: The hidden "rules of the game" the model gets before you say anything.
Like: A waiter's training. They greet every guest the same way because they were trained to — not because of how the guest asked.
Why it matters: If you want consistent behavior across many users or sessions, the system prompt is where that lives.

Adventure 8

What ChatGPT hides from you
ChatGPT's Hidden System Prompt
Every ChatGPT conversation starts with a system prompt YOU never see:
  • "You are ChatGPT, the current date is..."
  • Safety rules
  • Tool-use instructions
  • Format preferences
That's why ChatGPT acts the same way no matter who's using it.

When you switch to the API, you write this yourself. That's power.

Adventure 8

Part 3 — 5 patterns that always work
Pattern 1 — Role
Tell the model who it is.
❌ Vague
"Help me with my essay."
✅ Role prompt
"You are a high school English teacher with 20 years of experience. Help me revise this essay."

Why it works: Pulls the right examples from training data into the model's attention.

Adventure 8

Pattern 2
Pattern 2 — Format
Tell the model exactly what shape the output should take.
❌ Vague
"Tell me about the planets."
✅ Format prompt
"List each planet with: name, distance from sun (AU), and one weird fact. Output as a Markdown table."

Vague output = vague to use. Structured = directly usable.

Adventure 8

Pattern 3
Pattern 3 — Few-shot
Show the model 2–3 examples. Then give it a new input.
The model continues your pattern. Shockingly reliable for custom formats.

You're not telling the model the rule — you're showing it. The model infers the pattern from your examples.

Adventure 8

Pattern 4
Pattern 4 — Chain-of-Thought
For anything requiring reasoning, add: "Think step by step."
❌ Direct question
"What's 23 × 47?"
✅ Chain-of-thought
"What's 23 × 47? Think step by step before giving the final answer."

Why it works: The model can only see its own previous tokens. Force it to write reasoning → each step builds on the last → accuracy jumps.

Adventure 8

Pattern 5
Pattern 5 — Constraints
Tell the model what NOT to do, or set hard limits.
"Write a poem about autumn.
- Exactly 4 lines.
- No words longer than 6 letters.
- Don't mention 'leaves' or 'cold'.
- End with a question mark."

Constraints force the model out of its default path. Output gets sharper.

Adventure 8

The 5 Patterns — Cheat Sheet
Pattern
One-line use
Role
"You are a..."
Format
"Output as JSON..."
Few-shot
2–3 input→output examples
Chain-of-Thought
"Think step by step"
Constraints
"Exactly 4 lines, no X"

Adventure 8

5 Anti-patterns — and Fixes
Anti-pattern
Example
Fix
Vague
"Make it better."
"Improve by: (1) shorten to 80 words, (2) active voice, (3) one concrete example."
Multi-task
"Translate, summarize, fact-check."
Run 3 prompts. Each does one job.
Contradictory
"Be brief but explain in detail."
Pick one.
No context
"Why is this wrong?"
The model can't see your screen. Include "this."
Asking for fresh info
"What's today's weather?"
Use a tool (Week 4).

Adventure 8

Hands-on activity
Your Mission — The Wikipedia Haiku Challenge
Goal: Get an LLM to turn any Wikipedia article into a 3-line haiku that captures its essence.
The Rules
  • 3 lines, syllable pattern 5-7-5
  • Must mention one specific fact from the article
  • Don't name the subject directly in line 1
How to Play
  1. Pick an article
  1. Write your first prompt, run it, notice what's wrong
  1. Improve using the 5 patterns
  1. Iterate 5 times
  1. Compare with a classmate's — whose is sharper?

Adventure 8

The 3-Hour Summary
You now understand:
  • ✅ Generation = sampling from a probability distribution
  • ✅ Temperature, top-k, top-p are the three dials that shape it
  • ✅ Same prompt → different answers is normal — and you know exactly why
  • ✅ System prompt vs user prompt — and what ChatGPT hides from you
  • ✅ 5 patterns that always work, 5 anti-patterns to avoid

Adventure 8

Homework + preview
This Week's Mission + Week 4 Preview
This week
  • Finish the haiku challenge. Save your best prompt + output.
  • Bring your final prompt back next week — we'll use it.
Next week — Agents: When LLMs Can Use Tools
So far, the LLM is locked inside its own head. It can think, but it can't do. Next week: what happens when you give it hands — tools that let it read files, run code, browse the web. We'll watch Claude Code work and reverse-engineer the agent loop.

Adventure 8