Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

A language model does not actually "write" the next token.

It produces a probability distribution over the vocabulary and says, roughly:

Here are 50,000 things I could say next. These are how plausible I think they are.

Everything that happens after that is decoding.

And this is where things get interesting: the same model, with the same prompt, can produce radically different text depending on how you sample from that distribution.

In 2019, researchers studying GPT-2 generation found that decoding strategy alone could dramatically change the character of generated text, even when the underlying neural network was identical. Greedy and beam-style decoding tended toward bland repetition; unrestricted sampling could wander into low-probability nonsense; carefully truncated sampling produced text much closer to human writing.

This is why parameters such as temperature, top_k, top_p, min_p, and repetition_penalty are not merely API knobs.

They are different ways of answering one question:

Given the model's probability distribution, which parts of that distribution are we willing to trust?

This article builds from that intuition to the actual equations, implementation details, and operational consequences.

1. First: the model gives you a probability distribution, not an answer

Suppose the prompt is:

The cat sat on the

The model might assign probabilities approximately like:

mat       0.42
floor     0.20
chair     0.12
bed       0.08
table     0.05
roof      0.01
...

The model does not intrinsically say:

mat = correct
floor = incorrect
chair = incorrect

It says:

mat       is very plausible
floor     is also plausible
chair     is plausible
...

A decoder turns that distribution into one actual token.

There are two basic philosophies.

Deterministic decoding

Take the highest-probability token:

argmax_i P(token_i | context)

This is greedy decoding.

Stochastic decoding

Sample from the distribution:

token ~ P(token | context)

Now the model can occasionally choose "floor" instead of "mat".

That small distinction compounds over a sequence.

Suppose the model has two reasonable choices at every step, with probabilities:

0.8 / 0.2

Over 20 independent-ish decisions, the probability of taking the 0.8 choice every time is approximately:

0.8^20 ~= 0.0115

So even modest randomness creates a large space of possible generations.

That is why sampling is useful for creative writing, brainstorming, synthetic data, dialogue, and many other tasks.

It is also why sampling can produce garbage.

The core problem is that language-model distributions have a long tail: a few tokens may be very plausible, followed by thousands of increasingly questionable ones.

The history of modern sampling tricks is largely the history of figuring out where that tail should be cut.

2. Temperature: change how sharp the distribution is

Temperature is the simplest way to alter the distribution.

Start with logits:

z_i

Before softmax, temperature modifies them as:

z_i' = z_i / T

Then:

P_i = exp(z_i') / sum_j exp(z_j')

The intuition is more important than the equation:

T < 1  -> sharpen distribution
T = 1  -> leave distribution alone
T > 1  -> flatten distribution

Imagine:

Token A   0.70
Token B   0.20
Token C   0.07
Token D   0.03

Lower the temperature and the model becomes more committed to A.

Raise it and probability shifts toward B, C, and D.

A useful way to understand temperature mathematically is through odds ratios.

Suppose:

P(A) / P(B) = 4

After temperature scaling, the ratio becomes approximately:

(P(A) / P(B))^(1/T)

At T = 2:

4^(1/2) = 2

So a 4:1 preference becomes 2:1.

At T = 0.5:

4^(1/0.5) = 16

The same underlying preference becomes 16:1.

That is what temperature really does:

It changes how strongly the decoder believes the model's ranking.

This also explains why temperature does not solve the long-tail problem.

Suppose the distribution is:

A  0.40
B  0.25
C  0.15
D  0.08
E  0.05
F  0.03
G  0.02
...

Turning the temperature up does not distinguish between "reasonable alternatives" and "nonsense in the tail."

It simply gives more probability to everything.

Turning it down does the opposite: it pushes the model toward its favorite candidates.

This is exactly the tradeoff that became visible in early large-scale text generation experiments. The GPT-2-era generation recipes commonly used combinations such as temperature 0.7 and top-k=40; Holtzman and colleagues later showed that temperature alone could not adequately control the quality/diversity tradeoff.

So temperature answers:

How adventurous should I be?

But it does not answer:

Which candidates should I consider at all?

That is where top-k and top-p enter.

3. Top-k: keep exactly K candidates

Top-k is almost embarrassingly simple.

Sort the candidate tokens by probability and keep only the top K.

For example:

A  0.40
B  0.25
C  0.15
D  0.08
E  0.05
F  0.03
G  0.02

With:

top_k = 3

we keep:

A  0.40
B  0.25
C  0.15

Then renormalize:

A  0.40 / 0.80 = 0.50
B  0.25 / 0.80 = 0.3125
C  0.15 / 0.80 = 0.1875

Then sample from those three.

Top-k became a practical generation technique in work on hierarchical neural story generation by Angela Fan, Mike Lewis, and Yann Dauphin in 2018. The paper was working on a very concrete problem: getting neural models to produce long, coherent stories rather than collapsing into low-quality text.

The key limitation of top-k is visible immediately.

Consider two contexts.

Context A: very confident model

A  0.95
B  0.02
C  0.01
D  0.005
E  0.005
...

Context B: uncertain model

A  0.20
B  0.18
C  0.16
D  0.14
E  0.12
F  0.10
G  0.10
...

With top_k=5, you keep five tokens in both cases.

But that means very different things.

In Context A, five tokens may include several absurd tail candidates.

In Context B, five tokens may throw away perfectly reasonable possibilities.

So the problem with top-k is:

K is constant, while the model's uncertainty is not.

Top-k is therefore a blunt instrument.

A useful engineering intuition:

top-k asks:
"How many candidates may I consider?"

It does not ask:

"How much of the probability mass should I trust?"

That second question leads directly to nucleus sampling.

4. Top-p: keep enough probability mass

Top-p, or nucleus sampling, was introduced by Ari Holtzman and colleagues in their 2019 work on neural text degeneration.

Their observation was important enough to change how people thought about generation:

maximum likelihood is a good training objective, but simply choosing the most likely continuation is not necessarily a good generation objective.

Models were producing text that could have excellent token-level likelihood while becoming bland, repetitive, or trapped in loops.

Top-p changes the truncation rule.

Instead of:

keep K tokens

we say:

keep the smallest set of tokens whose cumulative
probability reaches p

Suppose:

A  0.45
B  0.25
C  0.15
D  0.08
E  0.04
F  0.02
G  0.01

With:

top_p = 0.80

we accumulate:

A       0.45
A+B     0.70
A+B+C   0.85

So the nucleus is:

{A, B, C}

The remaining tokens disappear.

Notice the important difference from top-k.

If the distribution becomes very concentrated:

A  0.95
B  0.02
C  0.01
...

then top_p=0.95 may retain only a handful of candidates.

If the model becomes uncertain:

A  0.15
B  0.14
C  0.13
D  0.12
E  0.11
...

the nucleus can expand substantially.

That is the central idea:

Top-p adapts the candidate set to the model's uncertainty.

Holtzman et al. found that nucleus sampling could produce distributions much closer to human text than several conventional decoding methods, while avoiding both the blandness of highly deterministic decoding and the incoherence of unconstrained sampling.

This is why a setting like:

temperature = 0.8
top_p = 0.95

became a common practical combination.

Temperature shapes the distribution.

Top-p decides how far into the tail you are willing to go.

That distinction is fundamental.

5. Min-p: scale the cutoff to the best token

Min-p is a newer idea, proposed by Minh Nguyen and colleagues and published at ICLR 2025.

It starts with a criticism of top-p:

Top-p looks at cumulative probability mass, but sometimes what matters more is the relative quality of each candidate compared with the model's best candidate.

Suppose the best token has probability:

P_max

Min-p keeps tokens satisfying approximately:

P(token) >= min_p * P_max

So if:

P_max = 0.60
min_p = 0.1

the threshold becomes:

0.1 * 0.60 = 0.06

Any token below 0.06 is discarded.

Now consider a much less confident model:

P_max = 0.12

The threshold becomes:

0.1 * 0.12 = 0.012

The candidate set expands.

This gives min-p a nice conceptual interpretation:

Keep tokens that are not too far below the model's current favorite.

Compare the three strategies:

top-k:
keep exactly K tokens

top-p:
keep enough tokens to cover probability mass p

min-p:
keep tokens above a fraction of the best token's probability

They are solving similar problems from different angles.

A useful concrete example:

Token   Probability
A       0.50
B       0.20
C       0.10
D       0.08
E       0.04
F       0.03
G       0.02
H       0.01

With:

top_k = 4

you get:

A B C D

With:

top_p = 0.80

you also get:

A+B+C = 0.80

so roughly:

A B C

With:

min_p = 0.10

the threshold is:

0.10 * 0.50 = 0.05

so:

A B C D

survive.

The important practical point is that min-p is not simply "top-p but newer."

It encodes a different assumption about what makes a token trustworthy.

There is also a useful lesson here for engineers: newer sampling methods are not automatically better sampling methods. The original min-p paper reported improvements in quality/diversity tradeoffs, particularly at higher temperatures, but a later 2025 critical re-analysis challenged several of those conclusions. That makes min-p interesting precisely because it is an active research area rather than a solved problem.

6. Repetition penalty: change the score of tokens you've already used

Sampling filters are mostly concerned with the current distribution.

Repetition penalty introduces history.

Suppose the model's current logits are:

" the"     8.0
" a"       7.5
" cat"     6.8
" dog"     5.9

and " the" has already appeared many times.

A repetition penalty says, approximately:

If a token has already appeared, make it less attractive.

The classic approach associated with CTRL applies a multiplicative transformation to previously seen tokens. In the common sign-aware implementation:

if z > 0:
    z' = z / penalty

if z <= 0:
    z' = z * penalty

with:

penalty >= 1

So with:

penalty = 1.2

a positive logit of:

6.0

becomes:

6.0 / 1.2 = 5.0

while:

-6.0

becomes:

-6.0 * 1.2 = -7.2

The sign handling matters.

Naively dividing every logit by 1.2 would turn:

-6.0 -> -5.0

which actually makes that token more likely, not less likely.

The CTRL generation code exposed a repetition penalty of 1.2, and the technique subsequently became common in open-source generation stacks.

But there is a deeper operational issue.

A repetition penalty is token-blind.

Suppose the model is generating:

{
  "name": "alice",
  "age": 37,
  "city": "London"
}

Repeating "name" later may be perfectly correct.

So might repeating:

class
return
self

in Python.

A generic repetition penalty does not understand that.

This is why aggressive anti-repetition settings can damage code, structured output, terminology, mathematical notation, and other domains where repetition is often intentional.

The practical rule is:

Use repetition penalties to combat pathological loops, not to enforce "variety" as a universal objective.

7. The real sampler is a pipeline, not one knob

The most useful mental model is to stop thinking of these as independent switches.

A typical generation step looks conceptually like:

model
  |
  v
logits
  |
  +--> repetition penalty / other logit adjustments
  |
  +--> temperature scaling
  |
  +--> top-k / top-p / min-p filtering
  |
  v
renormalize
  |
  v
sample next token
  |
  v
append token
  |
  +--------------------> model again

That ordering matters.

Consider:

temperature = 1.5
top_p = 0.95

Temperature first broadens the distribution.

Top-p then decides how much of that broadened distribution survives.

A small implementation detail can therefore produce noticeably different results depending on the order of operations.

This becomes particularly important when you combine multiple truncation rules.

Suppose:

top_k = 50
top_p = 0.9

You are effectively saying:

don't consider more than 50 candidates
AND
don't consider candidates outside the 90% probability nucleus

The resulting candidate set is usually the intersection induced by the implementation order.

A useful tuning strategy

Rather than randomly searching a huge hyperparameter grid, think in layers.

Start with:

temperature = 1.0

Then decide whether you want truncation:

top_p

or:

top_k

or perhaps:

min_p

Then introduce repetition control only if you actually observe repetition problems.

For example:

creative writing:
T ~= 0.8-1.1
top_p ~= 0.9-0.98
repetition penalty ~= 1.0-1.1

For deterministic-ish extraction:

T ~= 0-0.3

and relatively tight truncation, or no stochastic sampling at all.

For code:

T ~= 0-0.3
repetition penalty ~= 1.0

is often a more sensible starting point than aggressively punishing reused tokens.

These are starting points, not laws. The model itself matters enormously.

The economics of bad sampling

Sampling parameters also have an inference-cost dimension.

Imagine an application generating:

1 million requests/month

with:

500 output tokens/request

That's:

500 million output tokens/month

Now suppose poor decoding causes only:

2% of requests

to be regenerated because the output got stuck in a loop, became incoherent, or violated a downstream check.

That's:

10 million extra output tokens

per month.

A sampling parameter that looks like a purely qualitative UX decision can therefore become a directly measurable inference-cost decision.

And the impact can be larger when bad generations trigger tool calls, retries, human review, or multi-step agent loops.

The operational objective is consequently not:

maximize creativity

or:

minimize repetition

It is closer to:

maximize task success
subject to cost, latency, and failure-rate constraints

That is a much more useful way to think about decoding in production.

Conclusion: Sampling is really uncertainty management

The five parameters can now be reduced to five questions:

temperature
"How sharply should I trust the model's preferences?"

top-k
"How many candidates am I willing to consider?"

top-p
"How much probability mass am I willing to consider?"

min-p
"How far below the best candidate am I willing to go?"

repetition penalty
"How much should I distrust tokens I've already used?"

And there is a broader lesson hiding underneath all of this.

The language model is not producing certainty.

It is producing a ranked, probabilistic landscape.

Sampling is the policy you impose on that landscape.

In 2018, top-k helped make neural story generation more usable. In 2019, Holtzman and colleagues reframed the problem around neural text degeneration and introduced nucleus sampling. CTRL added repetition penalties to fight one particularly ugly failure mode. Years later, min-p revisited the same fundamental question from another angle.

The interesting part is that none of these methods changes the underlying model.

They change what you are willing to believe about its next-token distribution.

And that is why decoding is much closer to decision policy than to a cosmetic generation setting.

For developers, that is the useful mental model:

The model gives you probabilities. Your sampler decides what those probabilities are allowed to become.

What sampling configuration have you found works best in production—and what failure mode forced you to change it?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

     

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.

blast-radius-demo.mp4

LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box	Visualize blast radius at a glance	Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)

Here's the goal:

• A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
• A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase: