Approximate Inference: Sampling the Posterior vs Optimizing an Approximation

In a simple probabilistic model, posterior inference sounds straightforward: observe some data, compute the posterior distribution over hidden variables, and use it to answer the question you care about.

In complex models, that computation is often the hard part. The latent space may be high-dimensional, the posterior may have a complicated shape, and expectations under that posterior may be difficult to compute analytically.

The practical question becomes:

If the exact posterior is difficult to compute, how can we turn inference into something tractable?

Approximate Inference gives us two major answers:

• MCMC: approach the posterior through sampling.
• Variational Inference: approximate the posterior with a distribution found through optimization.

Both address the same posterior-inference problem, but they make it computationally manageable in different ways.

The core mental model

Suppose $x$ is observed data and $z$ represents hidden variables. Posterior inference is concerned with $p(z\mid x)$ and, when needed, expectations under that distribution.

The problem is not simply defining the posterior. It is computing with it.

When multiple hidden variables interact, their joint states determine the shape of the posterior. As those interactions become more complex, directly representing the exact posterior or calculating expectations under it becomes increasingly difficult.

Approximate inference changes the computational problem instead of insisting on an exact closed-form calculation.

Exact posterior is difficult to compute
                |
        Approximate Inference
          /             \
         /               \
      MCMC                 VI
       |                   |
   Sampling           Optimization
       |                   |
Markov Chain       Variational Family
       |                   |
Posterior-like      Best approximate
samples             distribution

The key distinction is simple: MCMC approaches the posterior through samples, while VI searches for a tractable distribution that can stand in for it.

MCMC: represent the posterior with samples

Markov Chain Monte Carlo starts from the fact that directly obtaining the samples we want from a complicated posterior may be difficult.

Instead of calculating the posterior itself in closed form, MCMC constructs a stochastic process that approaches that posterior over time.

The goal is to build an ergodic Markov Chain over the latent-variable space whose Stationary Distribution is the target posterior.

The computational flow is:

Construct an ergodic Markov Chain
whose stationary distribution
is the target posterior
                |
                v
Run the chain
                |
                v
Collect states as samples
                |
                v
Estimate posterior quantities empirically

Ergodicity matters because the chain should not remain trapped indefinitely in a single starting state. As it progresses, the chain must be able to explore the state space needed to approach its target Stationary Distribution.

Once the chain has approached that distribution, collected states can be used to approximate properties of the posterior. Metropolis-Hastings and Gibbs Sampling are representative examples of this MCMC approach.

The useful implementation-level mental model is that the samples become the practical representation of the posterior. You do not need a convenient closed-form expression for the entire distribution if you can obtain samples that reflect it well enough to estimate the quantities you need.

That flexibility comes with computational cost. MCMC may require many repeated transitions, and it can be difficult to determine whether the collected samples are sufficiently independent. For complex models or large problems, repeated sampling can therefore become expensive.

This also shows where approximation enters MCMC. The method does not approximate the target posterior by choosing a simpler distribution family. With sufficient computation, it can approach the exact result.

In practice, however, processor time is finite, so only a limited number of samples can be collected.

For MCMC, finite computation is the practical source of approximation.

Why high-dimensional sampling is difficult

The role of the Markov Chain becomes easier to understand when we look at sampling in high-dimensional spaces.

Not every part of a large state space matters equally. The full space may be enormous, while useful samples can be concentrated in regions to which the target distribution assigns high probability.

Randomly choosing points from the entire space is therefore an inefficient way to find the regions that matter.

MCMC approaches this problem sequentially. Starting from some state, the chain repeatedly moves to another state. It first needs to reach a High-Probability Region and can then continue exploring regions with substantial probability under the target distribution.

The mental model is not:

Search the entire high-dimensional space at once

It is closer to:

current state
     |
     v
next state
     |
     v
next state
     |
     v
high-probability region
     |
     v
continue exploring and sampling

Random-walk behavior provides useful intuition here. The point is not that MCMC learns a separate manifold.

The important idea is that finding probability-concentrated regions is itself difficult in high dimensions, and a Markov Chain provides a way to move toward and explore those regions.

That is why chain construction and sample collection are central to the method.

Variational Inference: turn inference into optimization

Variational Inference takes the same posterior-inference problem and makes it tractable in a different way.

Instead of repeatedly collecting samples that reflect the exact posterior, VI approximates the difficult posterior with a probability distribution that is easier to work with.

This turns inference into an optimization problem.

The first step is to define a Variational Family $\mathcal{Q}$ , a set of candidate distributions that are allowed to approximate the exact posterior.

Each $q(z)\in \mathcal{Q}$ is a candidate approximation to $p(z\mid x)$ . To keep the approximation tractable, the family may use a Factorized Approximation or a Parametric Form such as a Gaussian.

The next step is to select the candidate that is closest to the exact posterior.

VI expresses this as:

Here, $\mathcal{Q}$ is the Variational Family, $q(z)$ is a candidate approximate distribution, $p(z\mid x)$ is the Exact Posterior, and q∗(z) is the optimal approximation within $\mathcal{Q}$ according to KL Divergence.

The KL Divergence being minimized is:

The two equations play different roles.

The first defines the optimization problem: choose the candidate distribution in $\mathcal{Q}$ that minimizes the divergence from the exact posterior.

The second defines the quantity being minimized: the KL Divergence expressed as an expectation under $q(z)$ .

The overall calculation flow is:

1. Start with the difficult Exact Posterior $p(z\mid x)$ .
2. Choose a Variational Family $\mathcal{Q}$ .
3. Consider candidate distributions $q(z)\in \mathcal{Q}$ .
4. Measure their KL Divergence from $p(z\mid x)$ .
5. Optimize within $\mathcal{Q}$ .
6. Obtain the optimal approximation q∗(z) .
7. Use q∗(z) in place of the Exact Posterior for subsequent inference calculations.

This is the core idea behind Inference as Optimization.

MCMC explores the posterior through a sequence of samples. VI searches through a restricted space of probability distributions.

Where VI's approximation comes from

Restricting the search space is what makes VI easier to compute, but that restriction also creates its main limitation.

Suppose the true posterior cannot be represented well by the chosen Variational Family $\mathcal{Q}$ . Optimization can still find the best possible q∗(z) within that family, but even a perfectly solved optimization problem cannot produce a distribution outside the family that was allowed from the start.

This is where VI differs fundamentally from MCMC.

For VI, the approximation is not simply:

We stopped computing too early.

It is:

We deliberately restricted
what the approximate posterior
is allowed to look like.

Factorized or parametric assumptions can make inference tractable, but they also limit which posterior shapes can be represented.

For VI, the restricted analytical approximation is itself the source of approximation.

MCMC vs VI: the distinction that matters

Both methods address the same problem: posterior inference when direct exact computation is difficult.

Their computational structures, however, are different.

When reading a probabilistic model or inference implementation, the most useful question is:

Where has the original posterior-inference problem been moved?

With MCMC, it has been moved into the behavior of a Markov Chain. You construct the chain so that its Stationary Distribution is the target posterior, run it, collect samples, and estimate posterior quantities empirically.

With VI, it has been moved into an optimization problem. You define $\mathcal{Q}$ , search within that family, and use q∗(z) as the tractable replacement for $p(z\mid x)$ .

That structural difference is the most important mental model to carry into equations, models, and implementations.

Takeaway

Approximate Inference is a way to make difficult Posterior Inference computationally manageable.

MCMC says:

Do not compute the entire posterior directly. Construct a Markov Chain whose Stationary Distribution is the posterior, then use samples from that chain.

Variational Inference says:

Do not compute the difficult posterior directly. Search for a tractable distribution that approximates it.

The crucial difference is where approximation enters the computation. In MCMC, practical approximation comes from finite sampling and finite computation. In VI, approximation comes from restricting the posterior to a chosen Variational Family.

Both are computational answers to the same question: what should we do when the posterior we want is too difficult to compute exactly?

Originally published at zeromathai.com.

Original article: https://zeromathai.com/en/approximate-inference-course-en/