Sufficient Statistics & Information

1. When can you throw data away?

You watch $X$ sitting in some space $\mathcal{X}$ and it comes from $P_\theta$ with $\theta \in \Theta$, and maybe $X$ is a million video frames and maybe $\theta$ is a single number like the gravitational constant that made the motion happen.

A statistic $T(X)$ takes $\mathcal{X}$ and sends it to $\mathcal{T}$ and squeezes it down, and the real question is whether this squeezing loses anything.

$T(X)$ is sufficient for $\theta$ when the conditional distribution of $X$ given $T(X)=t$ does not lean on $\theta$ at all.

Once you know $T(X)=t$ you can cook up fake data that looks exactly like the real thing using just $t$ and a random number generator and no knowledge of $\theta$ at all, and so $X$ tells you nothing about $\theta$ that $T(X)$ has not already told you.

2. Halmos-Savage factorization

In practice nobody ever checks sufficiency by working out conditional distributions by hand and the tool everyone reaches for is factorization.

Theorem (Halmos-Savage, 1949). Take $\{P_\theta : \theta \in \Theta\}$ and have it sit under a $\sigma$-finite measure $\mu$, and then $T(X)$ is sufficient for $\theta$ exactly when the density $f_\theta(x) = \frac{dP_\theta}{d\mu}(x)$ splits up as

where $g_\theta \ge 0$ touches $x$ only through $T(x)$ and $h(x) \ge 0$ has no $\theta$ in it.

Why it works ($\implies$ direction). When $T$ is sufficient you can write $P_\theta(A) = \int_{\mathcal{T}} P(A | T=t) \, dP_\theta^T(t)$ and the conditional piece $P(X=x|T=t)$ has no $\theta$ in it and that piece becomes $h(x)$, and the marginal of $T$ carries all the $\theta$-dependence and that becomes $g_\theta(t)$.

For the full measure-theoretic argument take $\mathcal{A} = \sigma(T)$ and build a dominating mixture $\lambda = \sum 2^{-i} P_{\theta_i}$, and then $\frac{dP_\theta}{d\lambda}$ is $\mathcal{A}$-measurable and so a function of $T$, and $\frac{d\lambda}{d\mu}$ leans on the mixture and not on any specific $\theta$. The chain rule $\frac{dP_\theta}{d\mu} = \frac{dP_\theta}{d\lambda} \cdot \frac{d\lambda}{d\mu}$ hands you the factorization.

3. Fisher Information

Sufficiency has a clean story in terms of Fisher Information.

Data processing cannot make new information and so $I_T(\theta) \le I_X(\theta)$ for any statistic $T$.

And $T$ is sufficient exactly when equality holds everywhere, and the proof is short. With factorization you get $\ell(\theta) = \log g_\theta(T) + \log h(X)$ and so the score $\nabla_\theta \ell(\theta) = \nabla_\theta \log g_\theta(T)$ leans only on $T$, and the variance of the score (which is the Fisher Information) comes out the same whether you work it out from $X$ or from $T$.

Sufficiency holds exactly when no information is lost at all.

4. Minimal sufficiency and the likelihood ratio

There are always lots of sufficient statistics and $X$ itself is sufficient in a trivial way, and what you want is the most squeezed one and the coarsest partition of the sample space that still loses nothing.

$T$ is minimal sufficient when for any other sufficient statistic $S$ the $T$ is a function of $S$.

Lehmann-Scheffe criterion: $T(x)$ is minimal sufficient iff

Two data points land on the same summary exactly when their likelihood ratio goes flat.

Uniform $U[0, \theta]$. $L(\theta; x) = \theta^{-n} \mathbb{I}(\max(x) \le \theta)$. The ratio stays constant exactly when $x_{(n)} = y_{(n)}$ and so $T(X) = \max(X)$ is minimal sufficient, and one number holds everything about $\theta$ that $n$ observations have in them.

Cauchy. $f(x; \theta) = \frac{1}{\pi(1+(x-\theta)^2)}$. The likelihood ratio is a degree-$2n$ rational function of $\theta$ and for it to stay constant the polynomials have to share all their roots, and that forces $\{x_i\} = \{y_i\}$ as sets. The minimal sufficient statistic is the full order statistics and no squeezing is possible.

Cauchy tails are so heavy that every single observation tells you something about $\theta$ on its own and no lower-dimensional summary exists.

5. Pitman-Koopman-Darmois

When can you squeeze $n$ samples into a fixed number of summaries no matter how big $n$ gets? Almost never.

Theorem. Under regularity conditions where the support does not lean on $\theta$, when a family has a sufficient statistic of dimension $k$ that does not grow with $n$ the family has to be an exponential family.

Sketch. For $n$ i.i.d. samples factorization hands you $\sum_{i=1}^n \log f(x_i | \theta) = \alpha(T(\mathbf{x}), \theta) + \beta(\mathbf{x})$, and you take derivatives in both $x_i$ and $x_j$ to get cross-derivative constraints. For the Jacobian that maps data to parameter gradients to have rank at most $k$ as $n$ grows you need $\nabla_\theta \log f(x | \theta) = \sum_{j=1}^k w_j'(\theta) t_j(x)$, and you integrate and out pops the exponential family form.

Fixed-dimensional sufficiency is so tight that it basically picks out exponential families and nothing else.

6. Exponential families and convex duality

Write the canonical form:

Three facts make exponential families so easy to work with.

$A(\eta)$ is convex. Because $A(\eta) = \log \int h(x) e^{\eta T(x)} dx$ and this is a log-moment-generating function, and Holder’s inequality does the rest.

Moments from derivatives.

Covariance is PSD and so $A$ is strictly convex as long as you have a minimal representation.

MLE is moment matching. The log-likelihood for data $D$ is $\mathcal{L}(\eta) = n \eta^T \bar{T} - n A(\eta) + \text{const}$, and you set the gradient to zero.

The MLE is the parameter that makes the model’s expected sufficient statistics line up with the empirical ones, and information geometry reads this as projecting the empirical distribution onto the model manifold.

7. Basu’s Theorem

An ancillary statistic $A(X)$ has a distribution that does not lean on $\theta$ at all. Take $X_1 - X_2$ when $X_i \sim \mathcal{N}(\theta, 1)$ and you get one.

Ancillaries hold no likelihood information about $\theta$ but they do tell you how precise your experiment was.

Basu’s Theorem. When $T$ is boundedly complete and sufficient and $A$ is ancillary then $T \perp A$.

Proof. Take $B \in \sigma(A)$ and because $A$ is ancillary $P_\theta(B)$ stays constant in $\theta$. Let $g(t) = P(B | T=t)$ and sufficiency strips the $\theta$ out. Then $\mathbb{E}_\theta[g(T)] = P_\theta(B)$ is constant and so $\mathbb{E}_\theta[g(T) - P(B)] = 0$ for all $\theta$. Completeness of $T$ forces $g(T) = P(B)$ almost surely. And so $P(B|T) = P(B)$ and that is independence. $\square$

Why this matters for the $t$-test. For $X_i \sim \mathcal{N}(\mu, \sigma^2)$ the $\bar{X}$ is complete sufficient for $\mu$ given the joint structure and $S^2$ is ancillary for $\mu$, and Basu hands you $\bar{X} \perp S^2$. The independence of $\bar{X}$ and $S^2$ is why the $t$-test works because the top and the bottom are independent. And it only works for Gaussians by Geary’s Theorem, and you try it with any other distribution and the independence falls apart.

8. Rao-Blackwellization

You have a bad unbiased estimator $\hat{\theta}$ and a sufficient statistic $T$ and you set $\tilde{\theta} = \mathbb{E}[\hat{\theta} | T]$.

It is still unbiased because $\mathbb{E}[\tilde{\theta}] = \mathbb{E}[\mathbb{E}[\hat{\theta}|T]] = \theta$.

And it is computable because sufficiency means the conditional distribution has no $\theta$ in it.

And by the law of total variance you get

Conditioning on the sufficient statistic always cuts variance down and it cuts it strictly unless the estimator was already a function of $T$.

9. Information Bottleneck and neural networks

Tishby’s Information Bottleneck asks you to find a $T$ that pushes $I(T; Y)$ up and pulls $I(T; X)$ down, and the ideal $T$ is a sufficient statistic for $Y$ and minimal sufficient means perfect compression.

People have argued that SGD quietly finds these representations. The trouble is that for deterministic networks with continuous inputs the $I(T; X)$ is infinite or undefined and you need noise in the weights or activations to make any of this well-defined.

For Bayesian neural networks the posterior predictive leans on sufficient statistics of the training data. But neural networks are not exponential families because they have finite width, and so the sufficient statistic is the whole dataset and no compression actually happens.

In the NTK regime when the width goes to infinity the sufficient statistics collapse down to the kernel matrix and that gives you a real finite-dimensional summary.

10. Sufficient statistics in time series

For i.i.d. data sufficiency squeezes $N$ static points down. In time series you are chewing through a stream $x_1, x_2, \ldots$ and you want a running summary $h_t = T(x_{1:t})$ with

This $h_t$ is the state, and when a finite-dimensional $h_t$ exists you have a hidden Markov model or a state space model.

Kalman filter. For linear Gaussian systems the sufficient statistics for the future are $(\mu_t, \Sigma_t)$ and the Kalman filter is just their recursive update.

Fixed-memory tracking of a dynamic system. This is Pitman-Koopman-Darmois pushed onto conditional transition densities, and if the noise were Cauchy and not Gaussian you would have to hold onto the whole history.

11. Approximate sufficiency and Le Cam’s deficiency

What happens when a statistic is only almost sufficient?

Le Cam pinned this down with decision theory. Two experiments $\mathcal{E}$ where you watch $X$ and $\mathcal{F}$ where you watch $T(X)$ are equivalent when every decision rule in one can be matched in the other with the same risk.

The deficiency distance is

where $K$ is a Markov kernel, and $\delta = 0$ means $T$ is sufficient and $\delta < \epsilon$ means $T$ is $\epsilon$-sufficient.

And this hooks up to differential privacy. You want statistics that are sufficient for the signal and insufficient for picking out the user.

12. Fisher’s program

Fisher’s 1922 insight was that inference is data reduction. You start with a high-dimensional $X$ and you boil it down to a tiny $\hat{\theta}$, and sufficient statistics tell you exactly what you can throw away and factorization gives you the tool to check, and exponential families are the spot where finite-dimensional summaries actually exist, and Basu and Lehmann-Scheffe fill in the links between sufficiency and ancillarity and invariance.

In modern ML the word “sufficiency” goes by “representation learning” and the goal is the same. Find the smallest coordinates of whatever manifold the data actually lives on.

Simulation of Rao-Blackwellization and Fisher Information

We work out $\lambda$ for a Poisson with three estimators.

• $X_1$ alone, unbiased and wasteful.
• $\bar{X}$, which is the Rao-Blackwell improvement and the MVUE.
• $\hat{\lambda}$ from $T(X) = \mathbb{I}(X > 0)$, which is binary compression and throws the counts away.

The third estimator throws the count information away and should have much higher variance.

import numpy as np
import jax.numpy as jnp
from jax import grad, vmap
import matplotlib.pyplot as plt
 
def fisher_info_loss_demo(true_lambda=5.0, n_samples=10, n_trials=5000):
    estimates_raw = []
    estimates_rb = []
    estimates_bad = []
    
    for _ in range(n_trials):
        X = np.random.poisson(true_lambda, n_samples)
        
        # 1. Raw Estimator: Only first sample
        est_raw = X[0]
        estimates_raw.append(est_raw)
        
        # 2. Rao-Blackwellized (Mean)
        est_rb = np.mean(X)
        estimates_rb.append(est_rb)
        
        # 3. Bad Statistic (Lossy Compression)
        # We only know how many are non-zero.
        # k = count(X > 0). k ~ Binomial(n, 1 - e^-lambda)
        # p = 1 - e^-lambda => lambda = -log(1-p)
        # MLE for p is k/n.
        # est_lambda = -log(1 - k/n)
        k = np.sum(X > 0)
        # Avoid div by zero
        if k == n_samples:
            est_bad = -np.log(1 - (n_samples-0.5)/n_samples) # Smooth
        else:
            est_bad = -np.log(1 - k/n_samples)
        estimates_bad.append(est_bad)
 
    print(f"Theory Variance (Raw): {true_lambda:.4f}")
    print(f"Empirical Var (Raw): {np.var(estimates_raw):.4f}")
    
    # Cramer-Rao Lower Bound = lambda / n
    crlb = true_lambda / n_samples
    print(f"CRLB (Optimal): {crlb:.4f}")
    print(f"Empirical Var (RB): {np.var(estimates_rb):.4f}")
    
    print(f"Empirical Var (Bad Stat): {np.var(estimates_bad):.4f}")
    # The 'Bad Stat' throws away info (exact counts), keeping only binary info.
    # Variance explodes.

Completeness and Basu’s Theorem

Completeness of exponential families

The family $f(x|\theta) = h(x) \exp(\theta \cdot T(x) - A(\theta))$ is complete when $\Theta$ holds an open rectangle.

Suppose $\mathbb{E}_\theta[g(T)] = 0$ for all $\theta$, and then $\int g(t) e^{\theta t - A(\theta)} dt = 0$ and that hands you $\int g(t) e^{\theta t} dt = 0$. This is the Laplace transform of $g$ and when a Laplace transform vanishes on an open set the function is zero almost everywhere, and the result follows.

When completeness fails

Take $U[\theta, \theta+1]$ and pick $g(X) = \sin(2\pi X)$. Then $\mathbb{E}_\theta[\sin(2\pi X)] = \int_\theta^{\theta+1} \sin(2\pi x) \, dx = 0$ for every $\theta$ because it is a full-period integral. But $\sin(2\pi X) \ne 0$ and so the uniform family is incomplete and Basu’s theorem does not apply, and ancillaries can depend on sufficient statistics here.

Moments from the partition function

Every moment calculation you would ever want falls out by taking derivatives of $A(\eta)$.

Bernoulli. $P(x) = \mu^x (1-\mu)^{1-x} = \exp( x \log \frac{\mu}{1-\mu} + \log(1-\mu) )$. The canonical parameter is $\eta = \log \frac{\mu}{1-\mu}$ which is the logit. The inverse is $\mu = \sigma(\eta) = \frac{1}{1+e^{-\eta}}$. The log-partition is $A(\eta) = \log(1+e^\eta)$. Mean. $A'(\eta) = \sigma(\eta) = \mu$ Variance. $A''(\eta) = \mu(1-\mu)$

Poisson. $P(x) = \frac{\lambda^x e^{-\lambda}}{x!} = \frac{1}{x!} \exp( x \log \lambda - \lambda )$. The canonical parameter is $\eta = \log \lambda$ and the log-partition is $A(\eta) = e^\eta$. Mean. $A'(\eta) = e^\eta = \lambda$ Variance. $A''(\eta) = e^\eta = \lambda$

Gaussian (known $\sigma^2=1$). $P(x) \propto \exp( x\mu - \mu^2/2 )$. The canonical parameter is $\eta = \mu$ and the log-partition is $A(\eta) = \eta^2/2$. Mean. $A'(\eta) = \eta = \mu$ Variance. $A''(\eta) = 1$

Quantum sufficiency

Sufficiency carries over to quantum mechanics and the objects just change shape. A density matrix $\rho_\theta$ on Hilbert space $\mathcal{H}$ takes the place of the distribution and a quantum channel (CPTP map) $\mathcal{E}$ takes the place of the statistic.

And $\mathcal{E}$ is sufficient when there is a recovery channel $\mathcal{R}$ with

This hooks into the Petz recovery map. A measurement (POVM) is sufficient for a family of states when the quantum Fisher information stays put.

The monotonicity of relative entropy gives the criterion

and equality holds exactly when $\mathcal{E}$ is sufficient for $\{\rho_\theta, \rho_{\theta'}\}$. The data processing inequality stops any channel from making states easier to tell apart, and this is the same principle as in the classical case.

Sufficiency in latent variable models (EM)

Take a model with observed $X$ and latent $Z$. The complete likelihood $P(X, Z | \theta)$ often sits in an exponential family with sufficient statistics $T(X, Z)$. But we only get to see $X$.

EM works by computing expected sufficient statistics.

E-step: compute $P(Z | X, \theta_{\text{old}})$ and then

M-step: find $\theta_{\text{new}}$ by moment matching:

For Gaussian mixtures the sufficient statistics are $\sum \gamma_{ik}$ for the cluster masses and $\sum \gamma_{ik} x_i$ for the first moments and $\sum \gamma_{ik} x_i x_i^T$ for the second moments. Without factorization EM would be unworkable for most models.

Milestones

• 1922. R.A. Fisher lays down “sufficiency” in his foundational paper on mathematical statistics and sets up the idea that a statistic can hold all the information about a parameter.
• 1935. Koopman and Pitman and Darmois each prove on their own that fixed-dimensional sufficiency forces exponential family structure.
• 1945. C.R. Rao proves the Rao-Blackwell theorem and shows that conditioning on a sufficient statistic always cuts variance down.
• 1949. Halmos and Savage prove the factorization theorem with measure theory and put sufficiency on solid footing.
• 1955. D. Basu proves that complete sufficient statistics come out independent of ancillary statistics.
• 1972. Le Cam brings in deficiency distance and lets you compare approximate sufficiency between experiments.
• 1999. Tishby puts forward the Information Bottleneck principle and hooks sufficiency up to representation learning and deep networks.

Terminology used in this post

A sufficient statistic holds everything $X$ knows about $\theta$ and formally the conditional distribution of $X$ given $T(X)$ does not lean on $\theta$ at all. A minimal sufficient statistic is the coarsest such statistic and it squeezes the data as far as it can go with no information loss. Completeness means no non-trivial function of $T$ has expectation zero for all parameter values and it is the condition that makes Basu’s theorem work. The exponential family is the class of distributions whose log-likelihood is linear in the sufficient statistics and the Pitman-Koopman-Darmois theorem says these are basically the only families that allow fixed-dimensional sufficiency. Fisher Information measures how much an observable tells you about $\theta$ and Rao-Blackwellization is the move of conditioning an estimator on a sufficient statistic to cut variance down. An ancillary statistic has a distribution that does not lean on $\theta$ and interaction information can be positive or negative and it measures synergy or redundancy between variables.

References

1. Fisher, R. A. (1922). “On the mathematical foundations of theoretical statistics”.

2. Halmos, P. R., & Savage, L. J. (1949). “Application of the Radon-Nikodym theorem to the theory of sufficient statistics”.

3. Basu, D. (1955). “On statistics independent of a complete sufficient statistic”.

4. Lehmann, E. L., & Scheffe, H. (1950). “Completeness, similar regions, and unbiased estimation”.

5. Pitman, E. J. G. (1936). “Sufficient statistics and intrinsic accuracy”.

6. Csiszar, I. (1975). “I-divergence geometry of probability distributions and minimization problems”.