Skip to content
NeilGirdharPublic

About

Exponential families for JAX

Resources

Contributing

Stars

77 stars

Watchers

4 watching

Forks

Repository files navigation

PyPI - Version EffVer Versioning SPEC-0 Ruff ty PyPI - Python Version

EFAX: Exponential Families in JAX

EFAX gives every exponential family distribution its two natural coordinate systems, the natural and expectation parametrizations, behind one interface that works with JAX's jit, grad, and vmap.

The two parametrizations matter because the most common operations in probabilistic machine learning are simple in one of them. Independent evidence adds in natural parameters, maximum likelihood estimates are averages in expectation parameters, and cross entropy is cheapest between an expectation parametrization and a natural one. EFAX converts between them, so each operation stays a line of code:

from operator import add

import jax
import jax.numpy as jnp
import jax.random as jr

from efax import (
    BernoulliEP,
    BernoulliNP,
    DirichletEP,
    DirichletNP,
    Estimator,
    NormalVP,
    parameter_map,
)

# Bayesian updating: add the prior's and the likelihood's natural parameters.
prior = NormalVP(mean=jnp.asarray(0.0), variance=jnp.asarray(10.0))
likelihood = NormalVP(mean=jnp.asarray(1.1), variance=jnp.asarray(3.0))
posterior = parameter_map(add, prior.to_nat(), likelihood.to_nat())
posterior.to_variance_parametrization()  # mean 0.846, variance 2.308

# Maximum likelihood estimation: average the sufficient statistics.
samples = DirichletNP(alpha_minus_one=jnp.asarray([1.0, 2.0, 3.0])).sample(jr.key(0), (10_000,))
Estimator.from_type(DirichletEP).mle(samples, axis=0).to_nat()  # alpha - 1 ≈ 1.02, 2.05, 2.99

# Cross entropy, differentiated with respect to the prediction's log-odds.
target = BernoulliEP(jnp.asarray([0.3, 0.4, 0.7]))
prediction = BernoulliNP(jnp.zeros(3))
jax.grad(lambda q: jnp.sum(target.cross_entropy(q)))(prediction)  # BernoulliNP: 0.2, 0.1, -0.2

Installation

pip install efax

Features

  • More than 30 distributions, from the normal, gamma, and Dirichlet distributions to the von Mises-Fisher and Wishart distributions.
  • Vectorized objects: one object of shape (1000,) holds a thousand distributions.
  • Pytrees throughout, so distributions pass through jit, grad, vmap, and lax loops.
  • Entropy, cross entropy, KL divergence, densities, and sampling.
  • Maximum likelihood estimation and conjugate priors.
  • Fisher information and Jeffreys prior densities.
  • Joint distributions that combine independent distributions into one object.
  • Flattening of distributions into unconstrained arrays for neural networks and optimizers.

Documentation

Contributing and citing

Contributions are welcome, including new distributions, which usually take about a hundred lines. See the contribution guidelines.

To cite EFAX, use the "Cite this repository" button on GitHub, which reads CITATION.cff.

About

Exponential families for JAX

Resources

Contributing

Stars

77 stars

Watchers

4 watching

Forks

Releases

Used by

Contributors

Languages