Bayes or Bust: How Uncertainty Powers Smarter AI#

Random Variables and Probability Distributions#

To understand Bayesian methods, we need to start with the basics of probability. A random variable is any quantity that can vary due to chance. For instance, the outcome of flipping a fair coin can be represented by a random variable that takes the value 0 (Heads) or 1 (Tails).

A probability distribution assigns probabilities to the possible outcomes of a random variable. In the case of the fair coin, there’s a 50% chance for Heads and a 50% chance for Tails. More generally, continuous random variables (like the height of a person) have probability density functions (PDFs) rather than discrete distributions.

Bayes�?Theorem at a Glance#

Bayes�?Theorem states that:

P(A | B) = [ P(B | A) * P(A) ] / P(B)

where:

• P(A | B) is the posterior probability (the revised probability of A given B).
• P(A) is the prior probability (our initial belief about A before seeing B).
• P(B | A) is the likelihood (the probability of observing B if A is true).
• P(B) is the marginal probability (the total probability of observing B).

Bayes�?Theorem is important because it tells us how to update our prior belief (what we believe before we see new data) into a posterior belief (what we believe after seeing new data).

Frequentist Overview#

In the Frequentist world, probabilities are defined by long-run frequencies in repeated experiments. You estimate parameters (like the mean of a distribution), but those parameters are considered fixed (though unknown). Confidence intervals in the Frequentist approach are derived such that, if you were to repeat an experiment an infinite number of times, the true parameter would lie in those intervals a certain percentage of the time.

Bayesian Overview#

In Bayesian thinking, probability is a subjective measure of belief or uncertainty. Parameters themselves (like the mean of a distribution) are treated as random variables. We start with a prior belief and then condition on observed data to get a posterior belief. Bayesian credible intervals can be interpreted as intervals within which the parameter lies with a certain probability given the data.

Priors#

A prior probability distribution expresses what we know (or think we know) about a parameter before seeing any data. Priors can come in many forms:

• Informative prior: We use existing knowledge to shape the distribution (e.g., from previous experiments).
• Non-informative (or weakly informative) prior: We try to remain reasonably neutral if we lack specific prior knowledge.
• Conjugate prior: A mathematical choice that simplifies Bayesian updating (e.g., Beta distribution for the parameter of a Binomial).

Likelihood#

The likelihood function, often written as P(data | parameter), describes the probability of observing the data given a parameter value. In Bayesian analysis, this same function is used to update our beliefs from the prior to get a posterior distribution.

Posterior#

Bayes�?Theorem says:

Posterior �?Prior × Likelihood

(Here “∝�?means “proportional to,�?since we often drop the normalizing constant for convenience, and later adjust so that the probabilities sum or integrate to 1.)

Evidence or Marginal Likelihood#

The evidence or marginal likelihood is the probability of the observed data integrating (or summing) over all possible parameter values. Formally,
P(data) = �?P(data | θ) P(θ) dθ
for continuous θ, or a sum if θ is discrete.

Though important, computing the evidence can be difficult for complex models and is often the main computational challenge in Bayesian methods.

Coin Flip Example#

Suppose:

• Prior: θ ~ Beta(α, β).
• Data: We observe H heads and T tails.
• Likelihood: Binomial(H + T, θ).

Then the posterior is: θ | data ~ Beta(α + H, β + T).

Why? Because the Beta distribution is a conjugate prior for the binomial likelihood. Each time we observe heads, α increments by 1, and each time we observe tails, β increments by 1.

1
import numpy as np
2
import matplotlib.pyplot as plt
3
from scipy.stats import beta
4

5
# Prior parameters for Beta distribution
6
alpha_prior = 2
7
beta_prior = 2
8

9
# Suppose we flip a coin 10 times, and observe 7 heads, 3 tails
10
heads = 7
11
tails = 3
12

13
alpha_posterior = alpha_prior + heads
14
beta_posterior = beta_prior + tails
15

16
# Plot prior vs posterior
17
theta = np.linspace(0, 1, 200)
18
prior_pdf = beta.pdf(theta, alpha_prior, beta_prior)
19
posterior_pdf = beta.pdf(theta, alpha_posterior, beta_posterior)
20

21
plt.figure(figsize=(8, 5))
22
plt.plot(theta, prior_pdf, label='Prior', color='blue')
23
plt.plot(theta, posterior_pdf, label='Posterior', color='red')
24
plt.legend()
25
plt.title('Beta Prior vs Posterior')
26
plt.xlabel('θ')
27
plt.ylabel('Density')
28
plt.show()

Why Bayesian Networks?#

1. Modularity: Complex joint distributions are factored into smaller, manageable parts.
2. Inference: Bayesian networks support both forward (predicting future or unobserved variables given known evidence) and backward inference (updating beliefs about causes when effects are observed).
3. Causality: If the network’s structure captures real causal relationships, it can be used beyond correlation and helps us reason about interventions.

Formally, a Bayesian network factorizes the joint distribution of variables X1, X2, �? Xn as follows:

P(X1, X2, �? Xn) = �?P(Xi | Parents(Xi))

Where Parents(Xi) denotes the immediate parents of Xi in the graph.

Why MCMC?#

1. High-Dimensionality: Directly computing integrals or sums in high-dimensional parameter spaces can be intractable.
2. Flexibility: MCMC algorithms work for a large class of posterior distributions, regardless of their shape.

Common MCMC Algorithms#

• Metropolis-Hastings: Proposes new samples based on a proposal distribution and uses an acceptance rule.
• Gibbs Sampling: Special case of Metropolis-Hastings where we sample conditionally from each parameter in turn.
• Hamiltonian Monte Carlo (HMC): Uses gradient information of the log-posterior to explore the parameter space more efficiently.

Pseudo-Code for Metropolis-Hastings#

1. Initialize θ0.
2. Propose θ* from a proposal distribution q(θ* | θt).
3. Compute acceptance probability:
   α = min(1, [P(data | θ*) P(θ*) q(θt | θ*)] / [P(data | θt) P(θt) q(θ* | θt)]).
4. Accept proposal (θt+1 = θ*) with probability α, otherwise reject (θt+1 = θt).
5. Repeat many times.

Example: Estimating a Mean with MCMC#

Let’s do a simple example where we want to estimate the mean of a normal distribution, assuming we know the variance (σ²). For clarity, we’ll try a Metropolis-Hastings sampler in Python.

1
import numpy as np
2
import matplotlib.pyplot as plt
3

4
# Simulate some data
5
np.random.seed(42)
6
true_mu = 5.0
7
sigma = 2.0
8
data = np.random.normal(true_mu, sigma, 100)
9

10
# Define prior for mu ~ Normal(0, 5) as an example
11
def log_prior(mu):
12
    # Normal(0, 5) => standard deviation 5 => variance 25
13
    return -0.5 * (mu**2 / 25)
14

15
# Likelihood: data ~ Normal(mu, sigma^2)
16
def log_likelihood(mu, data):
17
    return -0.5 * np.sum((data - mu)**2) / (sigma**2)
18

19
# Combine to get log posterior
20
def log_posterior(mu, data):
21
    return log_prior(mu) + log_likelihood(mu, data)
22

23
# Metropolis-Hastings
24
n_samples = 5000
25
mu_current = 0.0  # initial guess
26
chain = []
27

28
def proposal(mu):
29
    return np.random.normal(mu, 1.0)
30

31
for i in range(n_samples):
32
    mu_proposal = proposal(mu_current)
33
    # Compute acceptance probability
34
    log_alpha = (log_posterior(mu_proposal, data)
35
                 - log_posterior(mu_current, data))
36
    if np.log(np.random.rand()) < log_alpha:
37
        mu_current = mu_proposal
38
    chain.append(mu_current)
39

40
# Discard burn-in
41
burn_in = 1000
42
chain_after_burnin = chain[burn_in:]
43

44
# Plot results
45
plt.figure(figsize=(8,4))
46
plt.plot(chain_after_burnin, color='blue', alpha=0.5)
47
plt.axhline(true_mu, color='red', linestyle='--', label='True Mean')
48
plt.title('Trace of mu')
49
plt.legend()
50
plt.show()
51

52
# Print statistics
53
estimated_mu = np.mean(chain_after_burnin)
54
print(f"Estimated mu: {estimated_mu:.2f}")

Explanation:

1. We simulate data from a Normal distribution with a true mean of 5.0 and a known standard deviation of 2.0.
2. We define a prior for μ, a Normal(0,5).
3. We define the likelihood under a Normal(μ, 4).
4. We combine them for the posterior.
5. We use Metropolis-Hastings to sample from this posterior.

This approach can give us an approximate posterior mean, credible intervals, and visual insight (through the trace plot) about how μ evolves during sampling.

9.1 Bayesian Linear Regression#

9.2 Gaussian Processes#

Gaussian Processes (GPs) allow us to place a prior over functions, not just parameters. A GP is completely specified by a mean function and a covariance function (kernel). Formally:

f(x) ~ GP(m(x), k(x,x�?)

where m(x) is the mean function and k(x,x�? is the kernel defining how points correlate. GPs are extremely flexible for regression tasks, especially when the exact form of the function is unknown.

9.3 Bayesian Neural Networks#

Neural networks typically have deterministic weights. In a Bayesian neural network (BNN), each weight has a posterior distribution. The idea is to capture model uncertainty about the weights, which can improve predictions in small-data regimes or in tasks that require credible intervals. However, BNNs can be challenging to implement and expensive to train, requiring approximate inference methods like variational inference or MCMC.

Medical Diagnosis#

Bayesian methods are very appealing in medicine due to the inherent uncertainty in diagnoses. Suppose you want to compute the probability of a disease given a particular lab test result. Bayesian updating is vital. For instance, if a test is “positive,�?the updated probability depends not only on the test sensitivity and specificity (likelihood) but also on the prior prevalence of the disease.

Spam Filtering#

Classic spam filters (like the Bayesian spam filter) use Bayes�?Theorem to update the probability that an email is spam based on the frequencies of words (likelihood) and prior probabilities. With each email processed, the spam filter refines its beliefs about which words indicate spam.

Robotics#

Robots often use Bayesian filters (e.g., the Kalman filter, Particle filter) to track uncertain states like location or velocity. By combining sensor measurements (noisy and partial) with motion models, they can maintain a probability distribution over states (a posterior), rather than a single guess.

A/B Testing#

Online A/B testing can become more flexible and informative with Bayesian methods. Instead of analyzing data at the end of a trial, you can continuously update your beliefs about which version (A or B) is better as data arrives. Credible intervals can instantly show how uncertain you are about a difference, rather than waiting for a p-value at the end.