Probability Crash Course: Building Strong Foundations for AI#

1.1 Random Experiments and Events#

A random experiment is any process that leads to well-defined outcomes, but the specific outcome on a given trial cannot be predicted with certainty. For instance:

• Tossing a fair coin: The possible outcomes are “Heads�?(H) and “Tails�?(T).
• Rolling a six-sided die: The possible outcomes are 1, 2, 3, 4, 5, 6.

An event is a set of outcomes in the sample space. If you define the sample space of a coin toss as S = {H, T}, an example event could be “obtaining Heads,�?which corresponds to the subset {H}.

1.2 Probability Axioms#

Let P(A) denote the probability of event A. The three fundamental axioms (Kolmogorov axioms) are:

1. Non-negativity: P(A) �?0 for every event A.
2. Normalization: P(S) = 1, where S is the entire sample space.
3. Additivity: For any two mutually exclusive events A and B, P(A �?B) = P(A) + P(B).

From these axioms, one can derive all standard rules of probability, including:

• Complementary rule: P(A’) = 1 �?P(A), where A’ denotes the complement of event A.
• Inclusion-exclusion principle: For any two events A and B,
  P(A �?B) = P(A) + P(B) �?P(A �?B).

1.3 Calculating Basic Probabilities#

For equally likely outcomes (e.g., fair dice, fair coins):

• Dice: Probability of rolling a 3 = 1/6.
• Coin toss: Probability of landing Heads = 1/2.

If outcomes are not equally likely (e.g., a biased coin or a real-world scenario), we assign probabilities based on available data or a model. For instance, if a coin is biased to land Heads 70% of the time, then P(H) = 0.7 and P(T) = 0.3.

2.1 Conditional Probability#

Conditional probability addresses the question: “If we know some information has occurred, how does that affect the probability of another event?�?Formally, for events A and B with P(B) > 0:

P(A | B) = P(A �?B) / P(B)

Example:

• Suppose P(Umbrella) is the probability you carry an umbrella, and P(Rain) is the probability of rain. If you tend to carry an umbrella more often when it rains, you’re interested in P(Umbrella | Rain).

2.2 Independence of Events#

Two events A and B are independent if and only if:

P(A �?B) = P(A) × P(B)

This means knowing that B occurred does not change the probability of A. Some examples:

• Two fair coin tosses are independent.
• The event of “rolling a 2�?on a die and the event of “the coin landing Heads�?are independent (assuming different, unrelated trials).

If events are not independent, they are said to be dependent, and you must rely on conditional probability to analyze them properly.

3.1 Probability Mass Function (PMF)#

A probability mass function gives the probability that a discrete random variable X takes a value x:

p(x) = P(X = x)

This function must satisfy:

1. p(x) �?0 for all x.
2. The sum of p(x) over all x in the support equals 1.

3.2 Bernoulli Random Variable#

A Bernoulli random variable represents a trial with two outcomes (often 1 for “success,�?0 for “failure�?. If the probability of success is p, then:

p(X = 1) = p
p(X = 0) = 1 �?p

3.3 Binomial Random Variable#

A Binomial random variable represents the number of successes in n independent Bernoulli trials, each with success probability p. The PMF:

p(X = k) = (n choose k) p^k (1 �?p)^(n �?k)

where k = 0, 1, 2, �? n, and (n choose k) = n! / (k!(n �?k)!)

3.4 Poisson Random Variable#

A Poisson random variable models the number of events occurring within a fixed interval of time (or space) when events happen with a known average rate λ (lambda) and independently of the last event. The PMF:

p(X = k) = (λ^k e^(-λ)) / k!

Example: Counting the number of times a website gets a particular kind of request in one minute.

3.5 Python Example for Discrete Distributions#

Below is a Python code snippet using libraries like NumPy and SciPy to demonstrate sampling from these distributions. Note that you need to install SciPy if it’s not already available:

1
import numpy as np
2
from scipy.stats import bernoulli, binom, poisson
3

4
# 1. Bernoulli
5
p = 0.6
6
bern_samples = bernoulli.rvs(p, size=10)
7
print("Bernoulli samples:", bern_samples)
8

9
# 2. Binomial
10
n = 10
11
binom_samples = binom.rvs(n, p, size=10)
12
print("Binomial samples:", binom_samples)
13

14
# 3. Poisson
15
lam = 3
16
pois_samples = poisson.rvs(lam, size=10)
17
print("Poisson samples:", pois_samples)

4.1 Probability Density Function (PDF)#

Instead of a PMF, continuous random variables have a probability density function f(x). The probability that X lies between a and b is:

P(a �?X �?b) = ∫[a to b] f(x) dx

The conditions are:

1. f(x) �?0 for all x.
2. The integral of f(x) over the entire real line is 1.

4.2 Cumulative Distribution Function (CDF)#

The cumulative distribution function F(x) is:

F(x) = P(X �?x) = ∫[−∞ to x] f(t) dt

The CDF is a non-decreasing function that goes from 0 to 1 as x goes from −∞ to �?

4.3 The Uniform Distribution#

A continuous uniform distribution over the interval [a, b] has PDF:

f(x) = 1 / (b �?a), for a �?x �?b
0, otherwise

Mean = (a + b) / 2
Variance = (b �?a)^2 / 12

4.4 The Normal (Gaussian) Distribution#

Arguably the most important distribution in statistics and AI. The PDF of a Normal distribution with mean μ and variance σ² is:

f(x) = (1 / (�?2π) σ)) * exp(-(x �?μ)² / (2σ²))

Many variables in real life (like measurement errors, heights, or exam scores) are well approximated by the Normal distribution. The Central Limit Theorem ensures that sums of independent random variables tend to be normal, making this distribution indispensable in AI for modeling noises and uncertainties.

4.5 The Exponential Distribution#

Characterized by a single rate parameter λ, its PDF is:

f(x) = λ e^(-λx), for x �?0
0, otherwise

This distribution is often used to model the time between Poisson-type events.

4.6 Python Example for Continuous Distributions#

1
import numpy as np
2
from scipy.stats import uniform, norm, expon
3

4
# 1. Uniform
5
a, b = 0, 10
6
uniform_samples = uniform.rvs(loc=a, scale=b-a, size=10)
7
print("Uniform samples:", uniform_samples)
8

9
# 2. Normal
10
mu, sigma = 0, 1
11
normal_samples = norm.rvs(mu, sigma, size=10)
12
print("Normal samples:", normal_samples)
13

14
# 3. Exponential
15
lam = 2
16
expo_samples = expon.rvs(scale=1/lam, size=10)
17
print("Exponential samples:", expo_samples)

5.1 Expected Value (Mean)#

The expected value of a random variable X is the long-run average value after many repetitions of the experiment.

• Discrete case: E[X] = �?x p(x)).
• Continuous case: E[X] = �?x f(x) dx).

Example: If X is a Binomial(n, p), then E[X] = np.

5.2 Variance#

Variance measures dispersion of a random variable about its mean. That is:

Var(X) = E[(X �?E[X])^2]

This can also be computed using:

Var(X) = E[X^2] �?(E[X])^2

Example: If X is Binomial(n, p), then Var(X) = np(1 �?p).

5.3 Covariance and Correlation#

Covariance between two random variables X and Y is:

Cov(X, Y) = E[(X �?E[X])(Y �?E[Y])]

Correlation is a normalized version of covariance:

Corr(X, Y) = Cov(X, Y) / (σ_X σ_Y)

where σ_X and σ_Y are the standard deviations of X and Y, respectively. Correlation ranges from �? to +1, indicating perfect negative or positive linear relationships, respectively, and 0 indicating no linear relationship.

7.1 Prior Distribution#

The prior captures what you believe about a parameter θ before observing new data. It can be motivated by historical data, expert knowledge, or convenience. Common choices:

• Beta distribution as prior for θ in Bernoulli/Binomial processes.
• Gaussian prior for parameters in linear regression.

7.2 Likelihood Function#

The likelihood function measures how “compatible” the observed data is with a given parameter θ. For example, for Binomial data (k successes in n trials), the likelihood is:

L(θ) = (n choose k) θ^k (1 �?θ)^(n �?k)

7.3 Posterior Distribution#

Multiplying the prior by the likelihood gives the unnormalized posterior. You must then normalize to ensure it integrates to 1:

Posterior(θ | data) = [ L(θ) × Prior(θ) ] / P(data)

where P(data) is a scaling factor ensuring the posterior is a valid distribution:

P(data) = �?L(θ) × Prior(θ) dθ

7.4 Example: Coin Toss with Beta-Binomial#

If you place a Beta(α, β) prior on θ (the probability of heads), and observe k heads in n tosses, the posterior for θ is:

Posterior = Beta(α + k, β + (n �?k))

We can demonstrate this in Python:

1
import numpy as np
2
from scipy.stats import beta
3

4
# Prior parameters
5
alpha, beta_param = 2, 2  # Beta(2, 2) prior
6
# Observed data: Suppose we see 6 heads out of 10 tosses
7
k, n = 6, 10
8

9
# Posterior parameters
10
alpha_post = alpha + k
11
beta_post = beta_param + (n - k)
12

13
# Sample from the posterior
14
posterior_samples = beta.rvs(alpha_post, beta_post, size=1000)
15
print("Posterior samples mean:", np.mean(posterior_samples))

Such Bayesian updates allow you to incorporate new evidence continuously, making Bayesian methods popular in AI for dynamic and adaptive models.

8.1 Transition Probabilities#

A Markov chain is described by a transition probability matrix P, where P(i, j) = P(X_{n+1} = j | X_n = i). For a system with N states, P is an N × N matrix:

P = | p(1�?) p(1�?) … p(1→N) | | p(2�?) p(2�?) … p(2→N) | | … … … | | p(N�?) p(N�?) … p(N→N) |

The rows of the matrix must sum to 1.

8.2 Steady State Distribution#

For certain conditions (e.g., irreducible and aperiodic), a Markov chain has a unique stationary (steady state) distribution π such that:

π P = π

This π is often used to understand the long-term behavior of the Markov chain. In AI, Markov chains are used in:

• Hidden Markov Models (HMMs) for speech recognition and sequence data.
• Markov Decision Processes (MDPs) for reinforcement learning.

8.3 Python Example: Simple Markov Chain Simulation#

1
import numpy as np
2

3
# Transition matrix for a simple weather model: [Sunny, Rainy]
4
P = np.array([
5
    [0.8, 0.2],  # If today is Sunny: 80% chance tomorrow sunny, 20% chance rainy
6
    [0.4, 0.6]   # If today is Rainy: 40% chance tomorrow sunny, 60% chance rainy
7
])
8

9
states = ["Sunny", "Rainy"]
10

11
# Initial distribution
12
initial_state = 0  # Suppose we start with 'Sunny'
13
num_steps = 10
14

15
current_state = initial_state
16
simulated_chain = [states[current_state]]
17

18
for _ in range(num_steps):
19
    current_state = np.random.choice([0, 1], p=P[current_state])
20
    simulated_chain.append(states[current_state])
21

22
print("Simulated Markov chain:", simulated_chain)

9.1 Basic Monte Carlo Estimation#

Suppose you want to estimate E[f(X)] for some random variable X. If you can generate samples X�? X�? �? X_N from the distribution of X, an estimate is:

(1 / N) �?f(X�?

9.2 Markov Chain Monte Carlo (MCMC)#

When direct sampling from the distribution is difficult, MCMC constructs a Markov chain whose stationary distribution is the target distribution. Popular MCMC methods:

• Metropolis-Hastings
• Gibbs Sampling

Using MCMC, you can approximate posterior distributions in Bayesian models by sampling rather than solving integrals analytically.

9.3 Example: MCMC via PyMC3 or PyStan#

Below is a simplified example using the Python library PyMC (or PyMC3) to demonstrate MCMC sampling for a coin toss scenario. (Note: PyMC needs to be installed; it’s a powerful library for Bayesian inference.)

1
# This snippet is illustrative and may require a Jupyter notebook environment
2

3
import pymc as pm
4
import numpy as np
5

6
# Observed data
7
n = 10
8
k = 6
9

10
with pm.Model() as coin_toss_model:
11
    # Prior
12
    theta = pm.Beta('theta', alpha=1, beta=1)
13
    # Likelihood
14
    y = pm.Binomial('y', n=n, p=theta, observed=k)
15

16
    # Posterior sampling
17
    trace = pm.sample(1000, tune=1000, cores=1, random_seed=42)
18

19
print("Posterior mean of theta:", np.mean(trace['theta']))

In real projects, you can specify more complex models (hierarchical, multi-parameter, etc.). MCMC allows you to handle situations where typical analytical methods become unmanageable.

10.1 Gaussian Mixture Models#

A Gaussian Mixture Model (GMM) is a probabilistic model that assumes data is generated from a mixture of finite Gaussian distributions, each with unknown parameters. Widely used in clustering (e.g., for unsupervised learning tasks) or density estimation.

10.2 Dirichlet Distributions in Topic Modeling#

For modeling proportions (like word distributions in documents), Dirichlet distributions are often used, particularly in Latent Dirichlet Allocation (LDA) for topic modeling.

10.3 Hidden Markov Models and Beyond#

Hidden Markov Models (HMMs) extend Markov chains by assuming that the observed data is emitted from hidden states that form a Markov chain. Widely used in:

• Natural language processing
• Speech recognition
• Bioinformatics

Extensions include:

• Conditional Random Fields (CRFs) for structured prediction.
• Hierarchical HMMs for more complex sequential dependencies.

10.4 Bayesian Neural Networks#

Neural networks with Bayesian approaches place distributions over weights, leading to estimates of uncertainty along with point predictions. Though computationally heavy, techniques like Variational Inference or MCMC can approximate posterior distributions over neural network parameters.