Bayesian Inference

INS-605: Data Analysis II

Lecturer: Dr. Sothea HAS

Content

• 1 Motivation
• 2 Bayes’ Theorem
• 3 Posterior Mean
• 4 Credible Intervals

helpful resources

🌐 Understading the Difference between Bayesian and Frequentist Statistics by Fornacon-Wood et al. (2021).

Learning Objectives

By the end of this lecture, you should be able to:

• Explain the motivation for Bayesian inference.
• Distinguish Frequentist and Bayesian interpretations.
• Apply Bayes’ theorem to estimate parameters.
• Explain the roles of the prior, likelihood, and posterior.
• Interpret Bayesian credible intervals.
• Perform Bayesian updating using a Beta-Binomial model.
• Understand posterior predictive inference.

1 Motivation

1.1 Frequentist Inference

• Frequentist framework:
  • The parameter is fixed.
  • The parameter is unknown.
  • The data are random.
• Example: Eggs survival model: \(X_1,\ldots,X_n \sim \mathcal{B}(\color{red}{p})\), \(\color{red}{p}\) is unknown.
• Frequentist inference:
  • Point estimate: find \(\color{red}{\hat{p}}\), for example by maximizing the likelihood function.

• Interval estimate: find an interval \(I_{\color{green}{1-\alpha}}=[a(\color{red}{\hat{p}}), b(\color{red}{\hat{p}})]\) with \(\mathbb{P}(\color{red}{p}\in I_{\color{green}{1-\alpha}})=\color{green}{1-\alpha}\).

Frequentist

The parameter is fixed and unknown. The fish doesn't move — we just observe what surfaces.

surface = observed data

fixed parameter

x₁

x₂

x₃

x₄

x₅

Position: a single fixed value, and the uncertainty is due to the bubbles rising.

1.2 Bayesian Thinking

• The parameter is uncertain & data is uncertain but observed.
• Ex: Before observing the data, you may belive that it’s fair (\(p=0.5\)). However, after some tosses, you may change your belief about the parameter \(p\).

Bayesian

The parameter itself has a distribution. The fish swims and leaves its own trail of data behind.

surface = observed data

random parameter

x₁

x₂

x₃

x₄

x₅

x₆

Position: The fish itself is moving and can be anywhere, governs by the density function.

• Prior: Your initial belief in value of parameter.
• Bubbles: Data \(X_1,X_2,...\) are observed.
• Posterior: Your updated belief in value of parameter after observing the data.

Frequentist vs Bayesian

Question	Frequentist	Bayesian
What is the parameter?	Unknown and fixed	Unknown and uncertain
What is random?	Data	Parameter uncertainty \(\to\) data
Main tool	Likelihood	Bayes’ theorem
Main result	Point estimate (MLE)	Posterior distribution
Interval	Confidence interval	Credible interval
Prediction	Based on estimated parameters	Based on posterior uncertainty

2 Bayes’ Theorem

2.1 Bayes’ Theorem for Events

• For two events \(A\) and \(B\) with non-zero probabilities, Bayes’ theorem states:

\[P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}\]

where:

• \(P(A)\): prior probability
• \(P(B\mid A)\): likelihood
• \(P(B)\): evidence
• \(P(A\mid B)\): posterior probability

Example: Disease Testing

• Suppose a disease affects 50% of the population (\(\color{blue}{D=1}\)).
• A test is 99% accurate:
  • Correctly identifies 99% of those with the disease,
  • Correctly identifies 99% of those without the disease.
• If a person tests positive, how likely do you think they actually have the disease?

\[P(\color{blue}{D=1}\mid \color{red}{T=1}) = \frac{P(\color{red}{T=1}\mid \color{blue}{D=1})P(\color{blue}{\color{blue}{D=1}})}{P(\color{red}{T=1})}.\]

• \(P(\color{red}{T=1}\mid \color{blue}{D=1}) = 0.99\).
• \(P(\color{blue}{D=1}) = 0.5\).
• Testing positive:
  \(\begin{align*}& P(\color{red}{T=1})\\ = &\ 0.99\times0.5 + 0.01\times0.5\\ = &\ 0.5 \text{ (why?)}\end{align*}\)
• Hence, \(P(\color{blue}{D=1}\mid \color{red}{T=1}) = 0.99\).
• Is this surprising?

Example: Rare Disease Testing

• Suppose a disease affects only 1% of the population.
• \(P(\color{red}{T=1}\mid \color{blue}{D=1}) = 0.99\).
• \(P(\color{blue}{D=1}) = 0.01\).
• Testing positive:
  \(\begin{align*}& P(\color{red}{T=1})\\ = &\ 0.99\times0.01 + 0.01\times0.99\\ = &\ 0.0198 \text{ (why?)}\end{align*}\)
• Hence, \(P(\color{blue}{D=1}\mid \color{red}{T=1}) = \frac{0.99\times0.01}{0.0198} = 0.5.\)
• Is this surprising?

• If there are 1M people, then

Test	Sick	Healthy
Positive	9,900	9,900
Negative	100	980,100
Total	10,000	990,000

• For very rare diseases, even a highly accurate test can produce a large number of false positives (9,900).
• A second test can help reduce false positives.

2.2 Bayes’ Theorem for Parameters

• Let \(\color{blue}{x}\) be the observed data and \(\color{red}{\theta}\) be an unknown parameter.

\[\underbrace{p(\color{red}{\theta}\mid \color{blue}{x})}_{\text{Posterior}}=\frac{\overbrace{p(\color{blue}{x}\mid\color{red}{\theta})}^{\text{Likeihood}}\overbrace{p(\color{red}{\theta})}^{\text{Prior}}}{\underbrace{p(\color{blue}{x})}_{\text{How likely to observe data }\color{blue}{x}}}\propto p(\color{blue}{x}\mid\color{red}{\theta})p(\color{red}{\theta}).\]

• Coin toss example, let \(\color{red}{\theta=p}\) be unknown probability of head per toss (success).
  • Prior: \(\color{red}{\theta}\sim\color{purple}{\operatorname{Beta}(2,2)}\), i.e., \(f(\color{red}{\theta})=\frac{\color{red}{\theta}(1-\color{red}{\theta})}{B(2,2)}\).
  • Data: \(N=30\) and \(n(H)=\color{green}{18}\).
  • Likelihood: \(p(\color{blue}{x}=\color{green}{18}\mid \color{red}{\theta})=\binom{30}{\color{green}{18}}\color{red}{\theta}^{\color{green}{18}}(1-\color{red}{\theta})^{12}\).

Read more about Beta distribution on Wikipedia.

2.2 Bayes’ Theorem for Parameters

• Let \(\color{blue}{x}\) be the observed data and \(\color{red}{\theta}\) be an unknown parameter.

\[\underbrace{p(\color{red}{\theta}\mid \color{blue}{x})}_{\text{Posterior}}=\frac{\overbrace{p(\color{blue}{x}\mid\color{red}{\theta})}^{\text{Likeihood}}\overbrace{p(\color{red}{\theta})}^{\text{Prior}}}{\underbrace{p(\color{blue}{x})}_{\text{How likely to observe data }\color{blue}{x}}}\propto p(\color{blue}{x}\mid\color{red}{\theta})p(\color{red}{\theta}).\]

\[\begin{align*}p(\color{red}{\theta}\mid \color{blue}{x}=\color{green}{18})&\propto p(\color{blue}{x}=\color{green}{18}\mid\color{red}{\theta})p(\color{red}{\theta})\\ &= \left[\binom{30}{\color{green}{18}}\color{red}{\theta}^{\color{green}{18}}(1-\color{red}{\theta})^{12}\right][\color{red}{\theta}(1-\color{red}{\theta})]\\ &\propto \color{red}{\theta}^{19}(1-\color{red}{\theta})^{13}\\ &=\color{red}{\theta}^{\color{purple}{20}-1}(1-\color{red}{\theta})^{\color{purple}{14}-1}\\ \Rightarrow \underbrace{(\color{red}{\theta}\mid \color{blue}{x}=\color{green}{18})}_{\text{New belief of }\color{red}{\theta}}&\sim\color{purple}{\text{Beta}(\color{purple}{20},\color{purple}{14})}\quad (\color{purple}{\text{Posterior}}).\end{align*}\]

Influence of Prior

• Prior belief is expressed by \(p(\color{red}{\theta})\) (dashed lines), and when observing data \(\color{blue}{x}\), it updates to a new belief (posterior) \(p(\color{red}{\theta}\mid \color{blue}{x})\).
• Different priors can result in different posterior distributions.

Weak and Informative Priors

• A prior can be:
  • Weakly or strongly informative.
  • Symmetric or asymmetric.
  • Based on previous studies or expert knowledge.
  • Chosen for mathematical convenience.
• For example: \(\color{red}{\theta}\sim\color{blue}{\operatorname{Beta}(1,1)}\) is uniform on \(\color{blue}{U[0,1]}\).
• Meanwhile: \(\color{red}{\theta}\sim\color{orange}{\operatorname{Beta}(20,5)}\) expresses a stronger belief that \(\color{red}{\theta}\) is relatively large.

Python: Visualizing Beta Priors

3 Posterior Mean

3.1 Posterior Mean

• In Bayesian Framework, Posterior Mean, the expected value of the parameter given the observed data, i.e., \(\mathbb{E}[\color{red}{\theta}|\color{blue}{x}]\), is a point summary of the parameter of the population.
• After computing the posterior, we compute its expectation.
  • For example if \((\color{red}{\theta}|\color{blue}{x})\sim\text{Beta}(\alpha, \beta)\Rightarrow \mathbb{E}[\color{red}{\theta}|\color{blue}{x}]=\frac{\alpha}{\alpha+\beta}\).

3.2 MLE vs Posterior Mean

• Let’s compare MLE and Posterior Mean on Coin Toss example.
• Maximum Likelihood Estimator (MLE):
  • Log-likelihood: \(\ell(\color{red}{\theta})=18\ln(\color{red}{\theta})+12\ln(1-\color{red}{\theta})+\text{cont.}\)
  • \(\ell'(\color{red}{\theta})=18/\color{red}{\theta}-12/(1-\color{red}{\theta})\)
  • MLE: \(\color{red}{\widehat{\theta}}=18/30=0.6.\)
  • This is purely data-driven.
• Posterior Mean:
  • For prior \(\color{red}{\theta}\sim \text{Beta}(2,2)\)
  • Posterior \((\color{red}{\theta}\mid \color{blue}{x})\sim \text{Beta}(20,14)\)
  • Posterior Mean: \(\color{green}{\widehat{\theta}}=\mathbb{E}(\color{red}{\theta}\mid \color{blue}{x})=\frac{\alpha}{\alpha+\beta}=\frac{20}{20+14}=0.5882.\)

4 Credible Intervals

4.1 Credible Intervals

• Posterior mean is point summary of the parameter.
• A credible interval is an interval estimate of a parameter that captures the uncertainty in the parameter.
• If \(\color{red}{\alpha}\in (0,1)\), be a level of error, then a credible interval at level \(1-\color{red}{\alpha}\) of \(\color{red}{\theta}\) denoted by \(\text{CI}_{1-\color{red}{\alpha}}=[a,b]\) such that \(\mathbb{P}(a\leq \color{red}{\theta}\leq b\mid \color{blue}{x})=1-\color{red}{\alpha}\).
• This means:

Given the observed data \(\color{blue}{x}\) and the prior \(p(\color{red}{\theta})\), there is a \(1-\color{red}{\alpha}\) posterior probability that \(\color{red}{\theta}\in[a,b]\).

• The interval is calculated from the posterior distribution.

Credible Interval for Coin Toss

Bayesian Prediction

• Bayesian inference can also predict a future observation.
• Suppose we want to predict the chance of getting head next:

\[\begin{align*}p(X_{new}=1\mid \color{blue}{x})&=\int_{0}^1\underbrace{p(X_{new}=1\mid \color{red}{\theta})}_{\text{Chance of head}}p(\color{red}{\theta}\mid \color{blue}{x})d\color{red}{\theta}\\ &=\int_{0}^1\color{red}{\theta}p(\color{red}{\theta}\mid \color{blue}{x})d\color{red}{\theta}\\ &=\mathbb{E}(\color{red}{\theta}\mid \color{blue}{x})=20/34=\color{green}{0.5882}.\end{align*}\]

• Remark Bayesian theory fits well with online learning as Posterior can be considered a new Prior for the next observation.

Frequentist vs Bayesian Workflow

• Frequentist:

Choose probability model
        ↓
Construct likelihood
        ↓
Optimize likelihood
        ↓
Obtain MLE
        ↓
Construct confidence interval
        ↓
Make predictions

• Bayesian:

Choose probability model
        ↓
Specify prior distribution
        ↓
Construct likelihood
        ↓
Combine prior and likelihood
        ↓
Obtain posterior distribution
        ↓
Calculate posterior summaries
        ↓
Construct credible intervals
        ↓
Make posterior predictions

Conjugate Priors

Likelihood	Prior	Posterior
Bernoulli/Binomial	Beta	Beta
Poisson	Gamma	Gamma
Normal mean with known variance	Normal	Normal
Multinomial	Dirichlet	Dirichlet

• Conjugacy makes Bayesian inference mathematically convenient.
• Many real-world models do not have a closed-form posterior.
  • Logistic regression
  • Neural network models…

🥳 Yeahhhh 🥂!!!

Let’s take a break!