Introduction to Maximum Likelihood Estimate in Statistical Modeling

Maximum Likelihood Estimate Jyh-Shing Roger Jang ( ) CSIE Dept, National Taiwan University 2025/6/3

Intro. to Maximum Likelihood Estimate MLE Maximum likelihood estimate Goal: Given a dataset with no labels, how can we find the best statistical model with the optimum parameters to describe the data? Applications Prediction Analysis 2/15

What Are Statistical Models? Statistical models are used to describe the probabilities of random variables Discrete variables Probability functions Continuous variables Probability density functions (PDF) Examples Discrete variables The outcome of tossing a coin or a dice Continuous variables Personalized PDF! The distance to the bull eye when throwing a dart The time needed to run 100-m dash The heights of second-grade students 3/15

More about Models Discrete variables Outcome of tossing a coin Pr{head}=Pr{tail}=1/2 Outcome of tossing a dice Pr{1}=Pr{2}= =Pr{6}=1/6 Continuous variables Temperatures during the summer A PDF of Gaussian or normal distribution 2 1 1 2 x ( ) ; , = 2 exp g x 2 Quiz! 6 , 4 6 = 2) Pr ( ; , x g x dx 4 Probability of x in [4, 6] 4/15

Basic Steps in MLE Steps Perform a certain experiment to collect the data. Choose a parametric model of the data, with certain modifiable parameters. Formulate the likelihood as an objective function to be maximized. Maximize the objective function and derive the parameters of the model. 1. 2. 3. 4. Examples Flip a coin To find the probabilities of head and tail Throw a dart To find your PDF of distance to the bull eye 5/15

Probability Functions for Discrete Variables Flip an unfair coin 5 times to get 3 heads and 2 tails By intuition: Pr{head}=3/5, Pr{tail}=2/5 By MLE Assume these 5 tosses are independent events to have the overall probability ( ) ( ) ) ( = dp = + = 3 2 , , , 1 , 0 0 J p q p q with p q p q 2 = 3 ( ) p 1 J p p p dJ 0 = = / 3 , 5 2 / 5 p q 6/15

Inequality of Arithmetic and Geometric Means AM-GM inequality n = x / 1 n i n 1 n , , 0 i x with x i i i Quiz! = 1 i = = = The equality holds only when . x x x 1 2 n Proof of this inequality Wikipedia How to use the inequality to solve MLE problem? q q p p p + + + + 5 / 1 3 2 p q 3 3 3 2 2 5 3 2 3 2 p q = = = 3 2 achieves maximum its when , p q p q 7/15 3 2 5 5

How to Prove AM-GM Inequality? How to prove it? Jensen s inequality a is ln + n m = concave m function + n m y x + ln ln mp nq p n q ln , , , , 0 with p q m n + Proof by induction n n = = ln x x i i 1 1 ln , , 0 i i with x i i n n Please refer to my detailed coverage of Jensen's inequality: https://www.youtube.com/watch?v=E8zsia64ceM 8/15

Proof by Induction = 1 n x x 1 1 ( ) + + ln ln x x x x 2 = 2 ln (Or . you start with can ) 0 1 2 1 2 n x x 1 2 2 2 + + x x x x + + 2 2 ln ln x x 1 2 1 2 + + + + 3 3 ln ln ln x x x x x x 2 2 = = 1 2 3 1 2 3 3 ln ln n 3 3 3 3 k k = i = i ln x x i i = 1 k 1 assumption by holds ln n k k k k = i = i ln x x i i k = i + + ln ln ln k x k x 1 k 1 x + + 1 1 k k + 1 k k k i = i = i + ln x x x + k x 1 k + 1 i k i + 1 k = + = = 1 ln 1 ln 1 k n k + + + + + 1 1 1 1 1 k k k k 9/15

Probability Functions for Discrete Variables Toss a 3-side die for many times and obtain n1of side 1, n2of side 2, and n3of side 3, then what is the most likely probabilities for sides 1, 2, and 3, respectively? Our intuition By MLE ( ) = + + = n n n Quiz! , , , , 1 , 0 , 0 0 J p q r p q r with p q r p q r 3 1 2 10/15

MLE for PDF of Continuous Variables in 1D Detailed coverage PDF 2 1 1 2 x ( ) ; , = 2 exp g x 2 Overall PDF, or likelihood ( = , X x x x 1 2 n ) ( ) n = i = 2 2 ; , ; , p X g x i 1 ( ) ( , = ; , 2 2 ) ln J p X n ( ) = ; , 2 ln g x i = 1 i Log likelihood MLE! n ( ) = ; , 2 ln g x i = 1 i 2 ( , 2 n n ) x 1 n J n 1 2 1 2 x ( ) = = = = 0 i x ln 2 ln i i = = = 1 1 i i 1 i 2 ( , ) n 1 n n x x J n 2 ( ) 2 1 2 x n n = = = 2 0 i i x ( ) = ln 2 ln i n i 2 2 = = 1 1 i i = 1 i Quiz! 11/15

MLE for PDF of Continuous Variables of ND Detailed coverage PDF 1 1 ( ) ( ) ( ) T = 1 ; , exp g x x x ( ) 2 d 2 Overall PDF, or likelihood = i = , X x x x 1 2 n n ( ) ( ) = ; , ; , p X g x i 1 Log likelihood 1 n ( n ) ( ) = = = 1 ( , ) ln ; , J p X ( , ) 2 J x i 2 1 i ( ) = i = ln ; , g x n = i i = 1 x n 1 i n 1 ( ) = n = ln ; , g x 1 n i = i = = ( , ) 0 J x 1 i i n 1 1 d 2 ( ) ( ) ( ) = i T 1 = 1 ln 2 ln x x MLE! i i 2 2 T 1 1 n = i 1 n nd 2 n 2 = x x ( ) ( ) ( ) = i T = 1 ln 2 ln x x i i n i i 2 1 1 12/15

Q & A Quiz! Questions Can we choose other PDFs instead of Gaussian/normal distributions? Yes! What are the other available PDFs? Uniform distribution Poisson distribution How do I know the selected PDF is appropriate? 13/15

Exercise: How to Compute Covariance Matrix Find the covariance matrix of two random variables with sample values as follows: X=[1 4 3 2 0], Y=[3 1 1 2 3]. Solution: mean(X)=2, mean(Y)=2 X2=X-mean(X)=[-1 2 1 0 -2], Y2=Y-mean(Y)=[1 -1 -1 0 1] A=[X2; Y2] (X2 on top of Y2, with a dim of 2X5) Covariance matrix = (A*AT)/4= 14/15

Exercise: MLE for Poisson Distribution Derive MLE for the Poisson distribution: 15/15

Exercise: MLE for Uniform Distribution Derive MLE for the uniform distribution: 16/15