2. Parameter Estimation Flashcards
estimation of a mean, estimation of a variance, estimation of a proportion
What is parameter estimation used for?
- in cases where we have already selected a family of models, but where there are still unknown parameter values
- parameter estimation is used to find the parameter value such that the model fits the data best
θ
- the symbol θ is used to denote a general parameter
- depending on which parameter estimation is being considered θ can stand for mean, variance or even a vector where there are several unknown parameters
Parameter Estimator
Definition
-a parameter estimator for θ∈R^d is a statistic:
θ^ : R -> R
-such that for a random sample X1,…,Xn ~ Pθ, we have:
θ^ (X1,…,Xn) ~ θ
-the idea of this is that we use random samples from the model, where we know the parameter value θ, to asses how well the estimator θ^ performs
iid
Definition
-independently and identically distributed
Estimation of a Mean
Description
-consider a model where X1,…,Xn∈R are iid with E(Xi)=μ for all i∈{1,2,…,n}
-we can use the sample average:
μ^ (x1,x2,…,xn) = 1/n Σxi
-for all x∈R^n as an estimator for the model mean μ
Estimation of a Mean
Expectation and Variance Lemma
-let X1,…,Xn be iid with expectation E(Xi)=μ and variance Var(Xi)=σ² for all i∈{1,2,…,n}
-then:
E(μ^ (x1,x2,…,xn)) = μ
-and:
Var(μ^ (x1,x2,…,xn)) = σ²/n
-this shows that μ^ is a good estimator for μ, on average μ^ gives the correct answer μ
-and looking at the variance, as the amount of data increases, the fluctuations in the estimate decrease and so it becomes more reliable
Bias
Definition
-the bias of an estimator θ^ for a parameter θ is:
bias(θ^) = E(θ^(X1,…,Xn) - θ
-where X1,…,Xn ~ Pθ
Unbiased
Definition
-an estimator is called unbiased if:
bias(θ^) = 0
-for all θ
Law of Large Numbers
-alternative justification for the use of the sample average as an estimator for the mean, μ
-let Xi (for i∈ℕ) be an iid sequence of random variables with E(Xi)=μ for all i∈ℕ
-then:
P(lim 1/n ΣXi=μ) = 1
Estimation of a Variance
Description
-as an estimator for the variance σ², we can use the sample variance:
σ²^(x1,…,xn) = 1/n-1 Σ(xi-x~)²
-for all x∈R^n where x~ denotes the sample mean
Estimation of a Variance
Unbiased Estimator Lemma
-let X1,…,Xn be idd with variance Var(Xi)=σ² for all i∈{1,2,…,n}
-then:
E(σ²^(X1,…,Xn)) = σ²
-i.e. σ²^ is an unbiased estimator for σ²
Estimation of a Proportion
When is this used?
-this is an estimator for attribute (non-numerical) data
Estimation of a Proportion
Outline
-if we have observed attribute data x1,…,xn∈{A,B} and want to describe these data using the model X1,…,Xn∈{A,B} iid
-where P(Xi = A) = p and P(Xi=B)=1-p
-parameter p describes the proportion of individuals in the population which have attribute value A
-we can estimate the parameter p using the proportion of A in the sample:
p^(x1,…,xn) = |{i=1,…,n | xi=1}| / n
Indicator Function
Definition
-the indicator function of a set S is the function 1s which is given by:
1s(x) = 1 if x∈S OR 0 else
Estimation of a Proportion
Indicator Function
-using the definition of an indicator function for S={A} we can write the number of A in the population as:
|{i=1,...,n | xi=1}| = Σ 1s(xi)
-thus we get:
p^(x1,...,xn) = 1/n Σ 1s(xi)
-where the sum is from i=1 to i=n
