數學基礎 Probability Theory

基礎知識

概率的定義 Axioms of Probability:

• Sample space Ω$\mathrm{\Omega }$ : The set of all the outcomes of a random experiment.
• Set of events(or event space) $\mathcal{F}$ : where each event A∈$A\in \mathcal{F}$ is a set containing zero or more outcomes (i.e., A⊆Ω$A\subseteq \mathrm{\Omega }$ is a collection of possible outcomes of an experiment).
• Probability measure: A function P:⟶ℝ$P:\mathcal{F}⟶\mathbb{R}$ that satisfies the following propeties:
  
  • P(A)≥0$P(A)\ge 0$ for all A∈$A\in \mathcal{F}$
  • P(Ω)=1$P(\mathrm{\Omega })=1$
  • If A1,...,Ak$A_1,...,A_k$ are disjoint events (i.e., Ai∩Aj=∅$A_i\cap A_j=\mathrm{\varnothing }$ whereever i≠j$i\ne j$ ), then P(∪iAi)=∑iP(Ai)$P({\cup }_i{A}_i)={\displaystyle \sum _{i}P(A_i)}$

理解：以擲骰子爲例，Ω$\mathrm{\Omega }$ 就是所有的可能出現的點數集{1, 2, 3, 4, 5, 6}，$\mathcal{F}$ 包括了各種事件，比如{1, 2, 3, 4}, {奇數點數}，{偶數點數}等等。

概率的基本屬性:

1. If A⊆B⟹P(A)≤P(B)$\text{If }A\subseteq B⟹P(A)\le P(B)$
2. P(A∩B)≤min(P(A),P(B))$P(A\cap B)\le min(P(A),P(B))$
3. P(A∪B)≤P(A)+P(B)$P(A\cup B)\le P(A)+P(B)$
4. P(Ω∖A)=P(A⎯⎯⎯⎯)=1−P(A)$P(\mathrm{\Omega }\mathrm{\setminus }A)=P(\overline{A})=1-P(A)$
5. If A1,...,Ak$A_1,...,A_k$ are a set of disjoint events such that ∪ki=1Ai=Ω${\cup }_{i=1}^k{A}_i=\mathrm{\Omega }$ , then ∑ki=1P(Ak)=1$\sum _{i=1}^{k}P(A_k)=1$ .

條件概率：

P(A|B)=P(A∩B)P(B)$$P(A|B)=\frac{P(A\cap B)}{P(B)}$$

P(A|B)$P(A|B)$ is the probability measure of the event A after observing the occurrence of event B. Two events are independent if and only if P(A∩B)=P(A)P(B)$P(A\cap B)=P(A)P(B)$ or P(A|B)=P(A)$P(A|B)=P(A)$ .

隨機變量

扔10次硬幣，出現的所有10個正反面(heads and tails)組合(考慮先後順序)便是樣本空間 Ω$\mathrm{\Omega }$ ，比如ω0=<H,H,T,H,T,H,H,T,T,T>∈Ω${\omega }_0=<H,H,T,H,T,H,H,T,T,T>\in \mathrm{\Omega }$ . 實際問題中，我們往往不關心出現某個特定正反面序列的概率，而更關心real-valued functions of outcomes，比如十次中出現正面的次數，或者連續反面的最長長度，這些函數便是random variables隨機變量。所以隨機變量是一個函數！
Random variable X$X$ is a function X:Ω⟶ℝ$X:\mathrm{\Omega }⟶\mathbb{R}$

• Discrete random variable: P(X=k):=P({ω:X(ω)=k})$P(X=k):=P(\{\omega :X(\omega )=k\})$

• Continuous random variable: P(a≤X≤b):=P({ω:a≤X(ω)≤b})$P(a\le X\le b):=P(\{\omega :a\le X(\omega )\le b\})$

CDFs, PDFs, PMFs

1.Cumulative distribution function (CDF) is a function FX:ℝ⟶[0,1]$F_X:\mathbb{R}⟶[0,1]$ such that

FX(x)≜P(X≤x)$$F_X(x)\triangleq P(X\le x)$$

By using this function one can calculate the probability of any event in $\mathcal{F}$.
Properties:

• 0≤FX(x)≤1$0\le F_X(x)\le 1$ .
• limx→−∞FX(x)=0${lim}_{x\to -\mathrm{\infty }}{F}_X(x)=0$ .
• limx→∞FX(x)=1${lim}_{x\to \mathrm{\infty }}{F}_X(x)=1$ .
• x≤y⟹FX(x)≤FX(y)$x\le y⟹F_X(x)\le F_X(y)$ .

2.Probability mass function (PMF) is a function pX:Ω⟶ℝ$p_X:\mathrm{\Omega }⟶\mathbb{R}$ such that

pX(x)≜P(X=x)$$p_X(x)\triangleq P(X=x)$$

Properties:

• 0≤pX(x)≤1$0\le p_X(x)\le 1$ .
• ∑x∈Val(X)pX(x)=1$\sum _{x\in Val(X)}{p}_X(x)=1$ , Val(X)$Val(X)$ is the set of all possible values X$X$ may assume.
• ∑x∈ApX(x)=P(X∈A)?$\sum _{x\in A}{p}_X(x)=P(X\in A)?$ .

3.Probability density functions (PDF) is the derivative of the CDF:

fX(x)≜dFX(x)dx$$f_X(x)\triangleq \frac{d{F}_X(x)}{dx}$$

PDF for a continuous random variable may not always exist and for very small Δx$\mathrm{\Delta }x$ ,

P(x≤X≤x+Δx)≈fX(x)Δx$$P(x\le X\le x+\mathrm{\Delta }x)\approx f_X(x)\mathrm{\Delta }x$$

The value of PDF at any given point x$x$ is not the probability of that event, i.e, fX(x)≠P(X=x)$f_X(x)\ne P(X=x)$ and fX(x)$f_X(x)$ can take on values larger than one.
Properties:

• fX(x)≥0$f_X(x)\ge 0$ .
• ∫∞−∞fX(x)dx=1${\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_X(x)dx=1$ .
• ∫x∈AfX(x)dx=P(X∈A)${\int }_{x\in A}{f}_X(x)dx=P(X\in A)$ .

Expectation 期望

X$X$ is a discrete random variable with PMF pX(x)$p_X(x)$ and g:ℝ⟶ℝ$g:\mathbb{R}⟶\mathbb{R}$ is an arbitrary function. In this case, g(X)$g(X)$ can be considered a random variable, and we define the expectation or expected value of g(X)$g(X)$ as

E[g(X)]≜∑x∈Val(X)g(x)pX(x)$$E[g(X)]\triangleq {\displaystyle \sum _{x\in Val(X)}g(x)p_X(x)}$$

If X$X$ is a continuous random variable with PDF fX(x)$f_X(x)$ , then the expected value of g(X)$g(X)$ is defined as,

E[g(X)]≜∫∞−∞g(x)fX(x)dx$$E[g(X)]\triangleq {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x)f_X(x)dx$$

The expectation of g(x)$g(x)$ can be thought of as a “weighted average” of the values that g(x) can taken on for different values of x$x$ , where the weights are given by pX(x)$p_X(x)$ or fX(x)$f_X(x)$ . E[X]$E[X]$ is the mean of random variable X$X$ .
Properties:

• E[a]=a$E[a]=a$ for any constant a∈ℝ$a\in \mathbb{R}$ .
• E[af(X)]=aE[f(X)]$E[af(X)]=aE[f(X)]$ for any constant a∈ℝ$a\in \mathbb{R}$ .
• E[f(X)+g(X)]=E[f(X)]+E[g(X)]$E[f(X)+g(X)]=E[f(X)]+E[g(X)]$ .
• For a discrete random variable X$X$ , E[1{X=k}]=P(X=k)$E[1\{X=k\}]=P(X=k)$ .

Variance 方差

The variance of a random variable X$X$ is a measure of how concentrated the distribution of a random variable X$X$ is around its mean:

Var[X]≜E[(X−E[X])2]$$Var[X]\triangleq E[(X-E[X]{)}^2]$$

An alternate expression:

E[(X−E[X])2]=E[X2]−E[X]2$$E[(X-E[X]{)}^2]=E[X^2]-E[X{]}^2$$

Properties:

• Var[a]=0$Var[a]=0$ for any constant a∈ℝ$a\in \mathbb{R}$ .
• Var[af(X)]=a2Var[f(X)]$Var[af(X)]=a^{2}Var[f(X)]$ for an constant a∈ℝ$a\in \mathbb{R}$ .

Some common random variables

Discrete random variables，以扔一次硬幣正面朝上的概率爲p$p$ 爲例:

• 伯努利分佈 X∽Bernoulli(p)$X\backsim Bernoulli(p)$ (where 0≤p≤1$0\le p\le 1$ ):
  
  p(x)={p1−pif x=1if x=0$$p(x)=\{\begin{array}{ll}p& \text{if }x=1\\ 1-p& \text{if }x=0\end{array}$$
• 二項式分佈 X∽Binomial(n,p)$X\backsim Binomial(n,p)$ (where 0≤p≤1$0\le p\le 1$ )：投擲n$n$ 次，正面朝上的次數，
  
  p(x)=(nx)px(1−p)n−x$$p(x)=(\genfrac{}{}{0ex}{}{n}{x})p^x(1-p{)}^{n-x}$$
• 幾何分佈 X∽Geometric(p)$X\backsim Geometric(p)$ (where p>0$p>0$ ): 投擲幾次第一次出現正面朝上,
  
  p(x)=p(1−p)x−1$$p(x)=p(1-p{)}^{x-1}$$
• 泊松分佈 X∽Poisson(λ)$X\backsim Poisson(\lambda )$ (where λ>0$\lambda >0$ )：a probability distribution over the nonnegative integers used for modeling the frequency of rare events,
  
  p(x)=e−λλxx!$$p(x)=e^{-\lambda }\frac{{\lambda }^x}{x!}$$
  
  Continuous random variable:
• 均勻分佈 X∽Uniform(a,b)$X\backsim Uniform(a,b)$ (where a<b$a<b$ ): 在a$a$ 和b$b$ 之間均等概率，
  
  f(x)={1b−a0if a≤x≤botherwise $$f(x)=\{\begin{array}{ll}\frac{1}{b-a}& \text{if }a\le x\le b\\ 0& \text{otherwise }\end{array}$$
• 指數分佈 X∽Exponential(λ)$X\backsim Exponential(\lambda )$ (where λ>0$\lambda >0$ )：概率密度隨着x$x$ 增加減弱，
  
  f(x)={λe−λx0if x≥0otherwise $$f(x)=\{\begin{array}{ll}\lambda e^{-\lambda x}& \text{if }x\ge 0\\ 0& \text{otherwise }\end{array}$$
• 正態（高斯）分佈X∽Normal(μ,σ2)$X\backsim Normal(\mu ,{\sigma }^2)$ (also known as the Gaussian distribution):
  
  f(x)=12π‾‾‾√σe−12σ2(x−μ)2$$f(x)=\frac{1}{\sqrt{2\pi }\sigma }{e}^{-\frac{1}{2{\sigma }^2}(x-\mu {)}^2}$$

PDF and CDF of a couple of random variables：

Summary of some of the properties of these distributions：

雙隨機變量

Joint and marginal distributions 聯合和邊際分佈

Joint cumulative distribution function 聯合累積分佈函數 of X$X$ and Y$Y$ :

FXY(x,y)=P(X≤x,Y≤y)$$F_{XY}(x,y)=P(X\le x,Y\le y)$$

Properties:

• 0≤FXY(x,y)≤1$0\le F_{XY}(x,y)\le 1$
• limx,y→∞FXY(x,y)=1${lim}_{x,y\to \mathrm{\infty }}{F}_{XY}(x,y)=1$
• limx,y→−∞FXY(x,y)=0${lim}_{x,y\to -\mathrm{\infty }}{F}_{XY}(x,y)=0$
• FX(x)=limy→∞FXY(x,y)$F_X(x)={lim}_{y\to \mathrm{\infty }}{F}_{XY}(x,y)$
• FY(y)=limx→∞FXY(x,y)$F_Y(y)={lim}_{x\to \mathrm{\infty }}{F}_{XY}(x,y)$

FX(x)$F_X(x)$ and FY(y)$F_Y(y)$ are the marginal cumulative distribution functions 邊際累積分佈函數 of FXY(x,y)$F_{XY}(x,y)$ .

Joint and marginal probability mass functions

Joint probability mass function pXY$p_{XY}$ : ℝ$\mathbb{R}$ x ℝ⟶[0,1]$\mathbb{R}⟶[0,1]$ :

pXY(x,y)=P(X=x,Y=y)$$p_{XY}(x,y)=P(X=x,Y=y)$$

Properties:

• 0≤pXY(x,y)≤1$0\le p_{XY}(x,y)\le 1$
• ∑x∈Val(X)∑y∈Val(Y)pXY(x,y)=1$\sum _{x\in Val(X)}\sum _{y\in Val(Y)}{p}_{XY}(x,y)=1$
• pX(x)=∑ypXY(x,y)$p_X(x)=\sum _y{p}_{XY}(x,y)$ => marginalization
• pY(y)=∑xpXY(x,y)$p_Y(y)=\sum _x{p}_{XY}(x,y)$

pX(x)$p_X(x)$ is the marginal probability mass function 邊際概率質量函數 of X$X$ .

Joint and marginal probability density functions

Joint probability density function:

fXY(x,y)=∂2FXY(x,y)∂x∂y$$f_{XY}(x,y)=\frac{{\mathrm{\partial }}^2{F}_{XY}(x,y)}{\mathrm{\partial }x\mathrm{\partial }y}$$

Properties:

• fXY(x,y)≠P(X=x,Y=y)$f_{XY}(x,y)\ne P(X=x,Y=y)$
• ∬x∈AfXY(x,y)dxdy=P((X,Y)∈A)${\iint }_{x\in A}{f}_{XY}(x,y)dxdy=P((X,Y)\in A)$
• ∫∞−∞∫∞−∞fXY(x,y)=1${\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{XY}(x,y)=1$
• fX(x)=∫∞−∞fXY(x,y)dy$f_X(x)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{XY}(x,y)dy$
• fY(y)=∫∞−∞fXY(x,y)dx$f_Y(y)={\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{XY}(x,y)dx$

fX(x)$f_X(x)$ is the marginal probability density function 邊際概率密度函數 of X$X$ .

Conditional distributions and Bayes’s rule

The conditional probability mass function of Y given X, assuming that pX(x)≠0$p_X(x)\ne 0$ :

pY|X(y|x)=pXY(x,y)pX(x)$$p_{Y|X}(y|x)=\frac{p_{XY}(x,y)}{p_X(x)}$$

The conditional probability density of Y given X, assuming that fX(x)≠0$f_X(x)\ne 0$ :

fY|X(y|x)=fXY(x,y)fX(x)$$f_{Y|X}(y|x)=\frac{f_{XY}(x,y)}{f_X(x)}$$

Bayes’s rule: derive expression for the conditional probability of one variable given another. 詳見貝葉斯決策理論
Discrete random variables X$X$ and Y$Y$ :

PY|X(y|x)=PXY(x,y)PX(x)=PX|Y(x|y)PY(y)∑y′∈Val(Y)PX|Y(x|y′)PY(y′)$$P_{Y|X}(y|x)=\frac{P_{XY}(x,y)}{P_X(x)}=\frac{P_{X|Y}(x|y)P_Y(y)}{\sum _{y^{\prime }\in Val(Y)}{P}_{X|Y}(x|y^{\prime })P_Y(y^{\prime })}$$

Continuous random variables X$X$ and Y$Y$ :

fY|X(y|x)=fXY(x,y)fX(x)=fX|Y(x|y)fY(y)∫∞−∞fX|Y(x|y′)fY(y′)dy′$$f_{Y|X}(y|x)=\frac{f_{XY}(x,y)}{f_X(x)}=\frac{f_{X|Y}(x|y)f_Y(y)}{{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{f}_{X|Y}(x|y^{\prime })f_Y(y^{\prime })d{y}^{\prime }}$$

Independence 獨立

Two random variables X and Y are independent:

• If FXY(x,y)=FX(x)FY(y)$F_{XY}(x,y)=F_X(x)F_Y(y)$ for all values of x$x$ and y$y$ .
• For discrete random variables:
  
  • pXY(x,y)=pX(x)pY(y)$p_{XY}(x,y)=p_X(x)p_Y(y)$ for all x∈Val(X),y∈Val(Y)$x\in Val(X),y\in Val(Y)$
  • pY|X(y|x)=pY(y)$p_{Y|X}(y|x)=p_Y(y)$ whenever pX(x)≠0$p_X(x)\ne 0$ for all y∈Val(Y)$y\in Val(Y)$
• For continuous random variables
  
  • fXY(x,y)=fX(x)fY(y)$f_{XY}(x,y)=f_X(x)f_Y(y)$ for all x,y∈ℝ$x,y\in \mathbb{R}$
  • fY|X(y|x)=fY(y)$f_{Y|X}(y|x)=f_Y(y)$ whenever fX(x)≠0$f_X(x)\ne 0$ for all y∈ℝ$y\in \mathbb{R}$

If X$X$ and Y$Y$ are independent then for any subset A,B⊆ℝ$A,B\subseteq \mathbb{R}$ :

P(X∈A,Y∈B)=P(X∈A)P(Y∈B)$$P(X\in A,Y\in B)=P(X\in A)P(Y\in B)$$

If X$X$ is independent of Y$Y$ then any function of X is indepedent of any funciton of Y$Y$ .

Expectation and covariance 期望和協方差

Two discrete variables X$X$ , Y$Y$ and g:ℝ2⟶ℝ$g:{\mathbb{R}}^2⟶\mathbb{R}$ is a function on them:

E[g(X,Y)]≜∑x∈Val(X)∑y∈Val(Y)g(x,y)pXY(x,y)$$E[g(X,Y)]\triangleq {\displaystyle \sum _{x\in Val(X)}{\displaystyle \sum _{y\in Val(Y)}g(x,y)p_{XY}(x,y)}}$$

For continuous random variables X$X$ , Y$Y$ , the analogous expression is

E[g(X,Y)]≜∫∞−∞∫∞−∞g(x,y)fXY(x,y)dxdy$$E[g(X,Y)]\triangleq {\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}{\displaystyle {\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}g(x,y)f_{XY}(x,y)dxdy}}$$

The relationship of two random variables with each other: covariance of two variables X$X$ and Y$Y$ is defined as:

Cov[X,Y]≜E[(X−E[X])(Y−E[Y])]$$Cov[X,Y]\triangleq E[(X-E[X])(Y-E[Y])]$$

we can rewrite this as:

Cov[X,Y]=E[(X−E[X])(Y−E[Y])]=E[XY−XE[Y]−YE[X]+E[X]E[Y]]=E[XY]−E[X]E[Y]−E[Y]E[X]+E[X]E[Y]=E[XY]−E[X]E[Y]$$\begin{array}{rl}Cov[X,Y]& =E[(X-E[X])(Y-E[Y])]\\ & =E[XY-XE[Y]-YE[X]+E[X]E[Y]]\\ & =E[XY]-E[X]E[Y]-E[Y]E[X]+E[X]E[Y]\\ & =E[XY]-E[X]E[Y]\end{array}$$

Properties:

• (Linearity of expectation) E[f(X,Y)+g(X,Y)]=E[f(X,Y)]+E[g(X,Y)]$E[f(X,Y)+g(X,Y)]=E[f(X,Y)]+E[g(X,Y)]$
• Var[X+Y]=Var[X]+Var[Y]+2Cov[X,Y]$Var[X+Y]=Var[X]+Var[Y]+2Cov[X,Y]$
• If X$X$ and Y$Y$ are independent, then Cov[X,Y]=0$Cov[X,Y]=0$
• If X$X$ and Y$Y$ are independent, then E[f(X)g(Y)]=E[f(X)]E[g(Y)]$E[f(X)g(Y)]=E[f(X)]E[g(Y)]$
• Cov[X,Y]=0$Cov[X,Y]=0$ , we say that X$X$ and Y$Y$ are uncorrelated 不相關. This is not the same thing as stating that X$X$ and Y$Y$ are uncorrelated. For example, if X∽Uniform(−1,1)$X\backsim Uniform(-1,1)$ and Y=X2$Y=X^2$ , then one can show that X$X$ and Y$Y$ are uncorrelated, even though they are not independent.
• 隨機變量的 不相關 和 獨立 在定義上就是不等價的。獨立是不相關的充分不必要條件，即獨立可以推出不相關，反之不行。獨立就是兩個隨機變量相互獨立，等價於p(x,y)=p(x)p(y)$p(x,y)=p(x)p(y)$ 。隨機變量uncorrelated的定義就是協方差爲0。