信息量、信息熵

1. 信息量

爲了理解這個概念，我們先以兩個具體的事件爲例：

• 事件1：德國隊獲得世界盃冠軍。
• 事件2：中國隊獲得世界盃冠軍。

憑直覺來說，信息量和概率是成反比的。具體來說，事件2發生的概率要比事件1小得多，所以事件2的信息量要比事件1大得多。事件E的信息量表達式爲：$I(E)=-log_2(p(E))$。

2. 信息熵

信息熵和信息量只差一個字，它們的區別在哪裏呢？個人理解是說，信息量指代的是單個事件，而信息熵指代的是整個系統（系統內所有事件）。信息熵的表達式爲：$H(X)=−\sum \limits _{i=1} ^{n} p(x_i)log(p(x_i))$。

剛看到信息量和信息熵的時候，不知道你也有類似的疑問，爲什麼要取對數。網上查了半天，也沒有找到答案，最後在香濃的paper(A Mathematical Theory of Communication)找到了解釋：

1. It is practically more useful. Parameters of engineering importance such as time, bandwidth, number of relays, etc., tend to vary linearly with the logarithm of the number of possibilities. For example, adding one relay to a group doubles the number of possible states of the relays. It adds 1 to the base 2 logarithm of this number. Doubling the time roughly squares the number of possible messages, or doubles the logarithm, etc.
2. It is nearer to our intuitive feeling as to the proper measure. This is closely related to (1) since we intuitively measures entities by linear comparison with common standards. One feels, for example, that
   two punched cards should have twice the capacity of one for information storage, and two identical channels twice the capacity of one for transmitting information.
3. It is mathematically more suitable. Many of the limiting operations are simple in terms of the logarithm but would require clumsy restatement in terms of the number of possibilities.

舉個例子，假設現有系統是2個硬幣，每個硬幣往上拋一下，則信息熵爲
$H(x)=−\sum \limits _{i=1} ^{4}\frac{1}{4}log(\frac{1}{4})=2$
假如在現有系統上加入一個硬幣，則信息熵爲
$H(x)=−\sum \limits _{i=1} ^{8}\frac{1}{8}log(\frac{1}{8})=3$
可見信息熵隨着系統規模的線性增加而呈現線性增長。

上述問題指的是相同且正常的硬幣，假設我們使用的是兩個截然不同而且定做的硬幣。假設第一個硬幣正面朝上和朝下的概率分別爲$p_1$、$p_2$，第二個硬幣正面朝上和朝下的概率分別爲$q_1$、$q_2$。
$H(x)=−\sum \limits _{i=1} ^{2} \sum \limits _{j=1} ^{2}p_iq_j log(p_iq_j)$ $H(x)=−\sum \limits _{i=1} ^{2}p_i log(p_i) -−\sum \limits _{j=1} ^{2}q_j log(q_j)$

可見信息熵代表了一個線性系統（線性相加和數乘）。

參考鏈接：https://stats.stackexchange.com/questions/87182/what-is-the-role-of-the-logarithm-in-shannons-entropy
https://blog.csdn.net/tsyccnh/article/details/79163834