91Ó°ÊÓ

Consider a distribution over possible outcomes with probabilities $p_1,p_2,K{p}_n$. Assume that each $p_i$is a power of 2. Consider the long sequence of $m$samples is drawn from the distribution and for all $1≤i≤n$, $i^{th}$the outcome occurs exactly $m{p}_i$times in the sequence.

(a)

Consider the encoded length of the element $p_i$is with the probability $\log \frac{1}{p_i}$. The probability of $\frac{1}{2^k}$is 1. The $c\left(k\right)$is at the level $k$in Huffman tree (assuming that the root is at level 0) and $\frac{1}{2}>\frac{2}{5}$, and for all role="math" localid="1659009731170" $\frac{1}{2^n}<\frac{1}{3},\left(n>1\right)$. Since the outcome is $c\left(\frac{1}{2}\right)$ and the probability of all the elements in the first layer is 1.

Consider the root to be 1, the subtree must be at the first level and this level elements have the probability $\left(\frac{1}{4}\right)$.Then the probability $c\left(k\right)$, results in the Huffman code is reduced to the length role="math" localid="1659010017242" $\underset{i=1}{\overset{n}{∑}}m{p}_i\log \frac{1}{p_i}$.

Therefore, the length of the sequence is role="math" localid="1659010031265" $\underset{i=1}{\overset{n}{∑}}m{p}_i\log \frac{1}{p_i}$.

(b)

$p_i=\frac{1}{n}$ is the largest entropy. $p_k=1,p_{i,\left(i≠k\right)}=0$is the smallest entropy.

Therefore, the largest and the smallest possible entropies are obtained.