Differential Entropy of Multivariate Distributions

The following computations are set forth because of their relevance to recent advanced multi-antenna techniques [1]. The presentation follows that given by Cover and Thomas [2]. Details on multivariate distributions can be found in Hogg and Craig [3] and in detailed computations given below [4, 5]. The joint differential entropy of {n} random variables satisfies

\displaystyle h\left({\bf x}\right) = - \int f\left({\bf x}\right)\log_2 f\left({\bf x}\right)\,d^nx. \ \ \ \ \

The p.d.f. (probability density function) of n jointly normal real random variables is given by

\displaystyle f\left({\bf x}\right)=\frac{1}{\sqrt{\left(2\pi\right)^n\left|{\bf V}\right|}}\exp\left[-\frac{1}{2}\left({\bf x}-{\mu}\right)^T{\bf V}^{-1}\left({\bf x}-\mu\right)\right],\ \ \ \ \

where {{\bf x}} and {\mu} are length n column vectors, {{\bf V}} is the n\times n covariance matrix and {\left|{\bf V}\right|} is its determinant. In this case,

\displaystyle h\left({\bf x}\right) = -\frac{1}{\ln 2}\int f\left({\bf x}\right)\left\{-\frac{1}{2}\ln\left[\left(2\pi\right)^n \left|{\bf V}\right|\,\right]-\frac{1}{2}\left[\left({\bf x}-\mu\right)^T{\bf V}^{-1}\left({\bf x}-\mu\right)\right] \right\}\,d^nx, \ \ \ \ \

\displaystyle = \frac{1}{2}\log_2\left[\left(2\pi\right)^n\left|{\bf V}\right|\,\right]+\frac{1}{2\ln 2}E \left[\left({\bf x}-\mu\right)^T{\bf V}^{-1}\left({\bf x}-\mu\right)\right]. \ \ \ \ \

\displaystyle = \frac{1}{2}\log_2\left[\left(2\pi\right)^n\left|{\bf V}\right|\right]+\frac{1}{2\ln 2}E\left[\sum_{i,j=1}^n \left(x_i-\mu_i\right)\left({\bf V}^{-1}\right)_{ij}\left(x_j-\mu_j\right)\right]. \ \ \ \ \

Since the expectation of a sum is the sum of the expectations h\left({\bf x}\right)

\displaystyle = \frac{1}{2}\log_2\left[\left(2\pi\right)^n\left|{\bf V}\right|\right]+\frac{1}{2\ln 2}\sum_{i,j=1}^n E\left[\left(x_i-\mu_i\right)\left({\bf V}^{-1}\right)_{ij}\left(x_j-\mu_j\right)\right], \ \ \ \ \


\displaystyle h\left({\bf x}\right) = \frac{1}{2}\log_2\left[\left(2\pi\right)^n\left|{\bf V}\right|\right]+\frac{1}{2\ln 2}\sum_{i,j=1}^n \left({\bf V}^{-1}\right)_{ij}E\left[\left(x_i-\mu_i\right)\left(x_j-\mu_j\right)\right]. \ \ \ \ \


\displaystyle E\left[\left({\bf x}-\mu\right)\left({\bf x}^T-{\mu}^T\right)\right]={\bf V},\ \ \ \ \

this becomes

\displaystyle h\left({\bf x}\right) = \frac{1}{2}\log_2\left[\left(2\pi\right)^n\left|{\bf V}\right|\right]+\frac{1}{2\ln 2}\sum_{i,j=1}^n \left({\bf V}^{-1}\right)_{ij}\left({\bf V}\right)_{ij}.\ \ \ \ \

But the covariance and its inverse are necessarily real symmetric matrices, so

\displaystyle h\left({\bf x}\right)= \frac{1}{2}\log_2\left[\left(2\pi\right)^n\left|{\bf V}\right|\,\right]+\frac{1}{2\ln 2}\sum_{j=1}^n\delta_{jj}, \ \ \ \ \

\displaystyle = \frac{1}{2}\log_2\left[\left(2\pi\right)^n\left|{\bf V}\right|\,\right]+\frac{1}{2}\log_2 e^n, \ \ \ \ \

\displaystyle = \frac{1}{2}\log_2\left[\left(2\pi e\right)^n\left|{\bf V}\right|\,\right], \ \ \ \ \

the desired result. Additional related derivations are provided in [4,5] and are available by clicking below.


[1] M. Debbah in A. Sibille, C. Oestges, A. Zanella, (editors) MIMO From Theory to Implementation, Academic Press, Amsterdam (2011), Chap. 1.

[2] T. M. Cover and J. A. Thomas, Elements of Information Theory, Wiley, N.Y. (1991), p. 230.

[3] R. V. Hogg and A.T. Craig, Introduction to Mathematical Statistics, Macmillan, N.Y. (1978), Chap. 12.

[4] H. L. Rappaport, Normal and Bivariate Normal Distributions and Moment-Generating Functions, 7G Communications, 7GCTN03, September (2014). bivar

[5] H. L. Rappaport, Multivariate Distributions and Associated Differential Entropy of Jointly Normal Random Variables, 7G Communications, 7GCTN04, October (2014). multi



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: