Article Source

• On Mutual Information Estimation

On Mutual Information Estimation

• Artem Sobolev, Samsung AI Center Moscow, Research Scientist

Abstract

Mutual Information is an important information-theoretic concept that captures an intuitive idea of the amount of information shared between two random variables. Mutual Information has been used extensively in numerous Machine Learning problems and should be of great interest for every ML researcher. In practice, however, accurately estimating the Mutual Information is a non-trivial task. Recently, it has been shown that many general estimators fail to produce reasonable estimates unless an exponential number of samples is taken. We will discuss this result with its manifestation in several widely used estimators, and then consider new estimators that sidestep the core issue.