Family portrait of Rényi entropies in the domain of energy

After the introduction of the concept of entropy in Signal Theory by Claude Shannon in 1948 (see the paper A Mathematical Theory of Communication), Alfred Rényi published in 1961 another milestone paper for the flourishing field of Information Theory titled On measures of entropy and information, in which he proposed a parametric family of entropy indices that own most of the properties owned by the Shannon entropy, except the most peculiar one (i.e. the recursivity).

A remarkable feature of the formula found by Rényi is that it allows to express different indices of entropy with a simple variation of a single parameter (alpha): for example, Hartley entropy coincides with Rényi entropy order zero, Shannon entropy is the limit for Rényi entropy when alpha tends to one, Collision entropy is Rényi entropy order two and Min-entropy is the limit for alpha tending to infinite.

All these main entropies belonging to the Rényi family became soon important indices in many scientific fields like ecology, biology, medicine, physics, economics, finance, etc. for indicating the degree of homogeneity of a distribution, in a complementary way to the indices of diversity.

Presentation

In order to study the main entropies of the Rényi family, in this article the shape of a generic probability distribution composed of six probabilities is varied with continuty and the values of its entropies are recorded. In particular, the limit distributions having maximum entropy (with the probabilities distributed as much as possible, except the main one) and those having minimum entropy (with the probabilities compacted as much as possible) are considered during the variation of their shape.

The specific energy is calculated as the complement to one of the specific Collision entropy. This fact reduces the possibility of observation of the features of Collision entropy but, in turn, it allows an excellent point of view for the study of the other two quantities (Shannon entropy and Min-entropy).

Discussion

Observing the diagram of the picture above, it can be seen that the most compact distributions generate limit curves with cusps located near the collision entropy line while the L-shaped distributions, i.e. those having a main probability and all the others equally distributed, generate a broad single-curve arc covering the full range of the diagram.

The area located between these limit cureves contains the state points representing all the possibile probability distributions obtainable with six events: the upper area (in pale orange) refers to state points characterized by the couple of indices specific energy and specific Shannon entropy (epsilon, eta), while the lower area (in pale violet) refers to state points characterized by the couple of indices specific energy and specific Min-entropy (epsilon, mu).

While a generic distribution produces, for each area, exactly a point, the converse is not true: this means that there can be many different distributions that can have the same energy and the same entropy. This interesting fact will be shown more clearly in another article.

For the moment it is worth of notice the fact that this diagram is very useful: in fact it allows the analytical classification of a generic distribution with a precise point on a well delimited area of a Cartesian plane. This approach, typically used for the solution of analytic geometry problems, has never ben proposed before neither in Probability Theory nor in Statistics.

Conclusion

The entropy diagrams contained in the Energy-Entropy plane allow for an improved study of the behavior of dynamical systems and stochastic processes. In fact, the reconstruction of their trajectories in state spaces having progressively increasing dimensions generates probability distributions characterized by Energy-Entropy state points representable on that plane. These state points, that generally move in the direction of an asymptotic state point during the progressive increment of the dimension of the state space, give finally the possibility to characterize the associated dynamical systems and allow for their automatic classification and their automatic comparison in a more detailed way than using a single index.