Full text
Derivation of Shannon’s Entropy for Constraints Sum over i g(ei)P(ei) with Probability P(ei) Part 2 Francesco R. Ruggeri Hanwell, N.B. Dec. 27, 2025 In Part 1, we argued that if one has a conserved quantity g(ei), as given in a constraint Sum over i g(ei) P(ei), then one may solve for the probability distribution P(ei) = P1(g(ei)) by considering a subset of P1(g(ei))s such that g(ei)+g(ej) = E1. Within this subset, one has a uniform distribution for P1(g(ei))P1(g(ej))=P1(g(ei)+g(ej)) with P1 unnormalized and this leads directly to P1(g(ei)) = C exp(-g(ei)/T), i.e. the Maxwell-Boltzmann form. A uniform distribution implies a minimal amount of bias in keeping with the conservation of g(ei) which is equivalent to the global constraint Sum over i g(ei) P(ei). We then suggested that the subset-uniform distribution-conservation result of Cexp(-g(ei)/T) implies that ln(P1(g(ei)) = -g(ei)/T. In Part 1, we argued that this allows one to create a math function F(P1(g(ei)) which when maximized subject to the constraint Sum over i g(ei) P1(g(ei)) yields Cexp(-g(ei)/T). The idea of maximizing is justified, we argue because one already deals with a minimal amount of bias in the subset-conservation approach and so this minimal bias must carry over to the full P1(g(ei)) = P(ei) set subject to the constraint Sum over i g(ei) P1(g(ei). This led to the derivation of Shannon’s entropy (equivalent to F) in Part 1 as a math function. Physically, it is known that if one has P(ei) and N particles, then NP(ei) = n(ei). As a result, one may physically permute the N particles creating B= N!/ Product over i n(ei)! distinct arrangements. Now, there may be various different { n(ei) } sets which respect the constraint Sum over i g(ei) P1(g(ei)) = number, but yields different values for B. One wishes to have the maximum value of B as this represents minimal bias. Thus, we suggest that Shanon’s entropy should be linked with a physical scenario. The P(ei) = C exp(-g(ei)/T) solution is an approximate one as P(ei)P(ej) = P(ei+ej) may include ei+ej > E total, as is well-known. Given the uniform distribution from the subset-uniform distribution-conservation approach for which p(ei)p(ej)=p(ei+ej) ((1)), this result yields values for n(ei), n(ej) and n(ei+ej). Such a result may be compared with approximations of ln(n(ei))! (as one knows that approximations are being used). One finds that the common Stirling approximation is implied by ((1)) so that the Shannon’s entropy expression is equivalent to B in the Stirling approximation. Thus, Shannon’s entropy is linked to a physical global feature and is not simply a mathematical expression. Subset-Uniform DistributionConservation Approach In Part 1, we argued that if one has a probability distribution P(ei) subject to a constraint: Sum over i g(ei)P(ei), one may consider g(ei) to be a conserved quantity and solve the problem using the notion of a uniform distribution for P1(g(ei))P1(g(ej))= P1(g(ei)+g(ej)) ((2)) for P1 unnormalized, with P1(g(ei)) = P1(ei). This leads to the solution P(ei) = P1(g(ei)) = C exp(-g(ei)T) ((3))
No notion of Shannon’s entropy is needed. ((3)), however, required that one work within a subset of P(ei)’s i..e g(ei)+g(ej) = E1 and one may rightly argue that there should exist an approach which applies globally, i.e. not within this subset or some other. This is why we introduced a global function F(P1(g(ei)) in Part 1 and showed that the subset-uniform distribution-conservation approach allows one to explicitly find F such that the maximization of F with respect to n.P1 subject to the constraint Sum over i g(ei) P1(g(ei)) yields the same solution as the subset-uniform distribution-conservation approach. We show the details in the next section. This F is Shannon’s entropy. Now F(P1(g(ei)) is a math function and we ask: Is it linked in any way to a physical scenario? We note that the subset-uniform distribution-conservation is directly linked to physical elastic two body scattering and suggest that F(P1(g(ei)) should also have a physical interpretation which we consider in the next section. The catch is that the subset-uniform distribution-conservation approach is an approximation (as we show in the next section) and so any physical interpretation of F must also be considered in terms of an approximation as well. P(ei)P(ej) = P(ei+ej) for P Unnormalized and Stirling’s Approximation P(ei)P(ej) = P(ei+ej) ((4)) for P(ei) unnormalized is an approximation because it is possible for ei+ej > Etotal for N particles Furthermore, given P(ei), multiplying by N, the number of particles, yields NP(ei)= n(ei) ((5)) In Part 1, we created a math function F(P1(g(ei))) and maximized it subject to P1(g(ei)) subject to the constraint Sum over i g(ei) P1(g(ei)). In order to find F, however, we used the subset-conservation approach which yields: ln(P1(g(ei)) = -g(ei)/T for P1 unnormalized ((6)) This yields F as Shannon’s entropy -Sum over i P(ei) ln(P(ei)). Physically, however, if one has n(ei) numbers then one may calculate the total number of distinct arrangements: N! / Product over i n(ei)! ((7)) It is possible to have different { n(ei) } sets satisfy: Sum over i g(ei) P(ei) = number where n(ei) = NP(ei). One wishes to have the set which maximizes ((7)), in order to minimize bias. The subset-conservation approach should not only yield a math expression for Shannon’s entropy, as it did in Part 1, but should also provide some insight into n(ei)! In ((7)) as an approximation, as Cexp(-g(ei)/T) is an approximation itself. We try to find out what the subset-conservation approximate approach implies as an approximation for n(ei)!.
One may compare the subset-uniform distribution-conservation approach to a well-known approximation for factorials, namely Stirling’s approximation to see if the two match or not. Stirling’s approximation is: ln(n!) approx= n ln(n) ((8)) We showed in Part 1, that: d F(P1(g(ei))/ dP1 = -g(ei)/T (for maximization) = ln(P1(g(ei)) + C from the subset approach ((9)) The subset-conservation approach is consistent with Stirling’s approximation and maximization of ((7)) subject to the global constraint Sum over i g(ei) n(ei) = number2 which maps to the global constraint Sum over i g(ei) P(ei)=number, showing that one may maximize ((7)) with respect to the global constraint, but only under the Stirling approximation in order to have a result which is consistent with the approximation implied by the subset-uniform distributionconservation approach. As a result, Shannon’s entropy is not simply a math function, but under the Stirling’s approximation, which is consistent with the subset-uniform distribution-conservation approximation, it also represents the physical number of arrangements of N identical particles with n(ei) having ei. Since one is only interested in ei, one considers each particle within n(ei) as identical in terms of permutations, i.e. ((7)). If one did not know Stirling’s approximation, then one would find that the approximation consistent with the subset-uniform distribution-conservation approach was indeed the Stirling’s approximation form. Conclusion In conclusion, we showed in Part 1, that given a constraint Sum over i g(ei) P(ei), one may consider a subset of all P(ei)=P1(g(ei))’s such that g(ei)+g(ej) = E1. Here g(ei) is considered to be a conserved quantity and one may consider physical 2-body elastic scattering. This leads to P1(g(ei)) P1(g(ej)) = P1(g(ei)+g(ej)) or P1 = Cexp(-g(ei)/T) as shown in Part 1. We then argued that if one does not wish to solve the problem in the subset space of P1(g(ei))s, one should be able to solve it globally, i.e. involving all P1(g(ei))s. This led to the creation of a function F(P1(g(ei)) such that when maximized subject to the constraint Sum over i g(ei) P1(g(ei)) (where P(ei) = P1(g(ei)), one obtains the subset-uniform distribution-conservation result. In other words, the subset approach fixes the form of F which turns out to be Shannon’s entropy. In this note, we argue that Shannon’s entropy should be linked to some physical scenario. We note that the subset approach yields P1(g(ei)) = P1(ei) and that n(ei) =N P(ei), where N is the total number of particles. We also note that the subset -uniform distribution -conservation result P1(g(ei)) P1(g(ej) = P1(g(ei)+g(ej)) is an approximation because g(ei)+g(ej) may be larger than a total value which holds for the N particles. The form exp(-g(ei)/T) drops quickly and so these larger values may still be used in an approximation as is well known. Thus, any physical interpretation of F must be considered in light of approximations as well.
We point that there exists a physical situation linked with bias, namely the permutation of N particles with n(ei) being the number with ei. Given that one is only interested in the value ei, one has B= N!/ Product over i n(ei)! as the number of distinct permutations. It is possible to have different sets of n(ei) satisfy the constraint, but yield different B values. One wants the largest B values to have the minimum bias. We show that the Stirling approximation, and maximization fo B subject to the constraint sum over i g(ei) P(ei) is equivalent to the subset-uniform distribution -conservation approach. If one did not know Stirling’s approximation, the subset -uniform distribution -conservation would have shown that ln(n!) = nln(n) is the approximation consistent with F, i.e. Shannon’s entropy found in Part 1 and the factorial expression, N!/Product n(ei)!.
