Microstate Mathematics in Statistical Mechanics for Neural Networks

Statistical Mechanics for Neural Networks and Deep Learning with Alianna J. Maren, Ph.D. The Mathematics of Microstates: Including Examples

Authors Note: The following material is another accompaniment for the Pr cis for the book-in-progress, Statistical Mechanics for Neural Networks and Deep Learning. For more information, Opt-In at: www.aliannajmaren.com Alianna J. Maren, Ph.D.

Populating a Microstate: The Math A System with Ten Units and Two Energy Levels Step 1: Randomly fill the system with units 8 10 5 1 2 6 7 4 3 9 System with Two Energy Levels (Not showing the energy levels yet) Collection of N units (Here, N = 10) 1. Start with a collection of labeled units, 1..N. (This is the collection-bag on the right.) 2. Randomly select a unit from the collection and put it into the system. 3. Math: Note that we had N different choices for putting the first unit into the system. 4. Randomly select a second unit and put it into the system. 5. Math: Note that we can now select from the N-1 units that were left in the collection. 6. Do this again. 7. Math: Note that we could select from the N-2 units left in the collection. 8. Keep going until we ve used up all of our units.

Number of Ways of Populating the System: The Math The number of ways in which we can put units into the system: 1 N-1 N-2 N * * * * Last selection: 1 possible choice First selection: N possible choices Second selection: N-1 possible choices Third selection: N-2 possible choices The number of ways of populating the system is a FACTORIAL: * ) 1 ( * N N QNum = = ( ) 2 * ... * 1 ! N N

Populating a Microstate: The Math A System with Ten Units and Two Energy Levels Step 2a: Decide on how many units we want in each energy level. For this example: Put three units in level e1, and the remaining seven units in e0. 3 7 10 8 5 Energy level e1 Energy level e0 9 6 1 4 2 System with Two Energy Levels (Not showing the energy levels yet) 3 7 10 8 5 9 6 1 4 2 System with Two Energy Levels (Randomly identifying which units will go to which level) Step 2b: Randomly assign units to one of the two energy levels, proportionate to the numbers of units that we want for each level.

Populating a Microstate: The Math Putting the Units into the Two Levels Step 3: From the units identified for a given energy level, randomly put them into that level in the system. Energy level e1 Energy level e0 3 7 10 8 5 9 6 1 4 2 System with Two Energy Levels (Randomly identifying which units will go to which level) ej = 1 3 8 5 ej = 0 4 7 3 2 1 9 6 System with Two Energy Levels (Now we re showing a random assignment of units per level) For this example: We had randomly assigned three units to level e1, and the remaining seven units to e0.

Populating a Microstate: Populating Levels Putting the Units into the Two Levels Step 3: From the units identified for a given energy level, randomly put them into that level in the system. For this example: We had randomly assigned three units to level e1, and the remaining seven units to e0. For level e1: 1. Three units assigned to this level; M = 3; count these units as 1..M. 2. Randomly select a unit from the collection of M units and put it into level e1. 3. Math: Note that we had M different choices for putting the first unit into level e1. 4. Randomly select a second unit and put it into level e1. 5. Math: Note that we can now select from the M-1 units that were left in the sub- collection of units set aside for level e1. 6. Do this again. 7. Math: Note that we could select from the M-2 units left in the sub-collection. 8. Keep going until we ve used up all of our units. For level e0: 1. Same process, we have N-M units; in this example, N -M = 7.

Number of Ways of Populating the Levels within the System: The Math The number of ways in which we can put units into level e1: 1 M-1 M-2 M * * * * Last selection: 1 possible choice First selection: M possible choices Second selection: M-1 possible choices Third selection: M-2 possible choices = ) 1 = * ( * ( ) 2 * ... * 1 ! QDen M M M M 1 The number of ways in which we can put units into level e0: 1 N-M-1 N-M-2 N-M * * * * Last selection: 1 possible choice First selection: M possible choices Second selection: M-1 possible choices Third selection: M-2 possible choices = ) 1 = ( ) * ( * ( ) 2 * ... * 1 ( )! QDen N M N M N M N M 2

Number of Ways of Populating Summary So Far: The number of ways in which we can put units into the system: QNum= ! N The number of ways in which we can put units into level e1: QDen= ! M 1 The number of ways in which we can put units into level e0: QDen = ( )! N M 2 Next Step: Assemble these terms into a single equation

Number of Ways of Populating Assembling the Equation: The number of ways in which we can put units into the system: Numerator: Total ways of putting units into the system ! Q N = = Num Q ( ! )! Q Q M N M 1 2 Den Den Denominator 1: The number of ways in which we can put units into level e1: Denominator 2: The number of ways in which we can put units into level e0:

Number of Ways of Populating Getting Some Intuition (1): ! Q N = = Num Q ( ! )! Q Q M N M 1 2 Den Den This equation shows how we can put units into the overall system (numerator), divided by the ways in which we can put units into the different energy levels (terms making up the denominator). Recall: When all units are in one level, the total number of microstates = 1. Example 1: System with all units are in the lower energy level. 1. The numerator is the total number of ways in which we can put units into a system. 2. If we had ONLY ONE ENERGY LEVEL, all those units would go into that level. 3. This means just one term in the denominator, or: Denominator = N! * 1. 4. Then, since the ORDER IN WHICH THE UNITS ARE ARRANGED is NOT IMPORTANT, we d have a denominator that is the same as the numerator. 5. Then Q = 1.

Number of Ways of Populating Getting Some Intuition (2): ! Q N = = Num Q ( ! )! Q Q M N M 1 2 Den Den When we had one unit in the upper energy level, and nine units in the lower energy level, we had M=1 and (N-M) = 9. One of ten different configurations like this with one unit in the upper level. Recall: When we had one unit in e1, and the remaining 9 in e0, the total number of microstates = 10 : Ten ways to create this configuration. Example 2: System with one unit in the upper level, remaining in the lower level. 1. The numerator is still the total number of ways in which we can put units into a system. 2. Number of units in the upper level = 1; M=1, and so M!=1. 3. Number of units in the lower level = 9; N-M = 9, and so (N-M)! = 9! 4. Then, since the ORDER IN WHICH THE UNITS ARE ARRANGED is NOT IMPORTANT, we d have a denominator that is M! * (N-M)! = 1*9!.

Number of Ways of Populating Putting More Units on Upper Level: Example 3: System with two units in the upper level, remaining in the lower level. ! 10 ! Q N = = = Num Q ( ! )! !* 8 ! 2 Q Q M N M 1 2 Den Den New Example: Two units on top, eight units on lower level; M = 2; N-M = 8. Decompose the factorial in the numerator, and cancel terms with the factorial in the denominator. 10 ! 10 * 9 ! 8 * 10 * 9 90 = = = = = 45 Q !* 8 ! 2 !* 8 ! 2 2 2 We mathematically get 45 different microstates, which is significantly more than the number of microstates for a given configuration than we ve found before.

Intuitive Observation: As we increase the number of options for putting units on a level, we increase the number of microstates System state n0 = # Units at e0 n1 = # Units at e1 Total number of microstates Due to symmetry, we get the same results when we have a certain number of units on the upper level as we when we have those same units on the lower level. 1 10 0 1 2 9 1 10 3 8 2 45 4 7 3 ? E.g., two units on top gives us 45 microstates; two units on the lower level gives us 45 microstates. 5 6 4 ? 6 5 5 ? 7 4 6 ? 8 3 7 ? Now we need to figure out the remaining possible configurations. 9 2 8 45 10 1 9 10 11 0 10 1

Finishing Up the Possible Configurations: Three units on top, seven units on lower level; M = 3; N-M = 7. 120 microstates 10 ! 10 * 9 * 8 ! 7 * 10 * 9 * 8 10 * 3 * 4 = = = = = 120 Q 7 !* ! 3 7 !* 3 * 2 3 * 2 1 Four units on top, six units on lower level; M = 4; N-M = 6. 210 microstates 10 ! 10 * 9 * 8 * 7 ! 6 * 10 * 9 * 8 * 7 10 * 3 * 7 = = = = = 210 Q 7 !* ! 3 6 !* ! 4 4 * 3 * 2 1 Five units on top, five units on lower level; M = 5; N-M = 5. 252 microstates 10 ! 10 * 9 * 8 * 7 * 6 ! 5 * 10 * 9 * 8 * 7 * 6 2 * 3 * 7 * 6 = = = = = 252 Q 7 !* ! 3 !* 5 ! 5 5 * 4 * 3 * 2 1

Finishing Our Table: We have the maximal number of microstates when units are equally distributed between two energy levels System state n0 = # Units at e0 n1 = # Units at e1 Total number of microstates Symmetric Results, we get the same results when we have a certain number of units on the upper level as we when we have those same units on the lower level. 1 10 0 1 2 9 1 10 3 8 2 45 4 7 3 120 5 6 4 210 Maximal number of microstates when equal numbers in upper and lower energy levels. 6 5 5 252 7 4 6 210 8 3 7 120 9 2 8 45 This will connect with our notion of entropy later on. 10 1 9 10 11 0 10 1

Our Next Step: We now know how to compute the numbers of different microstates for a system. The next step is to compute the most likely probability for a given configuration (i.e., given specific M and (N-M)). To do this, we ll connect the numbers of microstates with the energy level value for the upper state. (Still keeping the energy level for the lower state at 0.) This means that we ll play with the value for e1. This will be the subject of the next slidedeck.