Dynamic Neural Networks and Hopfield Network Concepts

Comp 3503 / 5013 Dynamic Neural Networks Daniel L. Silver March, 2014 1

Outline Hopfield Networks Boltzman Machines Mean Field Theory Restricted Boltzman Machines (RBM) 2

Dynamic Neural Networks See handout for image of spider, beer and dog The search for a model or hypothesis can be considered the relaxation of a dynamic system into a state of equilibrium This is the nature of most physical systems Pool of water Air in a room Mathematics is that of thermal-dynamics Quote from John Von Neumann 3

Hopfield Networks See hand out 4

Hopfield Networks Hopfield Network video intro http://www.youtube.com/watch?v=gfPUWwBkXZ Y http://faculty.etsu.edu/knisleyj/neural/ Try these Applets: http://lcn.epfl.ch/tutorial/english/hopfield/html/i ndex.html http://www.cbu.edu/~pong/ai/hopfield/hopfielda pplet.html 5

Hopfield Networks Basics with Geoff Hinton: Introduction to Hopfield Nets http://www.youtube.com/watch?v=YB3-Hn-inHI Storage capacity of Hopfield Nets http://www.youtube.com/watch?v=O1rPQlKQBLQ 6

Hopfield Networks Advanced concepts with Geoff Hinton: Hopfield nets with hidden units http://www.youtube.com/watch?v=bOpddsa4BPI Necker Cube http://www.cs.cf.ac.uk/Dave/JAVA/boltzman/Neck er.html Adding noise to improve search http://www.youtube.com/watch?v=kVgT2Eaa6KA 7

Boltzman Machine - See Handout - http://www.scholarpedia.org/article/Boltzmann_machine Basics with Geoff Hinton Modeling binary data http://www.youtube.com/watch?v=MKdvJst8a6k BM Learning Algorithm http://www.youtube.com/watch?v=QgrFsnHFeig 8

Limitations of BMs BM Learning does not scale well This is due to several factors, the most important being: The time the machine must be run in order to collect equilibrium statistics grows exponentially with the machine's size = number of nodes For each example sample nodes, sample states Connection strengths are more plastic when the units have activation probabilities intermediate between zero and one. Noise causes the weights to follow a random walk until the activities saturate (variance trap). 9

Potential Solutions Use a momentum term as in BP: wij(t+1)=wij(t) + wij+ wij(t-1) Add a penalty term to create sparse coding (encourage shorter encodings for different inputs) Use implementation tricks to do more in memory batches of examples Restrict number of iterations in + and phases Restrict connectivity of network 10

Restricted Boltzman Machine hj pj=1/(1-e- j) Recall = Relaxation j=wijvi j wij vo orho i=wijhj i vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 11

Restricted Boltzman Machine hj pj=1/(1-e- j) Recall = Relaxation j=wijvi j wij vo orho i=wijhj i vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 12

Restricted Boltzman Machine hj pj=1/(1-e- j) Oscar Winner SF/Fantasy j=wijvi Recall = Relaxation j wij vo orho i=wijhj i vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 13

Restricted Boltzman Machine hj pj=1/(1-e- j) Oscar Winner SF/Fantasy j=wijvi Recall = Relaxation j wij vo orho i=wijhj i vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 14

Restricted Boltzman Machine hj pj=1/(1-e- j) Learning = ~ Gradient Descent = Constrastive Divergence j=wijvi j Update hidden units P=P+vihj vo orho i i=wijhj vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 15

Restricted Boltzman Machine hj pj=1/(1-e- j) Learning = ~ Gradient Descent = Constrastive Divergence j=wijvi j Reconstruct visible units vo orho i i=wijhj vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 16

Restricted Boltzman Machine hj pj=1/(1-e- j) Learning = ~ Gradient Descent = Constrastive Divergence j=wijvi j Reupdate hidden units N=N+vihj vo orho i i=wijhj vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 17

Restricted Boltzman Machine hj pj=1/(1-e- j) Learning = ~ Gradient Descent = Constrastive Divergence j=wijvi j Update weights wij=wij + wij vo orho wij=<P>-<N> i=wijhj i vi pi=1/(1-e- i) Oscar Winner SF/Fantasy Source: http://blog.echen.me/2011/07/18/introduction-to-restricted-boltzmann-machines/ 18

Restricted Boltzman Machine RBM Overview: http://blog.echen.me/2011/07/18/introduction- to-restricted-boltzmann-machines/ Wikipedia on DLA and RBM: http://en.wikipedia.org/wiki/Deep_learning RBM Details and Code: http://www.deeplearning.net/tutorial/rbm.html 19

Restricted Boltzman Machine Geoff Hinton on RBMs: RBMs and Constrastive Divergence Algorithm http://www.youtube.com/watch?v=fJjkHAuW0Yk An example of RBM Learning http://www.youtube.com/watch?v=Ivj7jymShN0 RBMs applied to Collaborative Filtering http://www.youtube.com/watch?v=laVC6WFIXjg 20

Additional References Coursera course Neural Networks fro Machine Learning: https://class.coursera.org/neuralnets-2012- 001/lecture ML: Hottest Tech Trend in next 3-5 Years http://www.youtube.com/watch?v=b4zr9Zx5WiE 21