(Credit: © Johan Jarnestad/The Royal Swedish Academy of Sciences)
On Tuesday, the Nobel Prize in physics was awarded to John Hopfield and Geoffrey E. Hinton for their foundational discoveries and inventions that have enabled artificial neural networks to be used for machine learning—a widely used form of AI. The award highlights how the field of physics is intertwined with neural networks and the field of AI.
An artificial neural network is a collection of nodes that connect in a way inspired by neurons firing in a living brain. The connections allow a network to store and manipulate information. While neural network research is closely tied to the fields of neuroscience and computer science, it is also connected with physics. Ideas and tools from physics were integral in the development of neural networks for machine learning tasks. And once machine learning was refined into a powerful tool, physicists across the world, including at the University of Maryland, have been deploying it in diverse research efforts.
Hopfield invented a neural network, called the Hopfield network, that can work as a memory that stores patterns in data—this can be used for tasks like recognizing patterns in images. Each node in its network can be described as a pixel in an image, and it can be used to find an image in its memory from its training that most closely resembles a new image that it is presented to it. But the process used to compare images can also be described in terms of the physics that govern the quantum property of spin.
The spin of a quantum particle makes it behave like a tiny bar magnet, and when many magnets are near each other, they all work to orient themselves in specific ways (north poles repelled by the other north poles and attracted to south poles). Physicists can characterize a group of spins based on their interactions and the energy associated with the orientation all the spins want to fall into. Similarly, a Hopfield network can be described as characterizing images based on an energy defined by the connections between nodes.
Hinton used the tools of statistical physics to build on Hopfield’s work. He developed an approach to using neural networks called Boltzmann machines. The learning method of these neural networks fit the description of a specific type of spin orientation found in materials, called a spin glass. A Boltzmann machine can be used to identify characteristics of the data it was trained with. These networks can help classify images or create new elements that fit the pattern it has been trained to recognize.
Following the initial work of Hopefield and Hinton a broad variety of neural networks and machine learning applications have arisen. As neural networks have expanded beyond the forms developed by Hopfield and Hinton, they still resemble common physics models, and some physicists have chosen to apply their skills to understanding the large, often messy, models that describe neural networks.
“Artificial neural networks represent a very complicated, many-body problem, and physicists, especially condensed matter physicists, that's what we do,” says JQI Fellow Maissam Barkeshli, a theoretical physicist who has applied his expertise to studying artificial neural networks. “We study complex systems, and then we try to tease out interesting, qualitatively robust behavior. So neural network research is really within the purview of physicists.”
In a paper that Barkeshli and UMD graduate student Dayal Kalra shared at the Conference on Neural Information Processing Systems last year, the pair presented the results of their investigation of the impact of the learning rate—the size of steps that are made each time the network’s parameters are changed during training—on the optimization of a neural network. Their analysis predicted distinct behaviors for the neural network depending on the learning rate used.
“Our work focused on documenting and explaining some intriguing phenomena that we observed as we tuned the parameters of the training algorithm of a neural network,”Barkeshli says. “It is important to understand these phenomena because they deeply affect the ability of neural networks to learn complicated patterns in the data.”
Other similar questions in neural network research remain, including how the information is encoded in a network, and Barkeshli says that many of the open questions are likely to benefit from the perspectives and tools of physicists.
In addition to the field of machine learning benefitting from the tools of physics, it has also provided valuable tools for physicists to use in their research. Similar to the diverse uses of AI to play board games, create quirky images and write emails, the applications of machine learning have taken many forms in physics research.
“The laureates’ work has already been of the greatest benefit,” says Ellen Moons, Chair of the Nobel Committee for Physics. “In physics we use artificial neural networks in a vast range of areas, such as developing new materials with specific properties.”
Neural networks can also be useful for physicists who need to identify relevant data so they can focus their attention effectively. For instance, the scientific background document that was shared as part of the prize announcement cites the use of neural networks in the analysis of data from the IceCube neutrino detector at the South Pole. The project was a collaboration of many researchers, including UMD physics professors Kara Hoffman and Gregory Sullivan and UMD physics assistant professor Brian Clark, that produced the first neutrino image of the Milky Way.
Neutrinos are a group of subatomic particles that are notoriously difficult to detect since they have little mass and no electrical charge, which makes interactions uncommon (only about one out of 10 billion neutrinos is expected to interact as it travels all the way through the earth). The lack of interactions means neutrinos can be used to observe parts of the universe where light and other signals have been blocked or deflected, but it also means researchers have to work hard to observe them. The IceCube detector includes a cubic kilometer of ice that neutrinos can interact with. When an interaction is observed, researchers must determine if it was actually an interaction with a neutrino interaction or another particle. They must also determine the direction the detected neutrino came from and whether it likely originated from a distant source or if it was produced by particle interactions that occurred in the earth’s atmosphere.
UMD postdoctoral researcher Stephen Sclafani worked on the project as a graduate student at Drexel University. He was an author of the paper sharing the results and was a lead in the project’s use of machine learning in their analysis. Neural networks helped Sclafani and his colleagues select the desired neutron interactions observed by the detector from data that had been collected over ten years. The approach increased their efficiency at identifying relevant events and provided them with approximately 30 times the amount of data to use in generating their neutrino map of our galaxy.
“Initially, as many people were, I was surprised by this year's prize selection,” says Sclafani. “But neural networks are currently revolutionizing the way we do physics. The breakthroughs of Hopfield and Hinton are responsible for many other results and are grounded in statistical physics.”
Some physics research applications of neural networks draw more directly on the physics foundation of neural networks. Earlier this year, JQI Fellow Charles Clark and JQI graduate student Ruizhi Pan proposed new tools to expand the use of machine learning in quantum physics research. They investigated a type of neural network called a restricted Boltzmann machine (RBM)—a variation of Boltzmann machines with additional restrictions on the networks. Their research returned to the spin description of the network and investigated how well various numbers of nodes can do at approximating the state that results from the spin interactions of many quantum particles.
"We, and many others, thought that the RBM framework, applied by 2024 Nobel Physics Laureate Geoffrey Hinton to fast learning algorithms about 20 years ago, might offer advantages for solving problems of quantum spin systems," says JQI Fellow Charles Clark. "This proved to be the case, and the research on quantum models using the RBM framework is an example of how advances in mathematics can lead to unanticipated developments in the understanding of physics, as was the case for calculus, linear algebra, and the theory of Hilbert spaces."
There are numerous additional ways that neural networks are also being developed into valuable tools for physics research, and this year’s Nobel Prize in physics celebrates that contribution as well as its roots in the field.
For more information about the prize winners and their research that the award recognizes, see the press releases from the Royal Swedish Academy of Sciences.
Story by Bailey Bedford