Specializations within Deep Learning
When you first learn about neural networks and deep learning, it may feel reminiscent of when you first learn a new math concept. The utility of these mathematical tools are often not immediately apparent until you start applying them. In this section, we highlight some areas of research that are only possible thanks to neural networks and machine learning.
Deep Reinforcement Learning
Reinforcement learning or RL is a branch of machine learning that is interested decision making. This is a field of research that has long motivated autonomous systems and robotics. What action should a robot take to yield an optimal outcome? If it's trying to maneuver through a poorly lit cave with slippery rocks, how should it articulate all of its joints to stay upright? What path should it take to find the missing person quickly while minimizing risk of getting stuck or falling? These are really hard questions to put into closed-form analytic answers, but deep learning can give us the next best thing.
Specifically, there are ways to solve these decision making problems mathematically. Without going into all the details, there are these mathematical frameworks called Markov Decision Processes or (MDPs) which can be used to exactly compute the goodness or badness of an instantaneous decision (reward), as well as predict its long term effect (value). For example, there may be a decision that seems bad right now (low reward), but it enables a really good outcome (high reward) later in the future, and MDPs provide a way for us to capture these effects mathematically.
The problem is that its only possible to solve MDPs analytically for really simple situations. For example, if the aforementioned cave was perfectly 2D, flat, and discretized into a 10x10 grid of possible positions, and the robot only had to decide between the actions of going forward, backward, left, or right, then it would be possible (albeit slow) to solve for the optimal decision at each grid cell.
Deep learning allows us to break through these computational limitations. In a grossly oversimplified summary, the MDP can be solve at certain grid cells, and neural networks can be regressed over these discrete solutions to create a general solution across a fully continuous environment. This works far better than you may realize, and can actually trigger a recursive training process. The the MDP gets solved at discrete locations, a network is regressed to function which generalizes the solution to any possible location, and then the learned solution can then be reused to resolve the MDP and get a better solution, the network can be regressed again, and the process repeats on and on, ending with a fully general and robust policy (a function which spits out an action when given a state).
The introduction of neural networks as mechanisms to generalize reinforcement learning algorithms is referred to as deep reinforcement learning and it has taken control, robotics, and autonomy communities by storm. The magic lies in the fact that the neural network can learn a function that maps the decision maker's current state to the optimal decision that maximizes the decision maker's long term reward. This highlights how neural networks are truely universal function approximators. Who knew that there exists a function that can select the best decision in complex and uncertain environments, let alone that it could be learned and easy to evaluate in the form of a neural network.
Scientific Machine Learning
Another exciting application of deep learning comes from their ability to learn better models of complex natural phenomena. Think about a really turbulent fluid system, like waves crashing over rocks, or the motion of gas around a helicopter blade. Engineers and physicists alike do not have good models of these systems. We have some partial differential equations that capture first order effects, but they are only approximate, and they take a very long time to solve.
Recently, computer scientists have proposed the use of neural networks to learn better models of, and solutions to, complex differential equations. This constitutes the field of Scientific Machine Learning. If we are able to gather emperical measuremnets of the system, we can use machine learning to regress a function that models the underlying dynamics.
Physics Informed Neural Networks
The simplest instantation of this is a physics-informed neural network or PINN. PINNs are neural networks that take data from some dynamical system (e.g. position and acceleration), and tries to learn a mapping between these two quantities. However, these networks "physics-informed" the training process by adding known differential equations into the loss function. For example, if you know that you are modeling gravity, then you know the accelerations should have zero curl, so you loss function can just tack on that differential constraint: $$ \mathcal{L}(\theta) = \frac{1}{N}\sum_{i=0}^N (y_i - \hat{y}(x_i|\theta))^2 + \nabla \times \hat{y}(x_i|\theta) $$ This ensures that if the model produces a solution that is naturally compliant with the underlying dynamics. This ends up being enormously useful, as it effectively reduces the set of learnable functions for the neural network, which can dramatically increase sample efficiency of these systems. Moreover, the models learned by the neural networks can be far more compact and efficient than the alternative analytic approximations used today.
PINN-Gravity Model
PINNs for gravity modeling were the primary focus of my dissertation. PINNs fundamentally changed how I think about machine learning. Before I had encountered these models, I was among those who thought all you can do is naively "slap a neural network" on a hard problem, but the reality is that engineers and scientists have far more control over these neural networks than people realize. We force them to do all sorts of tricks for us. We can build them to satisfy certain physics, abide by certian mathematical proerpties, and even design them to turn on and off exactly where we want (boundary conditions). These things are like little engines and just provide horsepower; it's our job to decide where and how we want that power spent.
Neural Ordinary Differential Equations
In addition to learning solutions to known differential equations, neural networks can also be used to learn the underlying differential equation itself. These models are called neural ODEs. These are neural networks that, when integrated, produce the measured data at different time points. I.e.
Applications to Astrodynamics
Machine learning applied to astrodynamics is a relatively new field of research. Broadly speaking, people have begun exploring how to use reinforcement learning generate more robust control strategies for spacecraft to maneuver in dynamically complex environments. Others have explored RL for selecting the optimal flight mode when a spacecraft must balance safety and mission objectives. Our group historically specializes in coming up with models of complex dynamical phenomena using machine learning.