Chasing Symmetries: A New Wave of Physics-Inspired Deep Learning#

Classical CNNs and Translational Symmetry#

The first big success in leveraging symmetry in deep learning was provided by convolutional neural networks (CNNs). By moving a “convolutional kernel�?across an image, the network outputs features that remain meaningful regardless of where an object is located in the image. This explicitly uses the group of translations (in 2D images). CNNs have been the workhorse of computer vision tasks for years, demonstrating that capturing even a single type of symmetry (translational) can be extraordinarily profitable.

Group Equivariant CNNs#

Group Equivariant CNNs (G-CNNs) generalize translational convolution to other symmetry groups (e.g., rotations, flips). Instead of applying the same kernel to just different translations of the data, these networks apply specialized kernels to rotations, reflections, or even more general transformations. By doing so, they capture invariances in a structured way.

Physics-Informed Neural Networks (PINNs)#

G-CNNs focus on symmetry in the architecture. Another line of thought, Physics-Informed Neural Networks (PINNs), enforces physical constraints (e.g., the Navier–Stokes equations for fluid flow, or Maxwell’s equations for electromagnetism) on the training process. PINNs can discover solutions to differential equations or train with significantly less real-world data, leveraging known forms of physical laws as constraints.

Lie-Parameterized Neural Networks (Gauge Symmetries)#

For more advanced tasks, some network designs explicitly incorporate Lie group constraints, relevant for gauge theories in physics (as in electromagnetism or more advanced quantum field theories). These advanced networks enforce local group symmetries, or gauge symmetries, which broaden the concept of rotation evaluations at different points in space-time.

Example: Solving the Poisson Equation with a PINN#

Consider the Poisson equation in 2D: [

• \nabla^2 u(x, y) = f(x, y) ] with boundary conditions ( u = 0 ) on the domain boundary. A PINN for this scenario might involve these steps:

1. Architecture: A feedforward neural network taking ((x, y)) as input and outputting (u).
2. Loss Function:
   • Physics Loss: The PDE residual (\left| -\nabla^2 u - f(x, y) \right|^2).
   • Boundary Loss: The mismatch on boundary points ( \left| u_{\text{predicted}}(x_{\text{b}}, y_{\text{b}}) - 0 \right|^2).
   • (Optional) Data Loss: If some ground truth (u) is known at certain interior points, incorporate it too.

The total loss is a weighted sum: (\mathrm{Loss} = \alpha \cdot \mathrm{PhysicsLoss} + \beta \cdot \mathrm{BoundaryLoss} + \gamma \cdot \mathrm{DataLoss}).

Below is a simplified code snippet in Python (using a pseudo deep learning interface) that demonstrates the general structure of a PINN:

1
import numpy as np
2
import torch
3
import torch.nn as nn
4

5
# Define our network
6
class PoissonPINN(nn.Module):
7
    def __init__(self, hidden_units=32):
8
        super(PoissonPINN, self).__init__()
9
        self.layers = nn.Sequential(
10
            nn.Linear(2, hidden_units),
11
            nn.ReLU(),
12
            nn.Linear(hidden_units, hidden_units),
13
            nn.ReLU(),
14
            nn.Linear(hidden_units, 1)  # Output is u(x, y)
15
        )
16

17
    def forward(self, x):
18
        return self.layers(x)
19

20
# PDE residual function
21
def poisson_residual(u_pred, coords, f_func):
22
    # coords is shape [N, 2] with x, y
23
    # We'll approximate the Laplacian with autodiff:
24
    # u_pred is shape [N, 1]
25
    x = coords[:, 0].requires_grad_(True)
26
    y = coords[:, 1].requires_grad_(True)
27
    u = pinn(torch.stack([x, y], dim=1))
28

29
    # Gradient of u w.r.t x
30
    grad_u_x = torch.autograd.grad(u, x,
31
                                   torch.ones_like(u),
32
                                   create_graph=True)[0]
33
    # Gradient of u w.r.t y
34
    grad_u_y = torch.autograd.grad(u, y,
35
                                   torch.ones_like(u),
36
                                   create_graph=True)[0]
37

38
    # Second derivatives
39
    grad_u_xx = torch.autograd.grad(grad_u_x, x,
40
                                    torch.ones_like(grad_u_x),
41
                                    create_graph=True)[0]
42
    grad_u_yy = torch.autograd.grad(grad_u_y, y,
43
                                    torch.ones_like(grad_u_y),
44
                                    create_graph=True)[0]
45

46
    laplacian = grad_u_xx + grad_u_yy
47
    # PDE: -laplacian(u) - f(x, y) = 0
48
    return (-laplacian - f_func(x, y))**2
49

50
# Example f(x, y)
51
def f_func(x, y):
52
    return x * 0 + y * 0 + 1.0  # a constant for simplicity
53

54
# Instantiate the PINN
55
pinn = PoissonPINN(hidden_units=32)
56
optimizer = torch.optim.Adam(pinn.parameters(), lr=1e-3)
57

58
# Example training data
59
num_points = 1000
60
xy_inside = torch.rand(num_points, 2)
61
xy_boundary = torch.rand(num_points, 2)  # Suppose we know boundary points
62

63
for epoch in range(1000):
64
    optimizer.zero_grad()
65

66
    # Physics loss
67
    physics_loss = poisson_residual(None, xy_inside, f_func).mean()
68

69
    # Boundary loss (u should be 0)
70
    u_pred_boundary = pinn(xy_boundary)
71
    boundary_loss = torch.mean(u_pred_boundary**2)
72

73
    total_loss = physics_loss + boundary_loss
74
    total_loss.backward()
75
    optimizer.step()
76

77
    if epoch % 100 == 0:
78
        print(f"Epoch {epoch}, Loss: {total_loss.item():.4e}")

This snippet is a toy example, but it demonstrates the concept of enforcing physical laws via the PDE residual in the loss function. As you expand your approach to real-world PDEs or more complicated domains, the same principles will apply.

Group Equivariant Convolutional Networks (G-CNN)#

• Definition: For an input image, transformations from a group ( G ) (e.g., rotations, reflections) are applied, and the CNN’s convolution kernels are designed to handle those transformations in a structured manner.
• Advantages: They require fewer parameters for the same expressive power and are more robust in tasks where orientation or viewpoint changes are common.

Permutation Equivariant Networks#

• Applications: Commonly used in set-based or graph-based data. These networks are built so that permuting the elements of the input set results in a corresponding permutation of output indices.
• Key Idea: Using sum or mean pooling can achieve permutation invariance, while specialized layers can ensure permutation equivariance.

O(3)-Equivariant Networks for 3D#

• Use-Case: Common in computational chemistry (molecular property prediction) and 3D shape analysis. For molecular data, we want the network’s output to be invariant (e.g., total energy) or equivariant (e.g., forces) under rotations and translations in 3D space.

1. Combining PINNs with Equivariance#

Imagine you are modeling a physical system governed by PDEs, like fluid flows around an object with inherent rotational symmetry. You could combine the PDE constraints from PINNs with rotational-equivariant architectures to drastically reduce the complexity of your problem. For instance, reflecting or rotating boundary conditions might reduce the domain you need to model.

2. Gauge Equivariant Neural Networks#

Gauge symmetries appear in electromagnetism, quantum chromodynamics, and more. A gauge transformation might vary from point to point in your domain, but the physical observables remain invariant. Building neural networks that incorporate these gauge symmetries allows for stable generalization and physically meaningful representations—particularly relevant in advanced scientific computing and partial differential equation solving.

3. Lie Transformer and Continuous Symmetries#

Transformers have gained considerable traction in recent years. Researchers are exploring how to build Lie Transformers that incorporate transformations from continuous groups. They might replace standard self-attention with a layer that is equivariant under rotations or translations. This approach stands to benefit scenarios where attention-based models (like in natural language processing or 3D geometry tasks) meet continuous group symmetries.

4. Neural Operators (DeepONet, FNO)#

A growing field is focusing on Neural Operators, which map function spaces to function spaces. These handle infinite-dimensional analogs of data, capturing entire fields (like temperature or velocity profiles) rather than discrete points. Frameworks like Deep Operator Networks (DeepONet) or Fourier Neural Operators (FNO) have soared in popularity for solving PDEs quickly. They often embed or can be combined with symmetry considerations, ensuring that the operator respects the underlying symmetries of the physical system.

5. Uncertainty Quantification and Bayesian Approaches#

Real physical systems are seldom deterministic. Combining physics-based priors, group symmetries, and Bayesian inference can yield robust, interpretable models that provide uncertainty estimates—crucial for high-stakes domains like aerospace engineering, seismology, or medical imaging.

Key Takeaways of This Example#

• We used the Rot2dOnR2(N=4) group from e2cnn to build a network that is equivariant to 4 discrete rotations.
• Each convolution changes the field type (number and type of channels).
• GroupPooling leads to a globally invariant descriptor, suitable for classification.

Learning Symmetries from Data#

Although we have described how to hard-code known symmetries into a network, an ongoing area of research is learning symmetries directly from data. For example, if you do not know the true transformation group that preserves your data distribution, you might try to discover it automatically. These approaches typically involve:

• Generative modeling frameworks that decode latent variables in a way that unrolls transformations observed in the dataset.
• Self-supervised learning tasks that guess transformation invariants, encouraging the model to learn or reveal the underlying group structure.

Extensions to Large-Scale Systems#

In big engineering problems—like climate modeling or nuclear fusion simulations—the synergy between data-driven learning and physics-based constraints holds enormous promise. The ability to handle multiscale phenomena while respecting physical invariances can substantially cut computational costs and provide breakthroughs in high-complexity scenarios.

Final Thoughts#

The pursuit of symmetries in neural networks, reminiscent of the search for symmetries in fundamental physics, seeks universality, efficiency, and truth. Models that intrinsically recognize physical laws or symmetrical structures can better generalize, require less data, and produce more trustworthy outputs in real-world applications.

In summary, the new wave of physics-inspired deep learning is about:

1. Incorporating known transformations (translations, rotations, reflections, permutations, gauge transformations) into network architectures.
2. Enforcing physical constraints (e.g., PDEs) in the training process to produce models that comply with real-world laws out of the box.
3. Building robust, generalized solutions that scale to complex, potentially high-dimensional systems.

By mastering these techniques, you’ll be poised to contribute to cutting-edge research, tackle industrial-scale scientific problems, and perhaps discover new symmetries or principles along the way. The synergy of deep learning and physics is still in its infancy, and we can only imagine what revolutions await as these fields unite even further.