Harnessing Data and Equations: The Magic of Physics-Informed Neural Networks#

The Building Blocks: Neurons and Layers#

A neural network is a computational model composed of stacked layers of “neurons�? Each neuron essentially computes a weighted sum of its inputs and applies an activation function (like ReLU, Sigmoid, or Tanh). In a standard feed-forward network, the data flows from the input layer through one or more hidden layers and finally produces an output.

Typical neural networks are trained using a dataset of inputs and corresponding labels. We define a loss function (for example, mean squared error) and attempt to minimize it via gradient-based optimizers like stochastic gradient descent (SGD) or Adam.

Conventional Training#

In conventional supervised learning, we have:

1. Input: A set of features X = {x�? x�? �? xₙ}.
2. Output: A set of corresponding labels Y = {y�? y�? �? yₙ}.
3. Network: A function f�?parameterized by weights ϕ (the neural network weights and biases).
4. Loss Function: L(ϕ) = ∑ᵢ ||f�?x�? �?yᵢ||² + (possible regularization terms).

We compute gradients of L(ϕ) w.r.t. ϕ and update the parameters iteratively.

Limitations in Scientific Domains#

• Scarcity of data: Experimentally obtained data can be expensive or impractical to collect in large quantities.
• High noise: Real-world measurements might be subject to sensor artifacts or external disturbances.
• Need for interpretability: In many scientific and engineering tasks, we need solutions that adhere to known physical laws—something standard deep networks do not guarantee by default.

Overview#

1. Neural Network Architecture
   A typical PINN might use a fully-connected feed-forward network with several hidden layers. Activations can vary, but Tanh, ReLU, and Swish are commonly used.

2. Loss Function
   The loss function L can be constructed as:
   L = L_data + L_phys + L_boundary,
   where:

   • L_data enforces agreement with available data points.
   • L_phys enforces PDE residual minimization.
   • L_boundary enforces boundary/initial conditions.

3. Automatic Differentiation
   PINNs heavily rely on automatic differentiation (AD) libraries (such as in TensorFlow or PyTorch) to compute derivatives of the neural network outputs with respect to inputs. This is crucial for plugging the network predictions into PDEs.

Diagram of a Simple PINN Setup#

A simplified depiction (in text form) could be:

1
Inputs (x, t) ----> [Fully-Connected NN] ----> Predicted Output u�?x, t)
2
                \--- Automatic Differentiation ---/
3
                                |
4
                                v
5
                    PDE Residual Calculation
6
                                |
7
                                v
8
                           Loss Function

Analytical Solution#

An analytical solution to this linear ODE is u(t) = e^(−t).

PINN Approach#

1. Neural Network: Define a function u�?t).

2. Compute Derivative: We can use automatic differentiation to compute ∂u�?∂t.

3. Define Loss:

   • L_data for initial condition: L_initial = (u�?0) �?1)²
   • L_phys for PDE residual: L_phys = �?(∂u�?∂t + u�?² dt (approximated by sampling points in the domain).

4. Total Loss: L = λ₁L_initial + λ₂L_phys.

Simple Code Snippet (PyTorch)#

Below is an illustrative example of how one might implement a PINN in PyTorch for this simple ODE.

1
import torch
2
import torch.nn as nn
3
import torch.optim as optim
4

5
# Define the neural network
6
class PINN(nn.Module):
7
    def __init__(self, hidden_dim=20):
8
        super(PINN, self).__init__()
9
        self.fc1 = nn.Linear(1, hidden_dim)
10
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
11
        self.fc3 = nn.Linear(hidden_dim, 1)
12
        self.activation = nn.Tanh()
13

14
    def forward(self, t):
15
        x = self.activation(self.fc1(t))
16
        x = self.activation(self.fc2(x))
17
        x = self.fc3(x)
18
        return x
19

20
# Loss function components
21
def loss_function(model, t_domain):
22
    # PDE residual part
23
    t_domain.requires_grad = True
24
    u = model(t_domain)
25
    du_dt = torch.autograd.grad(u, t_domain,
26
                                torch.ones_like(u),
27
                                create_graph=True)[0]
28
    pde_residual = du_dt + u
29

30
    pde_loss = torch.mean(pde_residual**2)
31

32
    # Initial condition part
33
    u0_pred = model(torch.tensor([[0.0]]))
34
    ic_loss = (u0_pred - 1.0)**2
35

36
    return pde_loss + ic_loss
37

38
# Training
39
model = PINN()
40
optimizer = optim.Adam(model.parameters(), lr=0.01)
41

42
# Sample training points, e.g., t in [0,1]
43
t_values = torch.linspace(0, 1, 50).view(-1, 1)
44

45
for epoch in range(10000):
46
    optimizer.zero_grad()
47
    loss = loss_function(model, t_values)
48
    loss.backward()
49
    optimizer.step()
50

51
    if epoch % 1000 == 0:
52
        print(f"Epoch {epoch}, Loss: {loss.item()}")
53

54
# Test at t=0.5
55
t_test = torch.tensor([[0.5]])
56
u_pred = model(t_test).item()
57
true_u = torch.exp(-t_test).item()
58
print(f"Predicted u(0.5): {u_pred}, True: {true_u}")

In this snippet:

• We define a small network with a couple of layers and tanh activation.
• The PDE loss is computed by automatically differentiating through the network with respect to time.
• The initial condition loss is enforced by comparing the network’s prediction at t=0 to the known initial value (1.0).

Analytical Perspective#

If f(x) = π² sin(πx), for example, we know the true solution is u(x) = sin(πx). But let’s pretend we do not know this solution and want to solve it via a PINN.

PINN Formulation#

1. Neural Network: Define u�?x).

2. Compute Second Derivative: We obtain ∂²u�?∂x² via automatic differentiation.

3. Loss Function:

   • PDE Loss:
     L_phys = ∑ᵢ [∂²u�?x�?/∂x² �?f(x�?]²
     over collocation points x�?in (0,1).

   • Boundary Condition Loss:
     L_boundary = [u�?0)]² + [u�?1)]²
     since we want u(0) = 0 and u(1) = 0.

   • Total:
     L = L_phys + L_boundary.

Illustrative Code Snippet (PyTorch)#

1
import torch
2
import torch.nn as nn
3
import torch.optim as optim
4

5
# PINN for solving Poisson equation u''(x) = pi^2 sin(pi*x)
6
class PoissonPINN(nn.Module):
7
    def __init__(self, hidden_dim=20):
8
        super(PoissonPINN, self).__init__()
9
        self.fc1 = nn.Linear(1, hidden_dim)
10
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
11
        self.fc3 = nn.Linear(hidden_dim, 1)
12
        self.activation = nn.Tanh()
13

14
    def forward(self, x):
15
        x = self.activation(self.fc1(x))
16
        x = self.activation(self.fc2(x))
17
        x = self.fc3(x)
18
        return x
19

20
def poisson_loss(model, x_in):
21
    x_in.requires_grad = True
22
    u_pred = model(x_in)
23

24
    # First derivative
25
    du_dx = torch.autograd.grad(u_pred, x_in,
26
                                torch.ones_like(u_pred),
27
                                create_graph=True)[0]
28
    # Second derivative
29
    d2u_dx2 = torch.autograd.grad(du_dx, x_in,
30
                                  torch.ones_like(du_dx),
31
                                  create_graph=True)[0]
32

33
    # PDE: u''(x) = pi^2 sin(pi*x)
34
    f = (torch.pi**2) * torch.sin(torch.pi * x_in)
35
    pde_res = d2u_dx2 - f
36
    pde_loss_val = torch.mean(pde_res**2)
37

38
    # Boundary conditions
39
    u0 = model(torch.tensor([[0.0]]))
40
    u1 = model(torch.tensor([[1.0]]))
41
    bc_loss = u0**2 + u1**2
42

43
    total_loss = pde_loss_val + bc_loss
44
    return total_loss
45

46
# Instantiate and train
47
model = PoissonPINN(hidden_dim=20)
48
optimizer = optim.Adam(model.parameters(), lr=0.001)
49

50
x_data = torch.linspace(0, 1, 50).view(-1, 1)
51

52
for epoch in range(5000):
53
    optimizer.zero_grad()
54
    loss_val = poisson_loss(model, x_data)
55
    loss_val.backward()
56
    optimizer.step()
57

58
    if epoch % 500 == 0:
59
        print(f"Epoch {epoch}, Loss: {loss_val.item()}")
60

61
# Testing
62
x_test = torch.linspace(0, 1, 100).view(-1, 1)
63
u_pred = model(x_test).detach().numpy()
64
u_true = torch.sin(torch.pi * x_test).numpy()

Collocation Points#

In PDE-based PINNs, we need a set of collocation points in the spatial (and possibly temporal) domain. We evaluate the network at these points to compute the PDE residual. Common strategies for choosing collocation points include:

• Uniform Grids: Simple and easy but may not capture features in regions where the solution changes rapidly.
• Random Sampling: Helps to avoid aliasing and can be used with adaptive methods.
• Adaptive Approaches: Dynamically choose new collocation points where the PDE residual is larger.

Mini-Batching vs. Full-Batch#

Because many PDEs are high-dimensional, splitting collocation points into mini-batches can significantly reduce memory requirements and may speed up convergence (though the typical approach for smaller problems is often full-batch).

Optimization Techniques#

• Gradient Descent with Momentum (e.g., Adam or RMSProp) typically works well.
• Learning Rate Scheduling: Using an adaptive schedule can help the network converge more stably.
• Regularization: Because PDE constraints already provide a strong regularization, you may not always need additional L2 or dropout, but it can help in noisy or large-scale problems.

Handling Complex Geometries#

Many real-world PDEs are defined on complex domains (like irregularly shaped regions in 2D or 3D). One approach is to use parameterizations of the domain boundary or mesh sampling. Another advanced strategy is to use coordinate transformations or even embedding-based methodologies to represent curved boundaries.

Physics-Informed Generative Adversarial Networks (PI-GANs)#

In some problems, we might prefer a GAN-style approach to generate physically consistent samples. A discriminator can be designed to detect PDE violations, effectively ensuring solutions respect the underlying physics.

Multifidelity PINNs#

If you have data of varying fidelity (e.g., cheap, coarse simulations and expensive, high-accuracy measurements), multifidelity PINNs can combine these levels of resolution to achieve efficient training. They might use separate networks or shared encoders for each fidelity level.

Transfer Learning for PINNs#

In certain applications, you can train a PINN on one problem domain and then transfer the learned weights to a similar problem—reducing training times and capitalizing on learned physical priors (e.g., fluid flows with slightly varying Reynolds numbers).

(x, y, t)-Dependent PDEs and Spatiotemporal Problems#

For problems in higher dimensions and time, the process is similar but with higher-dimensional inputs and partial derivatives. Automatic differentiation remains a key tool.