Is it right?

Section 1.3 separated the training error from the test error. A PINN needs the same separation, adapted to a problem where the "data" is an equation. This section asks three questions of a trained network, in order of increasing cost: did it satisfy the equation where it was trained; does it satisfy it between those points; and is it the right function? It ends with a checklist that later chapters, and the worked problems, use.

The experiment

The problem is $-u'' = (4\pi)^2\sin(4\pi x)$ on $(0, 1)$ with $u(0) = u(1) = 0$, whose solution, $\sin(4\pi x)$, has four half-waves: enough structure for a small set of points to miss something. We train the same network twice, once with 8 collocation points and once with 64, both fixed for the whole run, and with the residual divided by its natural size, $(4\pi)^2$, as Section 6.2 taught.

import math
import torch
import torch.nn as nn

def d(y, x):
    return torch.autograd.grad(y, x, torch.ones_like(y), create_graph=True)[0]

k = 4 * math.pi
f = lambda x: k**2 * torch.sin(k * x)                         # -u'' = f; the solution is sin(4 pi x)
residual = lambda u, x: -d(d(u(x), x), x) - f(x)

def train(n_col, seed=0, epochs=4000):
    torch.manual_seed(seed)
    net = nn.Sequential(nn.Linear(1, 32), nn.Tanh(), nn.Linear(32, 32), nn.Tanh(),
                        nn.Linear(32, 32), nn.Tanh(), nn.Linear(32, 1))
    u = lambda x: net(2 * x - 1)
    x_col = torch.rand(n_col, 1).requires_grad_(True)
    x_b = torch.tensor([[0.0], [1.0]])
    opt = torch.optim.Adam(net.parameters(), lr=1e-3)
    for epoch in range(epochs):
        opt.zero_grad()
        loss = (residual(u, x_col) ** 2).mean() / k**4 + (u(x_b) ** 2).sum()
        loss.backward()
        opt.step()
    return u, x_col

x_fresh = torch.rand(1000, 1, generator=torch.Generator().manual_seed(123)).requires_grad_(True)
x_test = torch.linspace(0, 1, 2001).reshape(-1, 1).requires_grad_(True)
rms = lambda t: t.detach().pow(2).mean().sqrt().item()

for n_col in (8, 64):
    u, x_col = train(n_col)
    with torch.no_grad():
        err = (u(x_test) - torch.sin(k * x_test)).abs().max().item()
    print(f"{n_col:2d} points:  rms residual where trained {rms(residual(u, x_col)):8.2e}"
          f"   at 1000 fresh points {rms(residual(u, x_fresh)):8.2e}   largest error {err:.2e}")

 8 points:  rms residual where trained 9.57e-02   at 1000 fresh points 7.50e+01   largest error 2.56e+00
64 points:  rms residual where trained 9.13e-01   at 1000 fresh points 1.79e+00   largest error 6.75e-02

With 8 points the residual at those points is small, and the network is wrong by more than the solution's own height. The residual at fresh points gives it away at once: it is hundreds of times larger than at the training points. With 64 points the two residuals are of the same order and the error is several per cent (Figure 6.5). The gap between the two residual columns is the PINN's generalisation gap, the counterpart of Section 1.3's train and test errors, and it is free to compute: no reference solution is needed, only the equation.

When the residual bounds the error

For some problems a small residual everywhere guarantees a small error, and the guarantee can be computed. Write $e = u_\theta - u$ for the error. Because the equation is linear, $-e'' = r$, where $r$ is the residual, and $e$ takes the boundary errors $e(0)$, $e(1)$ at the ends. Solving this with the Green's function of $-d^2/dx^2$ on $[0, 1]$ gives the bound

$$\max_x |e(x)| \;\le\; \max\bigl(|e(0)|, |e(1)|\bigr) + \tfrac18 \max_x |r(x)|,$$

because the boundary values contribute a straight line, which never exceeds its end values, and $\int_0^1 G(x, s)\,ds = x(1 - x)/2 \le 1/8$. Every quantity on the right can be evaluated for a trained network, on a dense grid, without knowing $u$:

u, x_col = train(64)
r_max = residual(u, x_test).detach().abs().max().item()
with torch.no_grad():
    e_ends = u(torch.tensor([[0.0], [1.0]])).abs().max().item()
    actual = (u(x_test) - torch.sin(k * x_test)).abs().max().item()
print(f"largest residual {r_max:.2e},  largest boundary error {e_ends:.1e}")
print(f"bound on the error {e_ends + r_max / 8:.2e},  actual largest error {actual:.2e}")

largest residual 6.47e+00,  largest boundary error 9.1e-04
bound on the error 8.10e-01,  actual largest error 6.75e-02

The bound holds, with room to spare: it is pessimistic because it assumes the worst residual everywhere at once. Its value is that it needs no reference. The same argument works for other well-posed linear problems, with a different constant in place of $1/8$, and that constant measures how strongly the problem amplifies residuals into errors. For Poisson's equation it is modest. For an evolution problem over a long time, or a Helmholtz equation near resonance, it can be enormous, and a small residual then promises little. This is the quantitative form of Section 4.2's continuous dependence on data.

More than one run

A PINN's result depends on its starting weights and its points, as Sections 6.2 and 13.1 found. Five runs of the 64-point problem, differing only in the seed:

errors = []
for seed in range(5):
    u, x_col = train(64, seed)
    with torch.no_grad():
        exact = torch.sin(k * x_test)
        errors.append(((u(x_test) - exact).norm() / exact.norm()).item())
errors.sort()
print("relative L2 errors, sorted:", "  ".join(f"{e:.1e}" for e in errors))
print(f"median {errors[2]:.1e}, range {errors[0]:.1e} to {errors[-1]:.1e}")

relative L2 errors, sorted: 2.1e-03  1.0e-02  1.1e-02  5.6e-02  1.3e-01
median 1.1e-02, range 2.1e-03 to 1.3e-01

The best and worst runs differ by a factor of about sixty. A single run could have been reported as either, and neither number would have been false. Report the median and the range over several seeds, as the tables of Chapter 13 do, and say how the final weights were chosen: the generated scripts keep the weights of the last epoch, which may sit on one of Adam's spikes (Section 3.6).

Exercises

1. In the first experiment, why is the residual at the 8 training points not exactly zero, even though the network has far more weights than there are points?
2. Use the bound to say how small the largest residual must be to guarantee an error below $10^{-3}$, if the boundary errors are $10^{-4}$.
3. The Studio's Report block can write the residual at each point of a regular grid. How would you use it to carry out step 2 of the checklist for a generated script?

Recap

• The residual at fresh points is a free test of whether the network learned the equation or only its training points; a large gap between the two is the PINN's generalisation gap.
• For well-posed linear problems the residual bounds the error, with a constant that measures how hard the problem is; for 1D Poisson it is $1/8$.
• Results vary from run to run. Report several, with median and range, and check against a reference and against physics whenever you can.