Scientific ML Studio

Section 4.1 found that $dT/dt = -kT$ has a whole family of solutions, and that one fact, the temperature at $t = 0$, picks a single member. Every differential equation in this book needs facts of that kind. For an equation in space they live on the boundary of the domain; for an equation in time, at the start. Choosing them is part of stating the problem, not an afterthought, and a PINN has to be told every one.

How many conditions

Take the simplest second-order equation, $u'' = 0$ on $0 < x < 1$. Integrating twice gives $u = A + Bx$: a straight line with two free constants. Two facts are needed to fix them, for example the values at both ends. In general an ODE of order $n$ needs $n$ conditions. For a PDE the rule of thumb is one condition at every point of the boundary, which in practice means one condition on each edge of the domain, and, if the equation contains time, as many initial conditions as the order in time: one for the heat equation, two (position and velocity) for the wave equation.

Four kinds of boundary condition

Let $\mathbf{n}$ be the outward unit normal at a boundary point, so $\partial u/\partial n = \nabla u\cdot\mathbf{n}$ is the slope of $u$ as you step out of the domain. In one dimension, on $[0, 1]$, the outward normal points to the left at $x = 0$, so $\partial u/\partial n = -u'(0)$ there, and to the right at $x = 1$, so $\partial u/\partial n = u'(1)$.

Kind	Imposes	For heat in a bar, it means
Dirichlet	$u = g$	the end is held at a fixed temperature, by an ice bath for instance
Neumann	$\partial u/\partial n = h$	a prescribed heat flux; $h = 0$ is an insulated end
Robin	$\alpha u + \beta\,\partial u/\partial n = g$	the end loses heat to the air at a rate set by a heat-transfer coefficient
Periodic	$u$ and $\partial u/\partial x$ equal at opposite ends	the bar is bent into a ring

The signs deserve care. Fourier's law says heat flows from hot to cold, at a rate $\mathbf{q} = -k\nabla u$, where $k$ is the conductivity. The heat leaving through the boundary per unit area is therefore $\mathbf{q}\cdot\mathbf{n} = -k\,\partial u/\partial n$. A positive outward slope means the temperature rises as you leave the domain, so heat flows in. A Robin condition for cooling by air at temperature $u_\text{air}$ with heat-transfer coefficient $h_c$ states that the heat conducted out equals the heat carried away, $-k\,\partial u/\partial n = h_c\,(u - u_\text{air})$, which is $h_c u + k\,\partial u/\partial n = h_c u_\text{air}$ in the form of the table.

Well-posed problems

A problem is well posed, in the sense Hadamard gave the word, when three things hold: a solution exists, there is only one, and it depends continuously on the data, so that a small change in a boundary value or a source changes the solution only a little (Strauss, 2008, sec. 1.5). A PINN has no way of detecting a problem that breaks one of these rules. It will return something, often with a pleasingly small loss. So it pays to recognise the two common ways a problem goes wrong.

Too few conditions: the answer is not unique

Consider $-u'' = f$ on $(0, 1)$ with a Neumann condition at both ends, $u'(0) = u'(1) = 0$: a bar insulated at both ends, with a heat source. If $u$ solves it, so does $u + C$ for any constant $C$, because neither the equation nor the conditions ever look at the value of $u$, only at its derivatives. The problem has infinitely many solutions.

A computer sees this as a singular matrix. Here is the standard finite-difference version of the problem, $n + 1$ equally spaced nodes with the second-difference formula inside and one condition row at each end, for three choices of condition:

import numpy as np

def second_difference(n, left, right):
    """Matrix for -u'' on n + 1 equally spaced nodes of [0, 1], plus one condition row at each end.
    'value' imposes u at that end, 'slope' imposes u' there (by a one-sided difference)."""
    h = 1.0 / n
    A = np.zeros((n + 1, n + 1))
    for i in range(1, n):
        A[i, i - 1:i + 2] = [-1 / h**2, 2 / h**2, -1 / h**2]
    A[0, :2] = [1, 0] if left == "value" else [-1 / h, 1 / h]
    A[n, -2:] = [0, 1] if right == "value" else [-1 / h, 1 / h]
    return A

for left, right in [("value", "value"), ("value", "slope"), ("slope", "slope")]:
    A = second_difference(50, left, right)
    print(f"{left:5s} at x=0, {right:5s} at x=1:  rank {np.linalg.matrix_rank(A)} of {len(A)},"
          f"  smallest singular value {np.linalg.svd(A, compute_uv=False)[-1]:.1e}")

A = second_difference(50, "slope", "slope")
null = np.linalg.svd(A)[2][-1]          # the direction that A sends to (almost) zero
print("the direction it cannot see:", np.round(null[:4], 4), "...", np.round(null[-4:], 4))

value at x=0, value at x=1:  rank 51 of 51,  smallest singular value 2.0e-01
value at x=0, slope at x=1:  rank 51 of 51,  smallest singular value 1.2e-01
slope at x=0, slope at x=1:  rank 50 of 51,  smallest singular value 3.1e-13
the direction it cannot see: [0.14 0.14 0.14 0.14] ... [0.14 0.14 0.14 0.14]

With a value imposed at either end the matrix has full rank and the problem has one solution. With slopes at both ends it loses one rank, and the direction it cannot see is a vector whose entries are all equal: adding a constant. There is a second catch. Integrating the equation over the bar gives $\int_0^1 f\,dx = u'(0) - u'(1) = 0$, so a solution exists at all only if the source adds no net heat. That is a compatibility condition: a steady temperature in an insulated bar is possible only if the heat put in balances to zero.

Data that cannot be trusted: the backward heat equation

The heat equation smooths. A solution made of the modes $\sin(n\pi x)$ of Section 4.1 has each mode multiplied by $e^{-a n^2\pi^2 t}$ after a time $t$, and high modes (large $n$) die fastest. Run that backwards, to reconstruct an earlier temperature from a later one, and each mode is multiplied by $e^{+a n^2\pi^2 t}$ instead. Here is what that does to a measurement error of $10^{-4}$ of the signal (0.01 K on a temperature difference of 100 K, a very good thermometer), with $a = 0.1$ and $t = 0.1$:

a, t = 0.1, 0.1
print("mode   forward factor   backward factor   error 1e-4 becomes")
for n in [1, 2, 4, 8, 16]:
    f = np.exp(-a * (n * np.pi) ** 2 * t)
    print(f"{n:4d}   {f:14.2e}   {1/f:15.2e}   {1e-4/f:18.2e}")

mode   forward factor   backward factor   error 1e-4 becomes
   1         9.06e-01          1.10e+00             1.10e-04
   2         6.74e-01          1.48e+00             1.48e-04
   4         2.06e-01          4.85e+00             4.85e-04
   8         1.81e-03          5.54e+02             5.54e-02
  16         1.06e-11          9.40e+10             9.40e+06

The slowest mode is barely affected. Mode 8 turns an error of $10^{-4}$ into one of $0.06$, and mode 16 into one of ten million. Solutions exist, but they do not depend continuously on the data: the third of Hadamard's conditions fails. No method can recover the fine detail of the past from slightly noisy measurements of the present, and that includes a PINN. This matters again in Chapter 8, because many inverse problems have exactly this character.

Figure 4.2. Left: forward in time the heat equation wipes out the ripple ($a = 0.1$). Right: backward in time the same mechanism amplifies every mode $n$ by $e^{a n^2\pi^2 t}$, so tiny errors in high modes explode.

Initial conditions, and conditions that disagree

An initial condition is a boundary condition on the bottom edge of the space–time domain. Its number follows the order in time: the heat equation needs $u(x, 0)$; the wave equation needs $u(x, 0)$ and $u_t(x, 0)$, because a string released from rest and one that is thrown can start from the same shape.

Initial and boundary data meet at the corners of the space–time domain, and there they ought to agree. A bar that starts at $u(x, 0) = 1$ with both ends suddenly held at zero has data that disagree at $(0, 0)$ and $(1, 0)$. The mathematics copes, with a solution that is discontinuous at those two corners for an instant, but a smooth network cannot, and its error collects there. When you are free to choose the data, choose them compatible. Change a starter in the Studio Tour shows a case: swapping a fixed end for an insulated one also requires a new initial profile.

Where this goes in a PINN

Every condition becomes one more term in the loss (Section 6.3), or is built into the network so that it holds exactly (Section 6.4). Count them before you train. The Studio's Graph check counts them for you (No condition on BC2. Those boundaries are left free.), but it cannot tell whether the set you chose is well posed. With both ends insulated, a PINN converges to one of the infinitely many solutions, chosen by the accident of its starting weights; Section 6.3 shows it happening.

Exercises

The beam equation $EI\,w'''' = q$ is fourth order. How many conditions does a beam need? Write them for a cantilever of length $L$: clamped at $x = 0$, free at $x = L$. (At a free end the bending moment $EI\,w''$ and the shear force $EI\,w'''$ vanish.)
Show that $-u'' = 1$ on $(0, 1)$ with $u'(0) = u'(1) = 0$ has no solution at all.
On the right end of a bar on $[0, L]$, the outward normal points in the $+x$ direction. Write the Robin condition for cooling by air in terms of $u'(L)$. What does it become when $h_c \to 0$? When $h_c \to \infty$?

Answers

Four. Clamped at $x = 0$: $w(0) = 0$ and $w'(0) = 0$ (no deflection, no rotation). Free at $x = L$: $w''(L) = 0$ and $w'''(L) = 0$ (no moment, no shear force).
Integrate over the bar: $\int_0^1 -u''\,dx = u'(0) - u'(1) = 0$, but $\int_0^1 1\,dx = 1$. The source adds heat that an insulated bar cannot lose, so no steady state exists: the compatibility condition fails.
$\partial u/\partial n = u'(L)$, so $h_c\,u(L) + k\,u'(L) = h_c\,u_\text{air}$. As $h_c \to 0$ it becomes $u'(L) = 0$, an insulated end (Neumann). As $h_c \to \infty$, dividing by $h_c$ leaves $u(L) = u_\text{air}$, a fixed temperature (Dirichlet). Robin conditions sit between the two.

Recap

An ODE of order $n$ needs $n$ conditions; a PDE needs a condition on every part of the boundary and as many initial conditions as its order in time.
Dirichlet fixes values, Neumann fixes outward slopes (for heat, $-k\,\partial u/\partial n$ is the heat leaving), Robin mixes them, periodic ties opposite ends.
A well-posed problem has one solution that depends continuously on its data. Pure Neumann problems lack uniqueness; running the heat equation backwards loses continuous dependence. A PINN will not warn you.

Boundary and initial conditions