Academia.eduAcademia.edu

Figure from:Respecting causality is all you need for training physics-informed neural networks

See full PDFdownloadDownload figure
Figure 24: Navier-Stokes: Representative snapshots of the predicted w against the ground truth at t = 0.2, 0.4, 0.6, 0.8

Figure 24 Navier-Stokes: Representative snapshots of the predicted w against the ground truth at t = 0.2, 0.4, 0.6, 0.8

Related Figures (31)

Figure 1: Allen-Cahn equation: Top: Reference solution versus the prediction of a trained conventional physics- informed neural network. The resulting relative L? error is 49.87%. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at t = 0.0, 0.5, 1.0.
Figure 1: Allen-Cahn equation: Top: Reference solution versus the prediction of a trained conventional physics- informed neural network. The resulting relative L? error is 49.87%. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at t = 0.0, 0.5, 1.0.
Figure 2: Allen-Cahn equation: Left: Loss convergence of training a conventional physics-informed neural network for 2 x 10° iterations. Middle: Temporal residual loss L(t, @) at different iteration of the training. Right: Temporal convergent rate at different iteration of the training.
Figure 2: Allen-Cahn equation: Left: Loss convergence of training a conventional physics-informed neural network for 2 x 10° iterations. Middle: Temporal residual loss L(t, @) at different iteration of the training. Right: Temporal convergent rate at different iteration of the training.
Table 1: Allen-Cahn equation: Relative L? errors obtained by different approaches. 3 Causal training for physics-informed neural networks
Table 1: Allen-Cahn equation: Relative L? errors obtained by different approaches. 3 Causal training for physics-informed neural networks
Figure 3: Allen-Cahn equation: Top: Reference solution versus the prediction of a trained physics-informed neural network using Algorithm 1. The resulting relative L? error is 1.43e — 03. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at t = 0.0, 0.5, 1.0.
Figure 3: Allen-Cahn equation: Top: Reference solution versus the prediction of a trained physics-informed neural network using Algorithm 1. The resulting relative L? error is 1.43e — 03. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at t = 0.0, 0.5, 1.0.
Figure 4: Allen-Cahn equation: Left: Loss convergence of training a physics-informed neural network using Algorithm 1. Middle: Temporal residual loss £(t, @) at different iteration of the training. Right: Temporal weights at different iteration of the training.
Figure 4: Allen-Cahn equation: Left: Loss convergence of training a physics-informed neural network using Algorithm 1. Middle: Temporal residual loss £(t, @) at different iteration of the training. Right: Temporal weights at different iteration of the training.
Figure 5: Lorentz system: Comparison between the predicted and reference solutions.
Figure 5: Lorentz system: Comparison between the predicted and reference solutions.
Figure 6: Kuramoto—Sivashinsky equation (regular): Top: Reference solution versus the prediction of a trained physics- informed neural network using Algorithm 1. The resulting relative L? error is 3.49e — 04. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at ¢ = 0, 0.5, 1.0.
Figure 6: Kuramoto—Sivashinsky equation (regular): Top: Reference solution versus the prediction of a trained physics- informed neural network using Algorithm 1. The resulting relative L? error is 3.49e — 04. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at ¢ = 0, 0.5, 1.0.
Figure 7: Kuramoto—Sivashinsky equation (regular): Left: Timing of evaluating the loss function of a PINN model with different number of layers. The rest hyper-parameters are the same as in Table 3. Right: Timing of evaluating the forward pass of a PINN model with different batch sizes. The rest hyper-parameters are the same as in Table 3.
Figure 7: Kuramoto—Sivashinsky equation (regular): Left: Timing of evaluating the loss function of a PINN model with different number of layers. The rest hyper-parameters are the same as in Table 3. Right: Timing of evaluating the forward pass of a PINN model with different batch sizes. The rest hyper-parameters are the same as in Table 3.
Figure 8: Kuramoto-Sivashinsky equation: Left: Relative L? errors of Case I (regular). Right: Relative L? errors of Case II (chaotic).
Figure 8: Kuramoto-Sivashinsky equation: Left: Relative L? errors of Case I (regular). Right: Relative L? errors of Case II (chaotic).
Figure 9: Kuramoto—Sivashinsky equation (chaotic): Top: Reference solution versus the prediction of a trained physics- informed neural network using Algorithm 1. The resulting relative L? error is 2.26e — 02. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at t = 0, 0.25, 0.5.
Figure 9: Kuramoto—Sivashinsky equation (chaotic): Top: Reference solution versus the prediction of a trained physics- informed neural network using Algorithm 1. The resulting relative L? error is 2.26e — 02. Bottom: Comparison of the predicted and reference solutions corresponding to the three temporal snapshots at t = 0, 0.25, 0.5.
Figure 10: Kuramoto—Sivashinsky equation (chaotic): Reference solution versus the prediction of a trained physics. informed neural network using Algorithm 1. The initial condition is uo(a) = cos(a)(1 + sin(a)) An animation of the solution evolution is provided at https: //github.com/PredictiveIntelligenceLab/CausalPINNs# kuramotosivashinsky- equation.
Figure 10: Kuramoto—Sivashinsky equation (chaotic): Reference solution versus the prediction of a trained physics. informed neural network using Algorithm 1. The initial condition is uo(a) = cos(a)(1 + sin(a)) An animation of the solution evolution is provided at https: //github.com/PredictiveIntelligenceLab/CausalPINNs# kuramotosivashinsky- equation.
Figure 11: Navier-Stokes equation: Representative snapshot of the predicted velocity and vorticity versus the corre- sponding reference solution at t = 1. An animation of the solution evolution is provided at https: //github.com/ PredictiveIntelligenceLab/CausalPINNs#navier-stokes- equation.
Figure 11: Navier-Stokes equation: Representative snapshot of the predicted velocity and vorticity versus the corre- sponding reference solution at t = 1. An animation of the solution evolution is provided at https: //github.com/ PredictiveIntelligenceLab/CausalPINNs#navier-stokes- equation.
Figure 12: Navier-Stokes equation: Relative L? prediction errors for u, v, w, respectively.
Figure 12: Navier-Stokes equation: Relative L? prediction errors for u, v, w, respectively.
Figure 13: Navier-Stokes equation: Reference versus predicted normalized kinetic energy spectra at different time snapshots t = 0.0, 0.5, 1.0.
Figure 13: Navier-Stokes equation: Reference versus predicted normalized kinetic energy spectra at different time snapshots t = 0.0, 0.5, 1.0.
Figure 14: Parallel Performance: Left: Strong Scaling: Keeping the total-batch size for the problem fixed, we evaluate the speedup obtained when the batch is split across multiple devices. Centre: Weak Scaling: Keeping the batch-size on each GPU fixed, we report the efficiency of scaling by dividing the time taken on a single device over the time taken on n-devices. Right: Effect of batch-size: L? error for models trained till t = 0.1 using N; and N,, points per iteration in the temporal and spatial domain respectively.
Figure 14: Parallel Performance: Left: Strong Scaling: Keeping the total-batch size for the problem fixed, we evaluate the speedup obtained when the batch is split across multiple devices. Centre: Weak Scaling: Keeping the batch-size on each GPU fixed, we report the efficiency of scaling by dividing the time taken on a single device over the time taken on n-devices. Right: Effect of batch-size: L? error for models trained till t = 0.1 using N; and N,, points per iteration in the temporal and spatial domain respectively.
Table 2 summarizes the main symbols and notations used in this work.
Table 2 summarizes the main symbols and notations used in this work.
Table 3: Network architectures for each benchmark employed in this work. Table 3 summarizes the network hyper-parameters for all numerical experiments. We tuned these hyper-parameters manually, without attempting to find the absolute best hyper-parameter setting. This process can be automated in the future leveraging effective techniques for meta-learning and hyper-parameter optimization [59].
Table 3: Network architectures for each benchmark employed in this work. Table 3 summarizes the network hyper-parameters for all numerical experiments. We tuned these hyper-parameters manually, without attempting to find the absolute best hyper-parameter setting. This process can be automated in the future leveraging effective techniques for meta-learning and hyper-parameter optimization [59].
Table 4: Computational cost reported timings are obtained on NVIDIA RTX A6000 graphics cards. We remark that "Max Iteration" is the maximum iteration for every tolerance € in each time window. The default tolerance list is [10~7, 10~1, 10°, 101, 107] unless otherwise stated. The total number of iterations may vary for different examples due to the stopping criterion (see Algorithm 1). Training: Table 4 summarizes the computational cost of training PINNs. The size of different models as well as network architectures are listed table 3. All networks are trained using NVIDIA RTX A6000 graphics cards.
Table 4: Computational cost reported timings are obtained on NVIDIA RTX A6000 graphics cards. We remark that "Max Iteration" is the maximum iteration for every tolerance € in each time window. The default tolerance list is [10~7, 10~1, 10°, 101, 107] unless otherwise stated. The total number of iterations may vary for different examples due to the stopping criterion (see Algorithm 1). Training: Table 4 summarizes the computational cost of training PINNs. The size of different models as well as network architectures are listed table 3. All networks are trained using NVIDIA RTX A6000 graphics cards.
Figure 15: Allen-Cahn equation: Top: Exact solution versus the prediction of a trained physics-informed neural network using Algorithm 1 and modified MLP. The resulting relative L? error is 2.46e — 04. Bottom: Comparison of the predicted and exact solutions corresponding to the three temporal snapshots at t = 0.0, 0.5, 1.0. Validation: We solve the Allen-Cahn equation using conventional spectral methods. Specifically, assuming periodic boundary conditions, we start from the initial condition ug (x) = x? cos(7) and integrate the system up to the final time T = 1. Synthetic validation data are generated using the Chebfun package [60] with a spectral Fourier discretization with 512 modes and a fourth-order stiff time-stepping scheme (ETDRK4) [61] with time-step size 10~°.
Figure 15: Allen-Cahn equation: Top: Exact solution versus the prediction of a trained physics-informed neural network using Algorithm 1 and modified MLP. The resulting relative L? error is 2.46e — 04. Bottom: Comparison of the predicted and exact solutions corresponding to the three temporal snapshots at t = 0.0, 0.5, 1.0. Validation: We solve the Allen-Cahn equation using conventional spectral methods. Specifically, assuming periodic boundary conditions, we start from the initial condition ug (x) = x? cos(7) and integrate the system up to the final time T = 1. Synthetic validation data are generated using the Chebfun package [60] with a spectral Fourier discretization with 512 modes and a fourth-order stiff time-stepping scheme (ETDRK4) [61] with time-step size 10~°.
Figure 16: Allen-Cahn equation: Left: Loss convergence of training a physics-informed neural network using Algorithm 1. Middle: Temporal residual loss £(t, ™) at different training iteration. Right: Temporal weights at different training iteration.
Figure 16: Allen-Cahn equation: Left: Loss convergence of training a physics-informed neural network using Algorithm 1. Middle: Temporal residual loss £(t, ™) at different training iteration. Right: Temporal weights at different training iteration.
Figure 17: Lorentz system: Left: Loss convergence of training a physics-informed neural network using Algorithm | for every time window.
Figure 17: Lorentz system: Left: Loss convergence of training a physics-informed neural network using Algorithm | for every time window.
Figure 18: Lorentz system: Reference solutions versus the predicted solutions obtained by training a physics-informed neural network using Algorithm 1 with fixed iterations.
Figure 18: Lorentz system: Reference solutions versus the predicted solutions obtained by training a physics-informed neural network using Algorithm 1 with fixed iterations.
Figure 19: Lorentz system: Left: Loss convergence of training a physics-informed neural network using Algorithm | for every time window.
Figure 19: Lorentz system: Left: Loss convergence of training a physics-informed neural network using Algorithm | for every time window.
Figure 20: Kuramoto—Sivashinsky equation (regular): Loss convergence of training a physics-informed neural network using Algorithm 1 for every time window.
Figure 20: Kuramoto—Sivashinsky equation (regular): Loss convergence of training a physics-informed neural network using Algorithm 1 for every time window.
Figure 21: Kuramoto—Sivashinsky equation (chaotic): Loss convergence of training a physics-informed neural network using Algorithm 1 for every time window.
Figure 21: Kuramoto—Sivashinsky equation (chaotic): Loss convergence of training a physics-informed neural network using Algorithm 1 for every time window.
Furthermore, in Appendix we also present our results simulating the turbulent flow up to T’ = 2. Figure 26 presents the visualizations of the predicted velocity and vorticity field at the final state. The predictive accuracy is quantified in Figure 27. Although the resulting relative L? error is above 10%, our model predictions seem to be qualitatively correct against the corresponding ground truth.
Furthermore, in Appendix we also present our results simulating the turbulent flow up to T’ = 2. Figure 26 presents the visualizations of the predicted velocity and vorticity field at the final state. The predictive accuracy is quantified in Figure 27. Although the resulting relative L? error is above 10%, our model predictions seem to be qualitatively correct against the corresponding ground truth.
Figure 23: Navier-Stokes: Representative snapshots of the predicted v against the ground truth at t = 0.2, 0.4, 0.6, 0.8
Figure 23: Navier-Stokes: Representative snapshots of the predicted v against the ground truth at t = 0.2, 0.4, 0.6, 0.8
Figure 25: Navier-Stokes: Loss convergence of training a physics-informed neural network using Algorithm 1 for every time window.
Figure 25: Navier-Stokes: Loss convergence of training a physics-informed neural network using Algorithm 1 for every time window.
Figure 26: Navier-Stokes: Predicted u, v, w against the ground truth at t = 2.
Figure 26: Navier-Stokes: Predicted u, v, w against the ground truth at t = 2.
Figure 27: Navier-Stokes: Relative L? errors of u,v, w, respectively.
Figure 27: Navier-Stokes: Relative L? errors of u,v, w, respectively.