Academia.eduAcademia.edu

Figure from:Improved Architectures and Training Algorithms for Deep Operator Networks

See full PDFdownloadDownload figure
Figure 14: Stokes equation: Weight distribution defined in equation (4.39) for the same input sample as in Figure 12.

Figure 14 Stokes equation: Weight distribution defined in equation (4.39) for the same input sample as in Figure 12.

Related Figures (42)

Figure 1: Physics-informed DeepONets: The DeepONet architecture [4] consists of two sub-networks referred as the branch network and the trunk network, which extract latent representations of input functions u and input coordinates y at which the output functions are evaluated, respectively. A continuously differentiable representation of the output functions is then obtained by merging the outputs of each sub-network via a dot product. Automatic differentiation can then be employed to formulate appropriate regularization mechanisms for biasing the DeepONet outputs to satisfy a given system of PDEs.
Figure 1: Physics-informed DeepONets: The DeepONet architecture [4] consists of two sub-networks referred as the branch network and the trunk network, which extract latent representations of input functions u and input coordinates y at which the output functions are evaluated, respectively. A continuously differentiable representation of the output functions is then obtained by merging the outputs of each sub-network via a dot product. Automatic differentiation can then be employed to formulate appropriate regularization mechanisms for biasing the DeepONet outputs to satisfy a given system of PDEs.
Figure 2: Anti-derivative operator: Exact solution versus the predicted solution of a trained physics-informed DeepONet for the same input sample with different output scale k = 0.01 (left) and & = 100 (right).
Figure 2: Anti-derivative operator: Exact solution versus the predicted solution of a trained physics-informed DeepONet for the same input sample with different output scale k = 0.01 (left) and & = 100 (right).
Figure 3: Anti-derivative operator: Histogram of back-propagated gradients corresponding to the input samples with different output scale k € [10~?, 107]. Now we generalize the original definition of the NTK to physics-informed DeepONets. where N* = 2NR, and for every k, T? (-,-) can be the identity operator, differential operator \V(-, -) or the boundary condition B(-,-). We will refer to a loss function written in the above equation as a "fully-decoupled" loss. Similarly, we will refer to a loss function written in the form of equation (2.19) as a "partially-decoupled" loss.
Figure 3: Anti-derivative operator: Histogram of back-propagated gradients corresponding to the input samples with different output scale k € [10~?, 107]. Now we generalize the original definition of the NTK to physics-informed DeepONets. where N* = 2NR, and for every k, T? (-,-) can be the identity operator, differential operator \V(-, -) or the boundary condition B(-,-). We will refer to a loss function written in the above equation as a "fully-decoupled" loss. Similarly, we will refer to a loss function written in the form of equation (2.19) as a "partially-decoupled" loss.
Lemma 3.4. Given the loss function 3.6, the physics-informed DeepONet outputs is governed by the following ODE:
Lemma 3.4. Given the loss function 3.6, the physics-informed DeepONet outputs is governed by the following ODE:
Figure 4: Anti-derivative operator: Left: Mean and standard deviation of the relative L? prediction errors of the trained DeepONets using different weighting schemes, averaged over the test data sampled from a GRF with different output scale k € [10~?, 107]. A detailed description of the legend is summarized in Table 2. Right: Weight distributions of trained models with different weighting schemes in increasing order of ||u!|2.
Figure 4: Anti-derivative operator: Left: Mean and standard deviation of the relative L? prediction errors of the trained DeepONets using different weighting schemes, averaged over the test data sampled from a GRF with different output scale k € [10~?, 107]. A detailed description of the legend is summarized in Table 2. Right: Weight distributions of trained models with different weighting schemes in increasing order of ||u!|2.
from Wang et. al. [17] and modify the forward pass of an L-layer DeepONet as follows where © denotes point-wise multiplication, @ denotes a activation function, and 0 represents all trainable parameters of the DeepONet model. In particular, fw, pra and {W, wi? byt) }7+/ are the weights and biases of the branch and trunk networks, respectively.
from Wang et. al. [17] and modify the forward pass of an L-layer DeepONet as follows where © denotes point-wise multiplication, @ denotes a activation function, and 0 represents all trainable parameters of the DeepONet model. In particular, fw, pra and {W, wi? byt) }7+/ are the weights and biases of the branch and trunk networks, respectively.
Table 2: Weighting schemes and their associated notation, as considered in the numerical studies presented in this work.
Table 2: Weighting schemes and their associated notation, as considered in the numerical studies presented in this work.
Figure 5: Modified DeepONet architecture: Two encoders are applied to the input samples and the input coordinates, respectively. The embedded features are then inserted into each hidden layer of the branch network and the trunk network using a * operation, which is defined in equation (3.26) - (3.27). The final outputs is obtained by the same way as conventional DeepONets, i.e, merging the outputs of the branch and trunk networks via a dot production.
Figure 5: Modified DeepONet architecture: Two encoders are applied to the input samples and the input coordinates, respectively. The embedded features are then inserted into each hidden layer of the branch network and the trunk network using a * operation, which is defined in equation (3.26) - (3.27). The final outputs is obtained by the same way as conventional DeepONets, i.e, merging the outputs of the branch and trunk networks via a dot production.
Table 3: DeepON et architectures considered in the numerical studies presented in this work. We remark that DeepONet with MLP refers to a conventional DeepONet model, while DeepONet with modified MLP consists of two independent sub-networks employing modified MLP networks as backbone that does not share any parameters. One main difference between the modified DeepONet and conventional DeepONet architectures is that two encoders are employed to exchange information between the branch and the trunk networks before merging them together. To simplify the notations, we may omit “DeepONet” in the rest of this work.
Table 3: DeepON et architectures considered in the numerical studies presented in this work. We remark that DeepONet with MLP refers to a conventional DeepONet model, while DeepONet with modified MLP consists of two independent sub-networks employing modified MLP networks as backbone that does not share any parameters. One main difference between the modified DeepONet and conventional DeepONet architectures is that two encoders are employed to exchange information between the branch and the trunk networks before merging them together. To simplify the notations, we may omit “DeepONet” in the rest of this work.
and Ape, Aic, Ar Can be set to 1 or some other user-specified value, and remain fixed during training. For using different weighting schemes, we reformulate the loss function into a fully-decoupled form where f(x) = sin(7x) and g(t) = sin(t). To ensure numerical stability, we make the input function u(«) strictly positive by letting u(a) = v(a) — min, v(x) + 1 where v(a) is sampled from a GRF with a length scale / = 0.2. Our goal is to learn the solution operator G mapping the variable coefficient u(a) to the associated spatio-temporal PDE solution s(x, t).
and Ape, Aic, Ar Can be set to 1 or some other user-specified value, and remain fixed during training. For using different weighting schemes, we reformulate the loss function into a fully-decoupled form where f(x) = sin(7x) and g(t) = sin(t). To ensure numerical stability, we make the input function u(«) strictly positive by letting u(a) = v(a) — min, v(x) + 1 where v(a) is sampled from a GRF with a length scale / = 0.2. Our goal is to learn the solution operator G mapping the variable coefficient u(a) to the associated spatio-temporal PDE solution s(x, t).
Table 4: Advection equation: Test errors of trained physics-informed DeepONets using different weighting schemes and network architectures. In particular, the results of Ai, = Ape = shows the best accuracy obtained by training the physics-informed DeepONets for different \ € [10~?, 102]. The resulting test errors and their standard deviations are summarized in Figure 17.
Table 4: Advection equation: Test errors of trained physics-informed DeepONets using different weighting schemes and network architectures. In particular, the results of Ai, = Ape = shows the best accuracy obtained by training the physics-informed DeepONets for different \ € [10~?, 102]. The resulting test errors and their standard deviations are summarized in Figure 17.
Figure 6: Advection equation: Top: Exact solution versus the prediction of a trained conventional physics-informed DeepONet for a representative example in the test data-set. The resulting relative L? error is 9.32%. Bottom: Exact solution versus the prediction of a physics-informed DeepONet represented by modified DeepONet architecture and trained using Algorithm 1 with a = $ for the same example. The resulting relative L? error is 0.73%, which is 10x more accurate than the original formulation.
Figure 6: Advection equation: Top: Exact solution versus the prediction of a trained conventional physics-informed DeepONet for a representative example in the test data-set. The resulting relative L? error is 9.32%. Bottom: Exact solution versus the prediction of a physics-informed DeepONet represented by modified DeepONet architecture and trained using Algorithm 1 with a = $ for the same example. The resulting relative L? error is 0.73%, which is 10x more accurate than the original formulation.
Figure 7: Advection equation: Weight distribution defined in equation (4.11) for the same input sample as in Figure 6.
Figure 7: Advection equation: Weight distribution defined in equation (4.11) for the same input sample as in Figure 6.
To obtain a set of training data, we randomly sample N = 1,000 input functions from a GRF ~ N (0, 257(—A + 522)— r and for every input sample u, we uniformly sample P = 100 locations gO). = £5 aC, ey) and {(0, 1) ;) = pid, #, aie , from each boundary, and Q = 2500 collocation points { (x ee, Hy} © | from the domain interior. We generate the test data-set by sampling another 500 input functions from the same ie and solve the Burgers’ equation (4.12) using the Chebfun package [38] with a spectral Fourier discretization and a fourth-order stiff time-stepping scheme (ETDRK4) [39] with a time-step size of 10-4. Temporal snapshots of the solution are saved every At = 0.01 to give us 101 snapshots in total. Consequently, the test data-set contains 500 realizations evaluated at a 100 x 101 spatio-temporal grid.
To obtain a set of training data, we randomly sample N = 1,000 input functions from a GRF ~ N (0, 257(—A + 522)— r and for every input sample u, we uniformly sample P = 100 locations gO). = £5 aC, ey) and {(0, 1) ;) = pid, #, aie , from each boundary, and Q = 2500 collocation points { (x ee, Hy} © | from the domain interior. We generate the test data-set by sampling another 500 input functions from the same ie and solve the Burgers’ equation (4.12) using the Chebfun package [38] with a spectral Fourier discretization and a fourth-order stiff time-stepping scheme (ETDRK4) [39] with a time-step size of 10-4. Temporal snapshots of the solution are saved every At = 0.01 to give us 101 snapshots in total. Consequently, the test data-set contains 500 realizations evaluated at a 100 x 101 spatio-temporal grid.
Table 5: Burger’s equation: Test errors of trained physics-informed DeepONets using different weighting schemes and network architectures. In particular, the results of Aic = Apc = A shows the best accuracy obtained by training the physics-informed DeepONets for different \ € [10~?, 107]. The resulting test errors and their standard deviations are visualized in Figure 20.
Table 5: Burger’s equation: Test errors of trained physics-informed DeepONets using different weighting schemes and network architectures. In particular, the results of Aic = Apc = A shows the best accuracy obtained by training the physics-informed DeepONets for different \ € [10~?, 107]. The resulting test errors and their standard deviations are visualized in Figure 20.
Figure 8: Burger’s equation: Top: Exact solution versus the prediction of a trained conventional physics-informed DeepONet for a representative example in the test data-set. Bottom: Comparison of the predicted and exact solutions corresponding to the three temporal snapshots at t = 0.2, 0.5, 0.8. The resulting relative L? error is 30.72%
Figure 8: Burger’s equation: Top: Exact solution versus the prediction of a trained conventional physics-informed DeepONet for a representative example in the test data-set. Bottom: Comparison of the predicted and exact solutions corresponding to the three temporal snapshots at t = 0.2, 0.5, 0.8. The resulting relative L? error is 30.72%
Table 6: Solving a parametric Burger’s equation: Relative L? prediction error of a trained physics-informed DeepONet averaged over all examples in the test data-set, for different viscosity values.
Table 6: Solving a parametric Burger’s equation: Relative L? prediction error of a trained physics-informed DeepONet averaged over all examples in the test data-set, for different viscosity values.
Figure 9: Burger’s equation: Top: Exact solution versus the prediction of a physics-informed DeepONet represented by modified DeepONet architecture and trained using Algorithm | with a = $ for the same example as in Figure 8.. Bottom: Comparison of the predicted and exact solutions corresponding to the three temporal snapshots at t = 0.2, 0.5, 0.8. The resulting relative L? error is 1.19%, which is 30x more accurate than the original formulation.
Figure 9: Burger’s equation: Top: Exact solution versus the prediction of a physics-informed DeepONet represented by modified DeepONet architecture and trained using Algorithm | with a = $ for the same example as in Figure 8.. Bottom: Comparison of the predicted and exact solutions corresponding to the three temporal snapshots at t = 0.2, 0.5, 0.8. The resulting relative L? error is 1.19%, which is 30x more accurate than the original formulation.
Figure 10: Burger’s equation: Weight distribution defined in equation (4.11) for the same input sample as in Figure 8
Figure 10: Burger’s equation: Weight distribution defined in equation (4.11) for the same input sample as in Figure 8
where OF = [OI'(¢,), OF (¢2),..., OF (m)] denotes an input closed curve evaluated at evenly-spaced grid points {oO}, in [0, 27]. According to equation (4.23) - (4.27), a physics-informed loss can be formulated as follows
where OF = [OI'(¢,), OF (¢2),..., OF (m)] denotes an input closed curve evaluated at evenly-spaced grid points {oO}, in [0, 27]. According to equation (4.23) - (4.27), a physics-informed loss can be formulated as follows
Figure 11: Stokes equation: Mlustration of the computational domain and boundary conditions. Now we can define the following PDE residuals
Figure 11: Stokes equation: Mlustration of the computational domain and boundary conditions. Now we can define the following PDE residuals
Table 7: Stokes equation: Test errors of trained physics-informed DeepONets using different weighting schemes and network architectures. We remark that, in each cell, three rows represents the test error corresponding to the velocity u,v and the pressure p, respectively.
Table 7: Stokes equation: Test errors of trained physics-informed DeepONets using different weighting schemes and network architectures. We remark that, in each cell, three rows represents the test error corresponding to the velocity u,v and the pressure p, respectively.
Figure 12: Stokes equation: Exact solution versus the prediction of a trained conventional physics-informed DeepONet for a representative example in the test data-set. The resulting relative L? errors are 74.32%, 82.64%, 99.63% for U, U, Pp, respectively. The trained model completely fails to predict the correct solution.
Figure 12: Stokes equation: Exact solution versus the prediction of a trained conventional physics-informed DeepONet for a representative example in the test data-set. The resulting relative L? errors are 74.32%, 82.64%, 99.63% for U, U, Pp, respectively. The trained model completely fails to predict the correct solution.
Figure 13: Stokes equation: Exact solution versus the prediction of a physics-informed DeepONéet represented by modified DeepONet architecture and trained using Algorithm | for the same example as in Figure 12. The resulting relative L? errors are 1.88%, 5.74%, 2.41% for u, v, p, respectively.
Figure 13: Stokes equation: Exact solution versus the prediction of a physics-informed DeepONéet represented by modified DeepONet architecture and trained using Algorithm | for the same example as in Figure 12. The resulting relative L? errors are 1.88%, 5.74%, 2.41% for u, v, p, respectively.
Table 8 summarizes the main symbols and notation used in this work.
Table 8 summarizes the main symbols and notation used in this work.
Table 9: Default hyper-parameter settings for each benchmark employed in this work (unless otherwise stated) B_ Hyper-parameter settings
Table 9: Default hyper-parameter settings for each benchmark employed in this work (unless otherwise stated) B_ Hyper-parameter settings
Table 10: Physics-informed DeepONet architectures for each benchmark employed in this work (unless otherwise stated).
Table 10: Physics-informed DeepONet architectures for each benchmark employed in this work (unless otherwise stated).
Training: Table C summarizes the computational cost (hours) of training physics-informed DeepONet models with different network architectures and weighting schemes. The size of different models as well as network architectures are listed Table 10, respectively. All networks are trained using a single NVIDIA RTX A6000 graphics card.
Training: Table C summarizes the computational cost (hours) of training physics-informed DeepONet models with different network architectures and weighting schemes. The size of different models as well as network architectures are listed Table 10, respectively. All networks are trained using a single NVIDIA RTX A6000 graphics card.
Proof. Recall the definition of the loss function 3.6, E_ Proof of Lemma 3.4
Proof. Recall the definition of the loss function 3.6, E_ Proof of Lemma 3.4
Figure 15: Anti-derivative operator: Training loss convergence of a DeepONet models using different loss schemes for 4 x 10* 40,000 iterations of gradient descent using the Adam optimizer. Here we remark that all losses are weighted, and, therefore, the magnitude of these losses is not informative.
Figure 15: Anti-derivative operator: Training loss convergence of a DeepONet models using different loss schemes for 4 x 10* 40,000 iterations of gradient descent using the Adam optimizer. Here we remark that all losses are weighted, and, therefore, the magnitude of these losses is not informative.
Figure 17: Advection equation: Relative L? error of physics-informed DeepONets represented by different architectures and trained using different fixed weights Ape = Aic = A E (10-2, 107], averaged over the test data-set.
Figure 17: Advection equation: Relative L? error of physics-informed DeepONets represented by different architectures and trained using different fixed weights Ape = Aic = A E (10-2, 107], averaged over the test data-set.
Figure 18: Advection equation: Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 18: Advection equation: Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 19: Burger’s equation: Training loss convergence of a DeepONet models using different DeepONet architectures and weighting schemes for 2 x 10° iterations of gradient descent using the Adam optimizer. Here we remark that all losses are unweighted.
Figure 19: Burger’s equation: Training loss convergence of a DeepONet models using different DeepONet architectures and weighting schemes for 2 x 10° iterations of gradient descent using the Adam optimizer. Here we remark that all losses are unweighted.
Figure 20: Burger’s equation: Relative L? error of physics-informed DeepONets represented by different architectures and trained using different fixed weights Ape = Aic = A E (10-2, 107], averaged over the test data-set.
Figure 20: Burger’s equation: Relative L? error of physics-informed DeepONets represented by different architectures and trained using different fixed weights Ape = Aic = A E (10-2, 107], averaged over the test data-set.
Figure 21: Burger’s equation (v = 0.01): Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 21: Burger’s equation (v = 0.01): Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 22: Burger’s equation (v = 0.001): Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 22: Burger’s equation (v = 0.001): Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 23: Burger’s equation (v = 0.0001): Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 23: Burger’s equation (v = 0.0001): Predicted solution of the best trained physics-informed DeepONet for three different examples in the test data-set.
Figure 24: Stokes equation: Training loss convergence of a DeepONet models using different DeepONet architectures and weighting schemes for 2 x 10° iterations of gradient descent using the Adam optimizer. Here we remark that all losses are unweighted.
Figure 24: Stokes equation: Training loss convergence of a DeepONet models using different DeepONet architectures and weighting schemes for 2 x 10° iterations of gradient descent using the Adam optimizer. Here we remark that all losses are unweighted.
Figure 25: Stokes equation: Predicted solution of the best trained physics-informed DeepONet for one example in the test data-set.
Figure 25: Stokes equation: Predicted solution of the best trained physics-informed DeepONet for one example in the test data-set.
Figure 26: Stokes equation: Predicted solution of the best trained physics-informed DeepONet for one example in the test data-set.
Figure 26: Stokes equation: Predicted solution of the best trained physics-informed DeepONet for one example in the test data-set.
Figure 27: Stokes equation: Predicted solution of the best trained physics-informed DeepONet for one example in the test data-set.
Figure 27: Stokes equation: Predicted solution of the best trained physics-informed DeepONet for one example in the test data-set.