A new approach to uncertainty quantification of PINNs

Robbie Slos; Tom Lefebvre; Jolan Wauters; Guillaume Crevecoeur

doi:10.5281/zenodo.17435270

A new approach to uncertainty quantification of PINNs

Robbie Slos; Tom Lefebvre; Jolan Wauters; Guillaume Crevecoeur

Publisher: Zenodo
DOI: 10.5281/zenodo.17435270
Repository: zenodo.org
Language: en

Abstract

Physics-Informed Neural Networks (PINNs) refer to a deep learning approach to represent the spatial and temporal characteristics of a distributed physical phenomenon, such as thermal fields, using Neural Networks (NNs). The loss function used to train PINNs relies on a term that penalises any violation of the Partial Differential Equation (PDE) that governs the distributed phenomena, in addition to a least-squares-error penalisation of any available observations in the relevant domain. PINNs have been shown to be successful in approximating the solutions to PDEs and have proven effective even in the low data regime. A critical shortcoming is the lack of a systematic treatment of the uncertainty of the approximation. State-of-the-art approaches rely on Bayesian NNs, though these are computationally heavy, both during training and at inference, nor does the associated training procedure derive from first principles. To remedy these limitations, we propose Variational Inference PINNs (VI-PINNs). Our approach derives from first principles: the uncertainty intrinsic to the distributed phenomena is explained by adopting Stochastic PDEs, while relying on standard measurement uncertainty to explain the observational uncertainty. This leads to the formulation of a posterior probability for the distributed phenomenon. Drawing parallels with Bayesian inference in finite spaces and relying on VI techniques to circumvent the otherwise intractable posterior. We derive a training objective that allows us to train two NNs, representing the mean and covariance of the approximation, respectively. The solution may be interpreted as a Bayesian belief about the true distributed phenomenon. Importantly, in the limit, the original PINN framework is recovered. We compare our approach with Bayesian PINNs (B-PINNs). Our results suggest that VI-PINNs are easier to implement, have a lower training time, and yields results that better align with reality, especially when extrapolating outside the measurement range.

Full text

Contact [email protected] www.ugent.be/ea Universiteit Gent @ugent Ghent University VALIDATION –TOY EXAMPLE For the toy example, a 1D steady state Disturbed Thermal Field (DTF) in a 1 m long, homogeneous beam is chosen, described by the SPDE and Boundary Condition (BC): 𝑘𝑑2𝒖 𝒙 𝑑𝒙2=𝜔 𝒙 𝐵𝐶:𝒖 0 =𝒖 1 =1 With 𝒙∈[0.,1.], the spatial coordinates, k>0, conductivity constant, 𝜔 ∼𝒢𝒫(𝜇𝜔,𝜎𝜔 2), stochastic heat flux and 𝒖∼𝒢𝒫(𝜇𝑢,𝜎𝑢 2), the stochastic DTF. Over the length of the beam are some torches placed, as seen above. Such that after normalizing k = 1 and the heat flux is described by 𝜇𝜔=6𝜋 2sin 6𝜋⋅𝑥 and 𝜎𝜔 2=4⋅𝛿(𝒙−𝒙′). Implementation VFE-PINN The mean, 𝜇𝑞≈𝜇𝒖, is estimated with an NN an use the firs two parts of 𝐽𝑉𝐼 as governing equation. For the variance, 𝜎𝑞 2≈𝜎𝒖 2, the following kernel is proposed, with 𝜎𝑖…a NN and 𝛼 <0a trainable parameter: 𝜎𝑞 2𝒙,𝒙′= 𝜎𝑖𝒙 ⋅𝜎𝑖𝒙′exp 𝛼⋅ 𝒙−𝒙′ 2 The last tree part of 𝐽𝑉𝐼 are used as governing equation for the training of 𝜎𝑞 2. The general structures of B-PINN using HMC (left) and VFE-PINN (right) are show below: Results Conclusion •VFE-PINN uses an algebraic expression for quantifying the uncertainty, and therefore faster than sample-based prediction. •𝜎𝑞 2is included into the governing equation, therefore don’t explode outside the training range. •Training of 𝜇𝑞and 𝜎𝑞 2could be done in parallel. B-PINN (a) B-PINN (b) VFE-PINN Solving technique Training time * Mean Square Error Acceptance Rate (2⋅𝜎𝑞) B -PINN (a) 23 min 10,93 s 7,93 e-2 100,00 % B -PINN (b) 8 min 49,34 s 1,76 e-1 100,00 % VFE -PINN 4 min 39,57 s 6,97 e-6 100,00 % *All calculations are performed on 32 CPUs (Intel(R) Core(TM) i9 -14900K) BACKGROUND Physics Informed Neural Network Partial Differential Equations (PDEs) are deterministic representations of the dynamics of a system. A PDE has the general form: 𝒜𝒙,𝑡 𝑢 𝒙,𝑡 =𝜕𝑡𝑢 𝒙,𝑡 +𝒜𝒙𝑢 𝒙,𝑡 𝒙∈ℝ𝑛is a spatial vector, 𝑡 ∈ ℝthe time, 𝜕𝑡𝑢 𝑥,𝑡 the time derivative of 𝑢 𝒙,𝑡 and 𝒜𝒙𝑢 some spatial linear operator on 𝒖. The objective is to find the function 𝑢that satisfies the PDE. In a Physics Informed Neural Network (PINN), the unknown function 𝑢is approximated by a Neural Network (NN), with the output 𝒖𝜽, with the parameters 𝜽. The cost function is of the form: 𝐽𝑐𝑜𝑠𝑡 𝜃 = 1−𝜆 1 2𝒚−𝒖𝜽𝒙,𝑡 2+𝜆 1 2𝒜𝒙,𝑡 𝒖𝜽𝒙,𝑡 2 𝒚are measurements at 𝑥𝑚,𝑡𝑚, 𝜆is a tuning factor that determines the relative importance of each cost function. 𝒟𝐷𝑎𝑡𝑎 is the data set, and 𝒟𝑃𝐻𝑌𝑆 represent arbitrary set of space-time coordinates where the PDE is enforce to the NN. Bayesian Physics-Informed Neural Network The PINN-framework can be extended to a Bayesian–PINN (B-PINN) as a way for quantify the uncertainty of the prediction. All parameters θ (inc. NN and PDE parameters), are token stochastic, with the posterior : 𝑝 𝜃 𝒟𝐷𝑎𝑡𝑎,𝒟𝑃𝐻𝑌𝑆) ∝𝑝 𝒟𝐷𝑎𝑡𝑎 𝜃 𝑝 𝒟𝑃𝐻𝑌𝑆 𝜃 Instead of minimising a cost function, the objective is now to maximise the likelihood. Since the posterior is intractable, the literature proposes to approximate it using Monte Carlo samples. Possible strategies to establish these samples are HMC, or NUTS. VARIATIONAL FREE ENERGY PHYSICS INFORMED NEURAL NETWORK A B-PINN can find a good approximation, but it does so inefficiently due to the sampling. We desire to derive a more efficient approach. Our idea is to approximate 𝑝(𝒖)using a Variational density and learn the density using Variation Inference (VI). Problem sketch Applying Bayesian inference in a finite-dimensional context is a well-understood problem. The posterior for the quantity, u, given the measurements, y, is given by Bayes’ rule: 𝑝 𝒖 𝒚 =𝑝 𝒚,𝒖 𝑝(𝒚) =𝑝 𝒚 𝒖 ⋅𝑝 𝒖 𝑝 𝒚 We assume to have access to the measurement model 𝑝(𝒚,𝒖)and prior 𝑝(𝒖). Bayes’ rule is elegant, but quite often not applicable in practice due to the unknown data distribution 𝑝(𝒚). VI is a popular technique that is used to approximate the posterior. The idea is to fit a variational density 𝑞that is a member of some density family 𝒬by solving an optimisation problem. Using Kullback–Leibler (KL) divergence to determine the statistical distance between two distributions. To find q, the KL-divergence must be minimised. Assume 𝒖is a function and the solution of a Stochastic Partial Differential Equations (SPDE): 𝒜𝒙,𝑡 𝒖 𝒙,𝑡 =𝜕𝑡𝒖 𝒙,𝑡 +𝒜𝒙𝒖 𝒙,𝑡 =𝜔 𝒙,𝑡 𝜔 ∼𝒢𝒫 𝜇𝜔,𝜎𝜔 2 In the case that 𝒜𝒙𝒖is a linear operator, it is know that 𝒖∼𝒢𝒫 𝜇𝒖,𝜎𝒖 2. Once we have access to 𝜇𝒖and 𝜎𝒖 2, the problem is solved. The existing techniques depend on linear function decompositions of the function and shift all uncertainty to the coefficients. Rather, we would want to draw a connection with the PINN paradigm and approximating 𝜇𝒖and 𝜎𝒖 2. So, we approach an SPDE similar to how a PINN approaches a PDE. Variational Inference for Stochastic Partial Differential Equations Consider the static SPDE, with some linear operator 𝒜𝒙and the stochastic force term 𝜔: 𝒜𝒙𝒖 𝒙 =𝜔 𝒙 After derivations, the approximated VFE is described by: 𝐽𝑉𝐼 𝜇𝑞,𝜎𝑞 2≈1 2Δ𝒙2෍ 𝒟𝑃𝐻𝑌𝑆 2𝒜𝒙𝜇𝑞𝑇𝜎𝜔 −2 𝒙,𝒙′𝒜𝒙′𝜇𝑞+1 2෍ 𝒟𝐷𝐴𝑇𝐴 𝜎𝒚 −2 𝒚−ℬ𝒙𝜇𝑞2 +1 2Δ𝑥2෍ 𝒟𝑃𝐻𝑌𝑆 2𝜎𝜔 −2 𝒙,𝒙′𝒜𝒙𝜎𝑞 2𝒙,𝒙′𝒜𝒙′𝑇+1 2෍ 𝒟𝐷𝐴𝑇𝐴 𝜎𝒚 −2ℬ𝒙𝜎𝑞 2𝒙𝒎,𝒙𝒎ℬ𝒙𝑇+1 2log det Σ𝑞𝑞 𝑆𝑎𝑚𝑝𝑙𝑒 683 woorden A new approach to uncertainty quantification of PINNs DEPARTMENT OF ELECTROMECHANICAL, SYSTEMS AND METAL ENGINEERING Ir. Robbie Slos –Prof. Dr. Ir. Tom Lefebvre –Prof. Dr. Ir. Jolan Wauters –Prof. Dr. Ir. Guillaume Crevecoeur ෍ 𝒙,𝑡 ∈𝒟𝑃𝐻𝑌𝑆 ෍ 𝒙𝑚,𝑡𝑚,𝒚 ∈𝒟𝐷𝑎𝑡𝑎 𝑞∗𝒖 =argmin 𝑞∈𝒬 𝐾𝐿[𝑞(𝒖)| 𝑝 𝒖 =argmin 𝑞∈𝒬 𝔼𝑞 𝒖 log 𝑞 𝒖 𝑝 𝒚 𝒖 𝐽𝑉𝐼