Image fusion using a new evolution equation

Image fusion using a new evolution equation Sabira Benalia1and Mohammed Hachama2 1USTHB, Laboratoire AMNEDP, Facult´e de math´ematiques, B.P. 32, El Alia, Bab Ezzouar, 16111 Alger, Algeria 2National Higher School of Mathematics, Scientific and Technology Hub of Sidi Abdellah, P.O. Box 75, Algiers 16093, Algeria. Abstract In this paper, we solve the image fusion problem using a new mathematical model. We reformulate a recent osmosis model using nonlocal differential operators. Experimental results show that the nonlocal model obtained very good qualitative results compared with state-of-theart models, including modern deep learning techniques. Keywords: Image restoration, image fusion, Nonlocal differential operators, Energy minimization. 1 Introduction Our objective is to fuse two grayscale images f (foreground) and b (background), which are real-valued functions defined on the same closed bounded regular domain Ω ⊂R2(fand bare supposed in L∞(Ω,(0,+∞))). Note that the extension to color images can be easily accomplished by processing each color channel independently. The domain Ω is decomposed into three distinct regions: Ωfwhich represents the image region copied from the foreground, Ωbwhich represents the image region copied from the background, and Ωsb is a transition region in which fand bare mixed (refer to Figure 1). Figure 1: Image fusion results. From left to right: background, selected region, foreground, Seamless Poisson Editing [4], our nonlocal osmosis model. 2 Proposed model We propose a new nonlocal model for image fusion which consists in minimizing the energy: E(u) := S(u) + λF(u),(1) where Sis a fusion term and Fis a fidelity term. These terms are balanced using positive weight λ. The fusion term: This term is a nonlocal osmosis model: S(u) = 1 2ZΩ v(x)∇NL u v 2(x) dx, (2) where v(x) = fα(x)(x)b1−α(x)(x) (3) and αis defined as follows: α(x) =      1,if x∈Ωf, G(x),if x∈Ωsb, 0,if x∈Ωb. 69 where the function Gensures a smooth transition on Ωsb between 0 and 11, while v=fon Ωfand v=b on Ωb. The function vcan be seen as a ”rough solution” of the fusion problem. Fidelity term: To ensure that the solution ushould stay close to fin the foreground region, we minimize the following term F(u) = 1 2ZΩ α(x) v(x)(f(x)−u(x))2dx. (4) In (2), ∇NL : Ω×Ω→R+stands for a nonlocal gradient which permits to take into account nonlocal interactions between distant pixels. Here, we use the Gilboa operator [3] defined by ∇NLu(x, y) := (u(y)−u(x)) pw(x, y),∀x, y ∈Ω, where w: Ω×Ω→R+is a weight assumed to be symmetric. The inner product of two ”vector” functions v,v1: Ω ×Ω−→ Ris defined as < v, v1>:= ZΩ×Ω v(x, y)v1(x, y) dxdy. The magnitude of vis a function |v|: Ω →Rdefined by |v|(x) := sZΩ v(x, y)2dy. The nonlocal divergence divNLv: Ω →Ris defined as the adjoint of the nonlocal gradient: (divNLv)(x) = ZΩ (v(x, y)−v(y, x)) pw(x, y) dy. The nonlocal Laplacian 4NLuis a function defined on Ω by 4NLu(x) := 1 2(divNL∇NLu)(x) = ZΩ (u(y)−u(x)) w(x, y) dy. where the factor 1 2is introduced to be consistent with the graph Laplacian definition. 2.1 Evolution problem In this section, we derive the evolution equation associated with (1), which can be seen as a ”continuous” gradient descent algorithm. We seek a time-dependent solution u(t, ·) that evolves over time toward the minimizer of the energy (1). Proposition 1 For f, b ∈L∞(Ω,R+), the evolution process associated to 1is defined as follows:    ∂tu(t, x) = divNL v(x)∇NL u(t, .) v(x, y)+λα(x)(f(x)−u(t, x)) in Ωt, u(0, x) = f(x),in Ω, (5) where Ωt= (0, T )×Ω. Proof 1 Let ψ∈C∞ c(Ω) be a test function and t > 0. For convenience, we may drop the the notation of the variable t. We compute the Gˆateaux derivative of E(u)at uin the direction ψ. The Gˆateaux variations of Fis straightforward: F0(u) = α v(u−f). 1The exact definition of Gis given in the Section dedicated to the numerical resolution. 70 On the other hand, we have S0(u)(ψ) = lim t→0S(u+tψ)−S(u) t, =ZZΩ×Ω v(x)u(y) v(y)−u(x) v(x)ψ(y) v(y)−ψ(x) v(x)ω(x, y) dydx, =ZZΩ×Ω v(x)u(y) v(y)−u(x) v(x)ψ(y) v(y)ω(x, y) dydx −ZZΩ×Ωu(y) v(y)−u(x) v(x)ψ(x)ω(x, y) dydx, =ZZΩ×Ω v(y)u(x) v(x)−u(y) v(y)ψ(x) v(x)ω(y, x) dxdy −ZZΩ×Ωu(y) v(y)−u(x) v(x)ψ(x)ω(x, y) dydx, =ZΩ−1 v(x)ZΩ v(x)u(y) v(y)−u(x) v(x)w(x, y) dyψ(x) dx +ZΩ1 v(x)ZΩv(y)u(x) v(x)−u(y) v(y)w(x, y) dyψ(x) dx, =ZΩ−1 v(x)divNL v∇NL u v(x)ψ(x) dx. Thus, we obtain: E0(u) = −divNL v∇NL u v−λα(f−u). The Euler-Lagrange equation E0(u)=0is difficult to solve. Consequently, we use the suboptimal gradient descent procedure ∂tu(t, x) = −E0(u). Thus, we obtain the system (5). 3 Numerical resolution Let Ωdbe a discrete grid associated to the continuous domain Ω: Ωd={1, . . . , M}×{1, . . . , N}. We use the following notations: i= (i1, i2)∈Ωd,xi= (xi1, yi2)∈Ω, ui≈u(xi), αi≈α(xi). The image fis approximated by a discrete matrix fd. The weight function ωdefines a similarity between two pixels xiand xjby comparing their neighborhoods: ωi,j=ωxi,xj= exp −1 h2 r X t1,t2=−r Gσ(t1, t2)f(xi+t)−f(xj+t)2!,(6) where rdefines the neighborhood, t= (t1, t2), and h > 0 is a scale parameter and Gσis a gaussian function (Gσ(t1, t2) = 1 2πσ2e−t2 1+t2 2 2σ2, with σ > 0). The approximation of the nonlocal Laplacian is given by: ∆NLu(xi)≈(∆NLu)i=X j∈Ωd (uj−ui)ωi,j. The nonlocal divergence of p: Ω ×Ω→Ris approximated by divNL(p) (xi)≈(divNL(p))i=X j∈Ωd (pi,j−pj,i)√ωi,j. 71 We approximate space derivative by forward differences and time derivative using explicit Euler method. The discrete iterative scheme of (5) writes: un+1 i=un i+dt dx  (∆NLun)i+X j∈Ωd vi vj un jωi,j−X j∈Ωd vj vi un iωi,j  +dt λαi(fi−un i) where dt,dx are the discretization steps, nis the time index. Then, the discrete problem can be written as algebraic equations of the form Un+1 =AUn+PF, where U∈Rm,m=M×N, is the vector obtained by concatenating the columns of the discrete image u,nis the time index, F= (fi)1≤i≤mand the initial condition is set to U0=F. 4 Experimental results We conducted extensive experiments to evaluate our model and compare it to previous state-of-the-art techniques: Poisson Image Editing [2], Parisotto et al. [4]. We implemented our algorithm using Matlab 2020a, but the weight function had been implemented as a c-mex file (C-program). Simulations have been conducted on a Core i5-4210 (2.60 GHz) processor with 4GO RAM. Based on an empirical analysis, we determined the values of the parameter λand found that λ∈[0,1]. For the weight function, we choose, in most cases, a patch size of 5 ×5, search window of size 7 ×7, σ= 0.003 and the filtering parameter h=σ. We used the software and test data provided by the authors 2,3,4. In some approaches, the authors presented an automatic method for selecting region. However, in this work, we assume that the subdomain is given to focus on the numerical solution of the nonlocal equation . Let’s montion, for the images that have not the same size, we create a new image foreground with the same size of background and with the centroid of selected region in the position with othor coordinates. The function αallows to indicat from which of the two images the structural information comes from. In the case where Ωsb =∅,αis binar, i.e. α(x)∈ {0,1}, so αvanishes on the pixels of the background image band is equal to one otherwise. Alternatively, αcan be smoothed by means of a Gaussian convolution so as to favour a smooth transition on Ωsb 6=∅. Experimental results obtained by the nonlocal model are presented on Figure 2(qualitative evaluation). As demonstrated visually, our method outperforms previous state-of-the-art techniques in most cases. It produces a better skin tone than the seamless Poisson editing model and Parisotto et al. [4] model as showing in Figure 2(a) and Figure 2(b). It is a powerful tool for manipulating colors, two differently colored versions of those images can be mixed seamlessly. For exemple, in Figure 2(d), our model overcome the problem of the undesirable visible seam of Parisotto et al. [4] model. An example is shown in Figure 2(e), in which our model can facilitate the transfer of partly transparent objects (the rainbow). 4.1 Comparison with deep learning techniques We also assess the performance of the proposed image fusion method using the Lytro5dataset.The fusion results of the proposed method are compared with three recently developed fusion algorithms. The first one is GFDF [5]. The second compared algorithm is DCT EOL [1]. The third method is CNN [6]. All image fusion results6of these algorithms are available online. The results are presented on Figure 3. 3https://doi.org/10.5201/ipol.2016.163 4http://cs.brown.edu/courses/cs129/results/proj2/damoreno/ 5https://sandipanweb.wordpress.com/2017/10/03/some-variational-image-processing-possion-image-editing-and-itsapplications/ 5https://www.researchgate.net/publication/291522937 Lytro Multi-focus Image Dataset 6https://github.com/xingchenzhang/MFIF 72 Figure 2: Image fusion results of different models ((1)-(8)). From left to right: background, selected region, foreground, Seamless Poisson Editing [2], Parisotto et al. [4], nonlocal osmosis model (1). Figure 3: Qualitative comparison of results of Lytro dataset. (1) and (2) are: Foreground and background. From (3) to (6) are fused images obtained by: GFDF [5], DCT EOL [1], CNN [6], nonlocal osmosis model (1). 73 5 Conclusion In this paper, we presented a novel nonlocal model for image fusion that utilizes nonlocal derivatives in the recently developed osmosis model. Experimental analyses have shown that our proposed model demonstrates its effectiveness and superiority over local image fusion models. Our proposed method provides visually plausible image data fusion that is invariant to multiplicative brightnessb changes. References [1] M. Amin-Naji and A. Aghagolzadeh. Multi-focus image fusion in dct domain using variance and energy of laplacian and correlation coefficient for visual sensor networks. Journal of AI and Data Mining,, pages 33–250, 2018. [2] J. Matas Di Martino, Gabriele Facciolo, and Enric Meinhardt-Llopis. Poisson Image Editing. Image Processing On Line, pages 300–325, 2016. [3] Guy Gilboa and Stanley Osher. Nonlocal operators with applications to image processing. Multiscale Modeling & Simulation, 7(3):1005–1028, 2009. [4] Simone Parisotto, Luca Calatroni, Aur´elie Bugeau, Nicolas Papadakis, and Carola-Bibiane Sch¨onlieb. Variational osmosis for non-linear image fusion. IEEE Transactions on Image Processing, pages 5507– 5516, 2020. [5] L. Zhang X. Qiu, M. Li and X. Yuan. Guided filter-based multi-focus image fusion through focus region detection. Signal Processing: Image Communication, pages 35–46, 2019. [6] H. Peng Y. Liu, X. Chen and Z.Wang. Multi-focus image fusion with a deep convolutional neural network. Information Fusion,, pages 191–207, 2017. 74