A variational approach to camera motion smoothing

Differential Equations &Applications Volume 3, Number 4 (2011), 555–564 A VARIATIONAL APPROACH TO CAMERA MOTION SMOOTHING LUIS ALVAREZ,LUIS GOMEZ,PEDRO HENRIQUEZ AND LUIS MAZORRA Dedicated to Professor Jes´us Ildefonso D´ıaz on the occasion of his 60th birthday (Communicated by M. Paˇsi´c) Abstract. In this paper we study a variational problem derived from a computer vision application: video camera calibration with smoothing constraint. By video camera calibration we mean to estimate the location, orientation and lens zoom-setting of the camera for each video frame taking into account image visible features. To simplify the problem we assume that the camera is mounted on a tripod, in such case, for each frame captured at time t, the calibration is provided by 3 parameters : (1) P(t)(PAN) which represents the tripod vertical axis rotation, (2) T(t) (TILT) which represents the tripod horizontal axis rotation and (3) Z(t)(CAMERA ZOOM) the camera lens zoom setting. The calibration function t→u(t)=(P(t),T(t),Z(t)) is obtained as the minima of an energy function I[u]. In this paper we study the existence of minima of such energy function as well as the solutions of the associated Euler-Lagrange equations. 1. Introduction In this paper we deal with video camera calibration with smoothing constraint. We focus our attention on the case of video cameras mounted on a tripod. In such case, for each time tof the video sequence, camera calibration configuration is provided by 3 parameters : (1) P(t)(PAN) which represents the tripod vertical axis rotation, (2) T(t) (TILT) which represents the tripod horizontal axis rotation and (3) Z(t)(CAMERA ZOOM) the camera lens zoom setting. From these time-dependent functions we can easily deduce (using tripod information), for each time t, the camera calibration, that is, the camera intrinsic and extrinsic parameters which determine the position of the camera in the 3Dspace and the way 3Dobjects are projected in the camera projection plane (the CCD in the case of digital cameras). Human being visual system is very sensitive to motion, and small perturbations over time of P(t),T(t),andZ(t)values produce small oscillations in the camera motion disturbing the observer. To remove such perturbations is a critical issue in applications like the inclusion of artificial graphic objects in real video sequence scenarios. To illustrate this phenomenon, in http://www.ctim.es/demo102 we show a real video Mathematics subject classification (2010): 35J20, 35J50, 35J55, 35J60. Keywords and phrases: variational methods, camera calibration, Euler-Lagrange equations. This research has partially been supported by the MICINN project reference MTM2010-17615 (Ministerio de Ciencia e Innovaci´on. Spain). c    ,Zagreb Paper DEA-03-34 555 556 LUIS ALVAREZ,LUIS GOMEZ,PEDRO HENRIQUEZ AND LUIS MAZORRA Figure 1: One video frame of a typical application scenario. The primitives used for calibration are the white lines and cercles detected in the image. sequence where some graphic objects have been included. In this video we illustrate the main practical problem we deal with, that is, standard calibration techniques which do not take into account the expected time regularity of u(t)=(P(t),T(t),Z(t)) can introduce a significant noise in the camera motion estimation over time. In this paper we propose to include a smoothing constraint in the estimation of u(t)by minimizing the following energy : I[u]=t1 t0P(t)2+T(t)2+Z(t)2+ α F(u(t),t)dt,(1) where [t0,t1]is the time interval, α ⩾0 is a weight to balance the different components of the energy and, F(x1,x2,x3,t)⩾0 is a standard calibration function which forces, for each time t, that the projection of 3Dpoints be close to primitives detected in the image. In fact, when no time regularization is used, u(t)is usually estimated by minimizing F(u(t),t)independently for each time t.So, by using the proposed variational model (1), we introduce a time regularity condition in the video camera calibration procedure. The Euler-Lagrange equations associated to energy (1) yields to the following nonlinear system of differential equations: ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ −P(t)+ α∂ F ∂ x1(P(t),T(t),Z(t),t)=0in(t0,t1), −T(t)+ α∂ F ∂ x2(P(t),T(t),Z(t),t)=0in(t0,t1), −Z(t)+ α∂ F ∂ x3(P(t),T(t),Z(t),t)=0in(t0,t1), with adequate boundary conditions. The paper is organized as follows: in section 2, we briefly introduce the basic mathematical models we use for camera calibration. In AVARIATIONAL APPROACH TO CAMERA MOTION SMOOTHING 557 section 3, we analyze the calibration function F(x1,x2,x3,t). In section 4, we show the existence of minimizers of energy functional (1) and that such minimizers are solutions of the associated Euler-Lagrange equations. Finally, in section 5, we present some experiments to illustrate the smoothing performance of the proposed method. 2. Geometry and calibration of cameras mounted on a tripod A tripod is defined by two unitary rotation axes e0=e0 x,e0 y,e0 zTand e1= e1 x,e1 y,e1 zTand a center of rotation X0∈R3.PanparameterP(t)determines the tripod rotation angle with respect to e0. We denote by Re0,P(t)the associated rotation matrix. In a similar way we define Re1,T(t). The general rotation matrix generated by tripod motion is a composition of the above rotation matrix. Given a 3D point Xthe point transformation induced by the tripod motion can be expressed as: XP(t),T(t)=X0+Re0,P(t)Re1,T(t)X−X0.(2) We use the basic pinhole model to modelize the way the 3D scene is projected in the 2D image projection plane. Such projection is expressed in projective coordinates as a 4 ×3 projection matrix P(u(t)) defined by P(u(t)) ≡AZ(t)R0Id,−c0R(P(t),T(t)) tP(t),T(t) 01 ,(3) where: RP(t),T(t)≡Re0,P(t)Re1,T(t),(4) t(P(t),T(t)) = X0−RP(t),T(t)X0,(5) AZ(t)=⎛ ⎝ Z(t)0xc 0r·Z(t)yc 001 ⎞ ⎠,(6) R0=⎛ ⎝ r0 00 r0 01 r0 02 r0 10 r0 11 r0 12 r0 20 r0 21 r0 22 ⎞ ⎠,(7) Id,−c0=⎛ ⎝ 100−c0 x 010−c0 y 001−c0 z ⎞ ⎠.(8) R0and c0correspond to the initial tripod rotation and translation. For more details about the pinhole model see, for instance, [3], [4]. 558 LUIS ALVAREZ,LUIS GOMEZ,PEDRO HENRIQUEZ AND LUIS MAZORRA 3. The calibration function F(u(t),t) To calibrate a camera frame is to obtain the associated u(t)=(P(t),T(t),Z(t)) parameters which determine the projection matrix P(u(t)).To estimate u(t)we use the ”observable” information we can obtain in the image. For instance in the image of fig. 1, we can see some parts of the white lines and circles of the soccer court. In general, we note by Ω(t)⊂R2the finite collection of visible curves in the image we use to calibrate frame t.We assume that we know the actual 3D location of Ω(t)in the real scene. That is, for any curve ˜s(.)∈Ω(t), we know its actual 3Dlocation ˜ S(.). Then, the calibration function F(u(t),t)is defined as F(u(t),t)≡∑ ˜s∈Ω(t)˜s distance(P(u(t)) ˜ S(.),˜s(q))2dq,(9) where distance(C,x)is the usual euclidean distance between a point xand a curve C. We observe that for any u(t),F(u(t),t)⩾0 and, the smaller F(u(t),t), the better is the matching between the 3D scene and its 2D image projection. The usual way to calibrate a camera (that is to obtain u(t)) is by minimizing the calibration function F(u(t),t).We also observe that F(u(t),t)is non convex, strongly nonlinear and, in general, we can not expect uniqueness of the minima because it is strongly dependent on the geometry of observable curves Ω(t)(in fact, in some cases Ω(t)could be empty). 4. Variational formulation of video calibration problem We observe that when we minimize the calibration function F(u(t),t)with respect to u(t)no assumption is made concerning the regularity in time tof u(t).Toaddsuch regularity condition to the calibration model we propose to minimize the functional I[w]=t1 t0 L(Dw(t),w(t),t)dt,(10) where [t0,t1]is the time interval, L(p,z,t)=p2+ α F(z,t)(11) is a weight to balance the different components of the energy and α ⩾0. Next we will show the existence of minimizer of I[.].Let be A=w∈W1,2((t0,t1);R3)such that w(t0)=(P0,T0,Z0)and w(t1)=(P1,T1,Z1). THEOREM 1. (existence of minimizer) There exists u∈Asolving I[u]=min w∈AI[w]. AVARIATIONAL APPROACH TO CAMERA MOTION SMOOTHING 559 Proof. To show the existence of minimizer we use the following classical result that, for instance, is presented in [2]. THEOREM A (see Evans [2, Theorem 5, p.453]) Assume that L satisfies the coercivity inequality L(p,z,x)⩾ α pq− β (12) for constants α >0, β ⩾0and q >1and is convex in the variable p.Suppose also the admissible set A is nonempty. Then there exists u ∈A solving I[u]=min w∈AI[w]. In our case, we observe that L(p,z,t)is convex with respect to p.On the other hand, since F(z,t)⩾0and α ⩾0,L(p,z,t)satisfies the coercivity inequality L(p,z,t)⩾p2 and therefore the condition (12) is satisfied with q=2and β =0.On the other hand obviously Ais not empty (Acontains simple linear functions). So then, statement of the Theorem 1 follows by straight application of the above result. 2 Next, we study if the minima of I[.]are solutions of the associated Euler-Lagrange system. Using the classical theory we need to show some growth conditions on L(p,z,t). First we observe that, in practice, the projection image is given by a rectangle [0,a]× [0,b]and then the observed curves Ω(t)are included in such rectangles. In practice, we are interested in estimating the distance function ”distance(P(z)˜ S(.),˜s(q))”when the curve intercepts the image rectangle [0,a]×[0,b].Therefore, without loss of generality, we can change the distance(.) function in (9) by distanceM(x,y)=distance(x,y)if distance(x,y)⩽M, Mif distance(x,y)>M, where M=√a2+b2.We define the modified calibration function as FM(z,t)≡Ω(t) distanceM(P(z)˜ S(.),˜s(q))2dq and we can state the following result. THEOREM 2. (solution of Euler-Lagrange system) If Ω(t)is composed by a finite number of curves which length uniformly bounded in [t0,t1],then the calibration function u(t)=(P(t),T(t),Z(t)) satisfies IM[u]=min AIM[w] and it is a weak solution of the system ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ −P(t)+ α∂ FM ∂ x1(u(t),t)=0in (t0,t1), −T(t)+ α∂ FM ∂ x2(u(t),t)=0in (t0,t1), −Z(t)+ α∂ FM ∂ x3(u(t),t)=0in (t0,t1), (13) 560 LUIS ALVAREZ,LUIS GOMEZ,PEDRO HENRIQUEZ AND LUIS MAZORRA where IM[w]=t1 t0 LM(Dw(t),w(t),t)dt, and LM(p,z,t)=p2+ α FM(z,t).(14) Proof. To show the result we use the following classical result that, for instance, is presented in [2]. THEOREM B (see Evans [2, Theorem 7, p.454] ) Assume L verifies the growth conditions ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ L(p,z,x)⩽Cpq+zq+1, DpL(p,z,x)⩽Cpq−1+zq−1+1, DzL(p,z,x)⩽Cpq−1+zq−1+1 (15) for constants C >0and q >1and u∈Asatisfies I[u]=min AI[w]. Then u is a weak solution of (13). In our case, we apply the above theorem to LMdefined in (14). First we observe that, since the length of curves of Ω(t)are uniformly bounded in [t0,t1],and the function distanceM(., .)is bounded then the function FM(z,t)is bounded in [t0,t1].Onthe other hand the two first components of vector zare angles so FM(z,t)is periodic with respect to zxand zyand it has bounded derivatives with respect to z.So we can deduce that there exists C>0 such that ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ LM(p,z,t)⩽Cp2+1, DpLM(p,z,t)⩽Cp, DzLM(p,z,t)⩽C, and therefore the statement of the theorem 2 follows by straight application of the above result. 2 5. Experiments To illustrate the performance of the proposed variational model we compare the results obtained for a real video sequence using the minimization of the calibration function F(u(t),t)independently for each time tand the results obtained with the proposed variational model. In figures 2-7 we show the plots of the obtained P(t), T(t)and Z(t)using both methods as well as their difference. To observe by visual inspection the quality of the variational method, in http://www.ctim.es/demo102 AVARIATIONAL APPROACH TO CAMERA MOTION SMOOTHING 561 Figure 2: P(t)obtained minimizing F(u(t),t)independently for each time t and P(t)obtained minimizing I[u] Figure 3: Plot of the difference between the estimation of P(t)using both methods 562 LUIS ALVAREZ,LUIS GOMEZ,PEDRO HENRIQUEZ AND LUIS MAZORRA Figure 4: T(t)obtained minimizing F(u(t),t)independently for each time t and P(t)obtained minimizing T[u] Figure 5: Plot of the difference between the estimation of T(t)using both methods AVARIATIONAL APPROACH TO CAMERA MOTION SMOOTHING 563 Figure 6: Z(t)obtained minimizing F(u(t),t)independently for each time t and Z(t)obtained minimizing I[u] Figure 7: Plot of the difference between the estimation of Z(t)using both methods