A Mobile Platform for Movement Tracking Based on a Fast-Execution-Time Optical-Flow Algorithm

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 1 A mobile platform for movement tracking based on a fast-execution-time optical-flow algorithm Rafael de la Rosa-Vidal, Juan A. Le˜ nero-Bardallo, Member, IEEE, Jos´ e-Mar´ ıa Guerrero-Rodr´ ıguez, and ´ Angel Rodr´ ıguez-V´ azquez, Life Fellow, IEEE Abstract—A multi-purpose mechanical platform to track moving objects in three-dimensional space has been developed. It is composed of one main microcontroller board that processes all system data, two cameras, three motors, and one secondary microcontroller board to position a platform with three degrees of freedom. The system computes the optical flow and moves the cameras accordingly, tracking motion within the visual scene. The platform operates autonomously. To the best of our knowledge, there are no similar systems reported with low-resolution image sensors and low-cost microcontrollers. Existing solutions rely on personal computers and advanced FPGAs to process image data. This article concludes that the optical flow operation is efficient even using an image sensor with very low resolution. Thus, the system complexity and image data processing are alleviated significantly. The platform can be easily adapted to different application scenarios by adding new peripherals, sensors, or image processing algorithms. A detailed description of the system design and experimental results are provided. Index Terms—Mobile platform, object tracking, camera positioning, optical flow, image processing. I. INTRODUCTION MULTIPLE applications require to control the position of cameras to track moving objects. For instance, drone vision [1], [2] or automotive systems [3], [4] may require to stabilize the camera position to record static images of moving objects. Several authors have also devised sensors and instruments for space navigation that need precise positioning when they navigate. For instance, solar sensors or star trackers have to be continuously oriented directly to the sun or stars to control the attitude of space navigation systems [5], [6]. In parallel, the rise of Artificial Intelligence (AI) has developed many other vision systems [4], [7]–[9] that, processing camera data outputs, can identify moving objects This work was supported by Proyectos de I+D+i DE entidades p´ ublicas – Convocatoria 2020 P20 01206 (VERSO), by Ayudas a Proyectos de I+D+I Programa Operativo FEDER through Project US-1264940 (SPADARCH), by Proyecto Singular de Transferencia del Conocimiento: Ecosistema Innovador con Inteligencia Artificial para Andaluc´ ıa 2025 RIS3 through Project CEI07, by Spanish Government MINECO and European Regional Development Fund, (ERDF/FEDER) through Project RTI2018-097088-B-C31, and by ONR grant ONR NICOP N00014-19-1-2156. Rafael de la Rosa-Vidal was supported by the Spanish Government through Ayudas para la Formaci´ on del Profesorado Universitario (FPU) under Grant FPU 01561. Rafael de la Rosa-Vidal, Juan A. Le˜ nero-Bardallo, and ´ A. Rodr´ ıguezV´ azquez are with the Institute of Microelectronics of Seville (IMSE-CNM), CSIC-Universidad de Sevilla, Av. Am´ erico Vespucio, 28, 41092, Seville, Spain, (E-mails: {rdvidal, jlenero, arodri-vazquez}@us.es). Jos´ e M. Guerrero-Rodr´ ıguez is with the University of C´ adiz, Campus Universitario de Puerto Real, 11519 C´ adiz, Spain, (E-mail: [email protected]) Manuscript received XXX, 2021; revised December, 2021. or classify elements in the visual scene. Once the relevant scene visual elements have been identified, the corresponding actuators can be activated to navigate or exchange data with other systems. Camera positioning to track mobile objects is a complex problem whose difficulty increases exponentially when there is more than one image sensor involved in the operation [10]. Since image quality is not a must in these scenarios where fast and efficient object recognition is the priority, different image sensor architectures are being investigated. Nowadays the spread of event-driven asynchronous image sensors [11], [12] open enticing possibilities to refine the operation by reducing the computational cost [13]. Looking at nature for inspiration, biological organisms, with single or compound eyes, can track moving objects by identifying the optical flow variations. The concept of optical flow detection to estimate movement within the visual scene by biological organisms was proposed in the 1940s by Gibson, [14]. Afterward, many algorithms to compute it were developed by many authors [15]–[18]. However, classical optical flow algorithms have inherently high computational cost because they require processing and storing multiple image frames [16]–[18]. For this reason, realtime execution needs high-speed processing that would limit their implementation in many scenarios that demand lowpower systems. To the best of our knowledge, existing tracking systems based on optical flow computation or movement detection have been implemented on high-performance platforms, using personal computers, utilizing high-level programs to process high image data throughput [1], [3], [10], [19]. That is a limitation to deploy positioning systems on other autonomous systems like drones, satellites, robots, etc. that cannot cope with a high payload and subsystems with highpower consumption. To solve these limitations, we have implemented a compact platform with reduced payload and power consumption. Furthermore, previous authors employ medium or high-resolution arrays to improve the image quality, at the expense of increasing the computational cost. However, as will be discussed in the article, for an optimum scene interpretation, it is not strictly necessary a large number of pixels. Some biological organisms like dragonflies have complex eyes compounded of simpler eyes that capture only a portion of the visual scene [19]–[21]. Previously, the authors presented a live demo with a optical flow tracker [22]. It was an early version of the proposed system with one image sensor, one movement IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 2 Fig. 1. 3D system model and different ortogonal views of it rotated ±45°. degree of freedom, and much less-elaborated mechanical implementation than the one that will be described in this article. The system description was not provided. Extending this previous work, we report design insights about a new moving platform with stereo vision intended to track moving objects employing two low-resolution image sensors, a refined physical design with three degrees of freedom, and lower latency. We have adapted image processing algorithms to the system requirements. Those can be implemented on classic microprocessor-architectures and adapted to process data from image sensors with different nature and applicability. We demonstrate that the operation can be performed with very low-resolution image sensors, involving a very low computational cost that can be executed on a low-cost microcontroller. To the best of our knowledge, tracking involving the optical flow computation by means of the algorithms reported has never been implemented with a microcontroller with low computational capabilities and lowresolution image sensors. II. SYSTEM IMPLEMENTATION OVERVIEW A concept of the system implementation is depicted in Figure 1. The platform can position two cameras intended to track moving objects. Movement is detected by computing the optical flow variations [16]–[18], [23] within the visual scene. The field of view of the two cameras is different to achieve the possibility of implementing stereoscopic vision algorithms to estimate rotations and depth within the visual scene [24]. The two cameras are always moved solidarily. There are three independent motors to position the system with three degrees of freedom (Pitch,Roll and Yaw). The cameras assembly describes a spherical movement similar to a body head movement. There is a microcontroller STM32L476RG that processes the image sensors data. Its mission is to compute the optical flow and determine how the system has to be positioned. There is a dedicated IMU MPU6050 to track the motors positions and move them accordingly. III. OPTICAL FLOW DETECTION ALGORITHM The optical flow can be defined as the apparent movement of luminous intensity patterns within a frame [16], assuming that luminance variations inside the frame are only due to the displacement of such patterns within the frame [25]. It is a relative movement between the objects of the visual scene and an observer [14], [25], [26]. Thus, it can provide very useful information to locate and monitor the position of elements in the space through a frame sequence [27]. Biological organisms compute it to navigate and get across the environment [20], [21]. Although the concept of optical flow computation was proposed many decades ago [14], the first functional algorithms to computed it are more recent [16]–[18], [23]. Traditionally, optical flow computation has been limited by the strong computational requirements associated with real-time algorithm execution. Some authors have already devised systems that track moving objects by computing the optical flow variations [19], [28]. However, to the best of our knowledge, such systems require large computational capabilities that limit their autonomous implementation without using computers. In many cases, complex convolution operations, filtering, storage, and processing of several frames, etc. are required processing steps. Those increase the computational load remarkably with high-resolution pixel arrays. The basis for the optical flow computation is the motion constraint equation [25]. The light intensity on an instant t of a pixel in the (x, y)coordinates is given by the equation I(x, y, t). If the pixel displaces a distance ∆xy∆yin the Cartesian plane during a time interval ∆t, the new pixel intensity is given by the function I(x+ ∆x, y + ∆y, t + ∆t). Since I(x, y, t)and I(x+ ∆x, y + ∆y, t + ∆t)correspond to pixels with identical intensity values, the movement equation, I(x, y, t) = I(x+ ∆x, y + ∆y, t + ∆t), must be satisfied. This assumption is valid for first order approximations where ∆x,∆yy∆tare small values. Among the algorithms to compute the optical flow, three different families of algorithms were considered: •Differential techniques. They are based on spatiotemporal derivatives computation of image intensity (i.e. Lucas-Kanade [18] and Horn-Schunck [16]). •Frequency and phase based methods. They are based on the velocity-tuned filters response applied to the frames (i.e. Heeger [29] and Waxman [30]). •Region-based matching. Those based on the searching of the velocity vector which minimizes the error between two successive frames (i.e. Srinivasan [23]). For the proposed system implementation, a simplified version of Srinivasan’s algorithm [23] proposed by the authors was selected. The algorithm has interesting advantages over the previous ones to embed it on a microcontroller: •It does not require feature detection inside the frame. •The optical flow is computed with only one iteration. •It does not require spatial or temporal filtering. These two operations have a high computational cost. •The computation is robust to the image noise in nonsynthetic images. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 3 Fig. 2. Example illustrating the bidimensional functions fl, fr, fu, and fd when considering a frame with 8×8 pixels. A subframe with 6×6 pixels defined by the window function Ψ (x, y)represents a static region of interest. The following values for the displacement parameters were set: ∆xref = ∆yref = 1 To explain the Srinivasan’s algorithm operation, let us define a uniform spatial window Ψ (x, y)that selects a region of interest (group of pixels) inside a frame. The pixels intensities inside the window are defined by the function f(x, y). The plane movement in the xand y-directions within a time interval ∆tis given by c ∆xand c ∆y. These two terms represent the optical flow value in each direction. For convenience, let us also denote the function that gives the initial pixel intensity values (before the movement starts at t= 0) as f0(x, y). Also, the functions fl(x, y, ∆t)and fr(x, y, ∆t)will be defined. They represent the pixel intensities values after a time interval ∆tand a frame displacement over the x-axis of a value xref , i.e.: fl(x, y, ∆t) = f0(x+ ∆xref , y) fr(x, y, ∆t) = f0(x−∆xref , y)(1) Likewise, the functions fu(x, y, ∆t)and fd(x, y, ∆t)are defined as the pixel intensities after a frame displacement yref over the y-axis: fu(x, y, ∆t) = f0(x, y −∆yref ) fd(x, y, ∆t) = f0(x, y + ∆yref )(2) For simplicity, we will denote the functions of Equations 1 and 2 as f, f0, fr, fl, fu, fd. For illustrative purposes, in Figure 2, there is a representation of the different bidimensional functions (fl,fr,fu, and fd) that are created after a frame displacement in the xand the y-directions with the reference values xref =yref = 1. The frame has a dimension of 8×8 pixels, and the window function Ψ (x, y)defines a subframe with 6×6 pixels representing a static region of interest. Let us assume that the time interval ∆tis very short and during it, the pixel intensities do not change. Thus, f(x, y, ∆t)is just a translation of the original function f0(x, y, t). Under this assumption, it is possible to express f(x, y, ∆t)as a function of fl,fr,fu, and fd, as it is depicted by Equation 3. \ f(x, y, t)is an interpolated version of f(x, y, t). b f=f0+1 2 c ∆x ∆xref !(fr−fl) + 1 2 c ∆y ∆yref !(fu−fd) (3) The target is to obtain the values of c ∆xand c ∆ythat minimize the error between \ f(x, y, t)and f(x, y, t). These parameters correspond to the optical flow value. The error minimization between fand b fis calculated by optimizing the least quadratic error in the frame region of interest defined by the window function Ψ(x, y). For a frame with M×Npixels, the error, E, is: E= M−1 X x=0 N−1 X y=0 Ψ(x, y)·hf(x, y)−b f(x, y)i2(4) Where Mis the number of pixel rows and Nis the number of pixel columns in the frame. In the previous equations, ∆xref and ∆yref are algorithm parameters. They have pixel units and establish the algorithm sensitivity to the optical flow variations. For low-resolution pixel arrays like the one selected in the proposed system implementation, a value of ∆xref = ∆yref = 1 is adequate. This choice also reduces the execution time because no extra calculations are needed when dividing by ∆xref or ∆yref . For larger pixel arrays, ∆xref and ∆yref values must be increased to keep the same algorithm sensitivity to the optical flow variations. Substituting Equation 3 in Equation 4, the Equation 5 is derived. To minimize the error, the partial derivatives functions referred to c ∆xand c ∆yare forced to be equal to zero, leading to the equation system composed by Equation 6 and Equation 7. E= M−1 X x=0 N−1 X y=0 (Ψ (x, y)·(f−"f0+1 2 c ∆x ∆xref ! fr−fl!+ 1 2 c ∆y ∆yref ! fu−fd!#)2) (5) IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 4 c ∆x ∆xref !· M−1 X x=0 N−1 X y=0 Ψ (x, y)·(fr−fl)2 + c ∆y ∆yref !· M−1 X x=0 N−1 X y=0 [Ψ (x, y)·(fu−fd)·(fr−fl)] = 2 · M−1 X x=0 N−1 X y=0 [Ψ (x, y)·(f−f0)·(fr−fl)] (6) c ∆x ∆xref !· M−1 X x=0 N−1 X y=0 [Ψ (x, y)·(fr−fl)·(fu−fd)] + c ∆y ∆yref !· M−1 X x=0 N−1 X y=0 Ψ (x, y)·(fu−fd)2 = 2 · M−1 X x=0 N−1 X y=0 [Ψ (x, y)·(f−f0)·(fu−fd)] (7) The values of c ∆xand c ∆yresult from solving the equation system defined by Equations 8 and 9. They provide the optical flow value in the region defined by Ψ(x, y). c ∆x= 2 ·C·D−B·E A·D−B2·∆xref (8) c ∆y= 2 ·A·E−B·C A·D−B2·∆yref (9) where, A= M−1 X x=0 N−1 X y=0 Ψ (x, y)·(fr−fl)2(10) B= M−1 X x=0 N−1 X y=0 [Ψ (x, y)·(fr−fl)·(fu−fd)] (11) C= M−1 X x=0 N−1 X y=0 [Ψ (x, y)·(f−f0)·(fr−fl)] (12) D= M−1 X x=0 N−1 X y=0 Ψ (x, y)·(fu−fd)2(13) E= M−1 X x=0 N−1 X y=0 [Ψ (x, y)·(f−f0)·(fu−fd)] (14) These are coefficients defined by the authors that can be calculated while the sensor frame is readout, facilitating the entire system pipeline operation. There are two situations where the system equation is undetermined: •The entire frame is exposed to the same illumination value. Consequently, all the pixels intensities are theoretically the same. In this case, all the equation coefficients are null because fr=fl=fu=fd. •The frame only has one-dimension features (vertical or horizontal). For instance, this can happen whether there is STM32L476RG - NUCLEO EyeOF Module EyeOF Module SPI1 peripheral SPI2 peripheral UART PWM/RC Eyes OF Gimbal Platform App Gimbal Controller STorm32 BGC Fig. 3. Diagram showing the system connectivity among the different hardware elements. only an horizontal (fr=fl) or a vertical line (fu=fd) in the frame. In such cases, there are multiple solutions for the system equation leading to the well-known Aperture Problem [31]. Such situations are detected by examining the parameter values. Optical flow computation is discarded whether the denominator in Equations 8 and 9 is close to zero. However, this case is unlikely because frames incorporate Fixed Pattern Noise (FPN) added by the image sensor that benefits the algorithm operation. IV. HARDWARE IMPLEMENTATION In Figure 3, the different hardware elements are shown. The kind of connectivity between them to exchange information is depicted in the diagram. The main component is the STM32L476RG-NUCLEO microcontroller development board from ST Microelectronics. It controls the rest of the modules. It sends/receives data and configuration instructions to/from a PC through a UART interface. It is also connected to two modules called Eye of Optical Flow (EyeOF) that acquire raw images from the visual scene. Finally, the STM32L476RG microcontroller drives using PWM signals three motors with a STorm32 BGC Gimbal controller. The STM32L476RG-NUCLEO development board integrates as the main device an ARM®Cortex®-M4 core that can operate at 80 MHz consuming 26 mW. It has been conceived for low-power applications and presents multiple SPI interfaces to connect different peripherals. It allows Direct Memory Access (DMA) from the peripherals without using the CPU. With this functionality, pipeline operation between data transfer and data processing is implemented to reduce the system latency. A. Image acquisition module Demonstrating that it is feasible to compute the optic flow efficiently with very low-resolution images was one target in this work. With this aim, the low resolution ADNS2610 sensor was selected. This family of sensors was popular a few years IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 5 FD (a) (b) Fig. 4. (a) 3D design of the module EyeOF. (b) Detail of the optics assembled to it. FD is the focal distance from the optics to the sensor. ago. They were implemented in optic mouses to track the movement of a light source that was solidary with the mouse movement. The sensor is compounded of a low-resolution 18×18 image sensor and a DSP that can implement basic image processing operations. In this work, the ADNS2610 sensor just senses illumination values in the visual scene. Then its data is transmitted to the STM32L476RG microcontroller to compute the optical flow with the algorithm described. To host the ADNS2610 sensor and its optics, a custom PCB was designed and fabricated. We called it Eye of Optical Flow (EyeOF). The EyeOF module has a 4-wire connector to communicate the sensor with a microcontroller, power the sensor, and to send or receive configuration parameters. In Figure 4.(a), there is a 3D representation of the EyeOF module and its main components. A lens holder can be attached to the EyeOF module (see Figure 4.(b)) to easily mount and remove the sensors optics. B. Image acquisition and optical flow computation A Finite State Machine (FSM) was implemented on the microcontroller to control the EyeOF modules in charge of the image acquisition. In Figure 5, there is a diagram illustrating the different transitions between states in the FSM. To speed up the optical flow calculation and to avoid dead times between state transitions, all the required operations are performed while the microcontroller is awaiting data from the EyeOF modules. We provide a brief description of the operations conducted by the FSM: •SENSOR RESET. The EyeOF module and all its data registers are initialized. The module operation starts from SENSOR_RESET TRIGGER_FRAME REQ_READ_ FRAME READING_FRAME PROCESSING RESET RESET RESET RESET !PIXEL_STATUS !RESET !RESET PIXEL_STATUS PIXEL_QTY<323 PIXEL_STATUS && PIXEL_QTY==323 !RESET Fig. 5. Diagram of the FSM implemented in the microcontroller for image acquisition and processing. this state that is accessible by all the other FSM states. •TRIGGER FRAME. The ADNS2610 sensor is prepared to initiate a frame acquisition. The operation requires to write one control bit in a EyeOF configuration register and wait for the EyeOF module to be prepared to acquire a new frame. In the meantime, all the parameters from the EyeOF module, to be monitored in the user interface, can be transferred to the DMA. The frame pixels will be readout one by one in the upcoming two states. •REQ READ FRAME. In this state, a requirement to readout a pixel is sent to the EyeOF module. In the meantime, if the previous pixel was already readout, the pixel status is checked. If possible, some operations related to the optical flow computation are performed with the previous readout pixels. •READING FRAME. A register containing the pixel data is readout. The number of readout pixels is updated. If there are pending pixels, the FSM comes back to the previous REQ READ FRAME state. If the entire frame has been readout, the FSM machine moves to the next state, PROCESSING. •PROCESSING. At this state, the two frames required to compute the optical flow are available. Since only two frames are required for the computation, the oldest frame stored in memory is discarded. A register that stores the memory position that corresponds to the new frame and the previous one is updated. The optical flow values are determined and used to correct the motor position. After completing these operations, the FSM returns to the initial TRIGGER FRAME state. All the FSM previous steps are repeated to acquire a new frame, compute the optical flow, and update the motors’ position. The maximum frame rate that can be achieved is limited by IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 6 TABLE I TIME REQUIRED FOR THE DATA READING OPERATIONS WITH THE ADNS2610 IMAGE SENSOR. Operation Time required Time to await between pixels consecutive readings, t1. 50 µs Time to send a pixel reading requirement and to readout the pixel value, t2. 600 µs Time to prepare the image sensor to acquire pixel data and initiate the pixels readout operation, t3. 700 µs 17 35 53 71 89 107 125 143 161 179 197 215 233 251 269 287 305 323 16 34 52 70 88 106 124 142 160 178 196 214 232 250 268 286 304 322 15 33 51 69 87 105 123 141 159 177 195 213 231 249 267 285 303 321 14 32 50 68 86 104 122 140 158 176 194 212 230 248 266 284 302 320 13 31 49 67 85 103 121 139 157 175 193 211 229 247 265 283 301 319 12 30 48 66 84 102 120 138 156 174 192 210 228 246 264 282 300 318 11 29 47 65 83 101 119 137 155 173 191 209 227 245 263 281 299 317 10 28 46 64 82 100 118 136 154 172 190 208 226 244 262 280 298 316 9 27 45 63 81 99 117 135 153 171 189 207 225 243 261 279 297 315 8 26 44 62 80 98 116 134 152 170 188 206 224 242 260 278 296 314 7 25 43 61 79 97 115 133 151 169 187 205 223 241 259 277 295 313 6 24 42 60 78 96 114 132 150 168 186 204 222 240 258 276 294 312 5 23 41 59 77 95 113 131 149 167 185 203 221 239 257 275 293 311 4 22 40 58 76 94 112 130 148 166 184 202 220 238 256 274 292 310 3 21 39 57 75 93 111 129 147 165 183 201 219 237 255 273 291 309 2 20 38 56 74 92 110 128 146 164 182 200 218 236 254 272 290 308 1 19 37 55 73 91 109 127 145 163 181 199 217 235 253 271 289 307 0 18 36 54 72 90 108 126 144 162 180 198 216 234 252 270 288 306 13 31 49 67 85 12 30 48 66 8 4 11 29 47 65 8 3 10 28 46 64 8 2 9 27 45 63 8 1 8 26 44 62 8 0 7 25 43 61 7 9 6 24 42 60 7 8 5 23 41 59 7 7 4 22 40 58 7 6 3 21 39 57 7 5 2 20 38 56 7 4 1 19 37 55 7 3 0 18 36 54 7 2 (a) (b) Fig. 6. a) Table with pixels indexes: in blue, there are the indexes whose data can be used to perform any optical flow computation when they are readout. In red, there are the ones that require to wait for additional data to make a computation. b) Indexes required to calculate the partial sums. Illustration of the order they can be employed to make calculations. the fact that pixels values has to readout one by one. However, the FSM machine performs all the optical flow computations in parallel while pixels are readout avoiding dead time. Thereafter acquiring a frame, the motors position can be updated, and a new frame can be acquired. In Table I, the amount of time required for the data reading operations with the ADNS2610 image sensor is reported. Using this information, the amount of time to readout a frame is computed: tacq = (t1+t2)·M·N+t3= 211.3 ms (15) FPS =1 tacq = 4.732 ≈5fps (16) A maximum frame rate of 5 fps can be achieved with the selected image sensor. This value limits the system latency to track fast moving objects. Such value qualifies to track moving light sources that are far away, i.g. the sun, and objects that do not move fast. C. Optical flow algorithm implementation The implementation of the Srinivasan’s optical flow algorithm [23] has been devised to avoid a bottleneck in the device operation. There are several computational steps Common Plane Fig. 7. Spatial arrangement for the EyeOF modules. In red, the reference coordinate systems for each EyeOF module; in blue, the reference coordinate system for the optical flow vectors fusion. performed in parallel with the pixel readout operations. Thus, there are pipeline operations continuously executed by the microcontroller to speed up the system operation. The algorithm implementation is based on the use of the information provides in the tables of Figure 6.(a-b). The two frames required for the optical flow computation are stored on two one-dimensional arrays with M×N=18·18 elements. To compute the optical flow, the coefficients A,B, C,Dand E, given by Equations 10, 11, 12, 13, and 14, must be calculated. Each coefficient results from the partial sum of M×Nelements. These elements are calculated while the pixel values of the new frames are being readout. To calculate the partial sums, the pixel intensity values are arranged in the memory as it is depicted in Figure 6.(a). For simplicity, in the plots, we have assumed that ∆x= ∆y=1. Pixels are always readout in order, starting from the pixel labeled as 1 and finishing with the pixel labeled as 324. In Figure 6.(a), we classify with a color code the indexes. In red, there are the indexes that cannot be immediately used to compute a partial sum when the pixels that represent are readout. Observing the terms of the coefficients equations, it can be understood that until a pixel neighborhood is not readout, any partial sum computation is possible. In blue, there are marked the indexes that can lead to an immediate partial sum computation when they are readout. In Figure 6.(b), it is illustrated how the different partial sums can be calculated while the different pixels are readout sequentially in the direction the arrow indicates. For the first partial sum computation, the indexes 19, 18, 20, 1, 37 are required; for the second partial sum, the indexes 21, 20, 22, 3, 39 and required; and so on. It can be noticed that to calculate the first partial sum, the first 37th pixels have to be readout. For this reason, the first 36th indexes are marked in red color in Figure 6.(a). In the proposed system implementation (Figure 7), two coplanar cameras are mounted. This is the minimum amount IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 7 Pitch Motor Roll Motor Yaw motor Radio Control Signals External IMU DC power USB to PC Motor drivers On-board IMU Microcontroller Fig. 8. STorM32 BGC control board. Its main functional modules are highlighted. of sensors necessary to implement a three-dimensional object tracking that detects visual scene rotations referred to one axis. Moreover, by increasing the number of cameras, the field of view value is increased. The results of processing the frames captured with each image sensor are two vectors. They indicate the direction and the magnitude of the optical flow: −→ vleft = [∆xleft,∆yleft]Tand −→ vright = [∆xright,∆yright]T. Then two vectors are combined leading to: •An optic flow vector that is the result of of averaging the optical flow vectors provided by each image sensor, i.e., −→ vfused = −→ vleft+−→ vright 2, as it is depicted in Figure 7. •The two image sensors are aligned in the x-plane. Hence, the variation of the y-coordinates for the two optical flow vectors, ∆yfused = ∆yleft −∆yright provides information about the visual scene rotation referred to the platform x-axis. V. MOTORS AND CONTROLLERS Three 2208 brushless DC motors were selected to position the platform. They operate at 90 KV rotation-torque ratio and have 14 poles. To drive the three motors, the 3-axis STorM32 BGC controller board were selected. The main sub-components and interconnecting pin-outs and are shown in Figure 8. The upper connections are intended for the pitch, roll, and, yaw motor control. At the bottom, there is the power supply connection. We fed the board with a DC power supply with 12 V and 5 A (60 W). On the left side, their connections to three elements: a) Connections to control signals to modify the system position. b) An external Inertial Measurement Unit (IMU) with a gyroscope and accelerometer to determine the module’s position. This IMU must be allocated in the same plane as the motors and it is connected to the STorM32 BGC device with a twisted pair. c) An USB port to configure the MPU 6050 module STorM32 BGC controller Pitch Roll Yaw The slots allow balancing the gimbal z yx Inertial frame Body frame Fig. 9. Platform mechanical design: 3D structural model. board from a PC. In the center of the controller board, there is another on-board IMU to detect the board’s position when it is installed in moving surfaces, i.e. in a drone. A Graphical User Interface (GUI) is available to tuning the STorM32 BGC microcontroller. VI. MECHANICAL DESIGN In Figure 9, there is a detailed 3D view of the platform design. The mechanical structure is based on the Gimbal system concept. This type of structure is usual for system camera stabilization. For the platform mechanical design, the 3D modeling software Fusion 360®from AutoDesk was used. This software allows integrating the prior PCB design from the EyeOF modules depicted in Figure 4 with the system mechanical structure. The interaction of the moving structure with the system elements: wires, motors, optics, EyeOF boards was studied in detail before manufacturing. The different platform parts were fabricated with a 3D printer. For the representation of the position and attitude of a moving object in a three-dimensional space, an inertial reference system was considered. It is fixed and it does not move during the platform operation. Additionally, a local reference system is defined. Its coordinate origin is the system center of gravity. Its axes are disposed as it is depicted in Figure 9 and it is not inertial. Over the local reference system, the attitude of a moving object is defined by quaternions. The first quaternion component represents a rotation angle and the other three components define the axis that is rotated the angle defined by the first vector component. For the right platform operation, it must be mechanically balanced. This implies that the load’s center-mass must be exactly allocated along the rotation axis for each motor. Consequently, the load will be fixed at an optimum equilibrium position to speed up the displacement. Otherwise, vibrations of the gimbal system and consequently added overheating of IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 8 Frame on (k-1) Motors on the platform Frame on k IMU module Delay (To wait platform establishment) PID Controller STorM32 BGC controller Optical flow computation It implements PID controllers internally x y RC pitch RC roll RC yaw Pitch Roll Yaw Position control loop Tracking control loop Fig. 10. Diagram with the implemented control loops. the corresponding motor could also occur. To facilitate the balancing process, slots are present in some parts to balance the center-mass of the pitch axis, as it is depicted in Figure 9. The roll axis will be balanced because the loads allocated over it are identical. For the yaw axis, the equilibrium condition is not so critical because the platform will be held in a vertical position during its operation. Hence, there is not any relevant momentum; all the relevant forces are longitudinal to the motor rotation axis. In Figure 1, the 3D design of the entire platform is shown. The movement range is sufficient for the proposed application scenarios. VII. CONTROL LOOP IMPLEMENTATION This section describes the system configuration process, its stabilization, and how its parameters are trimmed. The first step to initialize the STorM32 BGC board is to calibrate the IMU that has been placed next to the EyeOF modules, as it is depicted in Figure 9. The controller GUI is used to define the IMU orientation. With the calibration process, undesired offset values from the gyroscopes measurement are canceled. The control law is based on the software implementation of three PID regulators to position every specific axis: pitch, roll, and yaw, according to the control loop diagram illustrated in Figure 10. The regulator input is the error between the target position and the current one. The KP,KI, and KD PID parameters were adjusted as described in Listing I. Their values can directly be set using an available microcontroller GUI. The parameters KPand KDare responsible for the dynamic platform response. They have selected to achieve a critically damped response without oscillations. Once the platform is stabilized, the modules EyeOF mounted on it move, tracking the scene visual flow variations. VIII. USER INTERFACE IMPLEMENTATION To debug the system and monitor its outputs a custom GUI was implemented. This GUI, shown in Figure 11, was programmed with the Microsoft WPF technology and .NET Core framework [32]. Currently, the WPF technology is integrated in .NET Core that aims to be a multiplatform environment compatible with Linux or MacOS. These programming tools allow creating refined and modern user interfaces that can exchange data and commands with custom systems of diverse nature. The GUI’s purpose is to monitor Listing I: PID control parameters adjustment. 1) All the control outputs are disabled, excepting the one corresponding to the pitch axis. Under this configuration, the following adjustments are performed: a) The KDparameter is gradually increased until the system starts vibrating at high frequency. Then the KDvalue is decreased until the system vibrations stop. It must be checked that there are no system vibrations for any of its possible axis positions. b) The KIparameter value is increased to the minimum possible value above zero. For the STorM32 BGC microcontroller, this value is 5. c) The KPvalue is increased until the system starts oscillating at low frequency. At this point, the KPvalue is reduced until the system is stable. Again, it must be checked that there no vibrations in any axes position. d) The KIvalue is increased until the system becomes unstable. In this situation, the motor position in the axis will vary randomly. Then, the KIvalue must be reduced until reaching the stability again. 2) The motor that controls the roll axis is activated. The steps 1.(a-d) are repeated. 3) The motor that controls the yaw axis is activated. The steps 1.(a-d) are repeated. 1 2 3 4 5 6 7 8 Fig. 11. Eyes OF Gimbal Platform custom interface. The main interface buttons and functionalities are numbered in the plot. the platform operation, debug it, and illustrate how it works. It must be remarked that the system is autonomous and does not require to be connected to the GUI to operate. In Figure 11, the custom user interface is shown. Its name is Eyes OF Gimbal Platform. The interface can represent simultaneously the two frames rendered by the EyeOF modules. In Figure 12, the face of one of the author is rendered. Although the image sensors pixel resolution is limited (18×18 pixels), some parts of the face can be easily distinguished. In practical situations, we do not target to identify objects within the visual scene. The aim is to detect the optical flow variations provoked by moving objects and track them. The interface allows to represent arrows vectors over the rendered frames (see Figures 11 and 12). This possibility leads to a very intuitive dynamic representation of optical flow IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. XX, NO. XX, AUGUST 2021 9 Fig. 12. Frames with a face captured with the EyeOF modules. In the left one, the optical flow is not shown. In the right one, the result of the optical flow computation and the system position correction to track the movement are represented with two colored arrows, red and green respectively. computation results, indicating its direction and magnitude. Two colored arrows are displayed over the incoming frames providing information continuously. This representation is necessary to debug the system. The red arrow is the result of the optical flow computation. The arrow length is proportional to the optical flow magnitude. The green arrow indicates the necessary system movement correction to capture a frame equal to the previous one taken as a reference. Several commands and operations are accessible through the GUI interface. In Figure 11, the most representative windows and interface buttons are numbered. We list them: •Sections to display the frames acquired by the image sensors, 1 , and 2 . •Button to start or stop the platform operation, 3 . •Button to start computing the optical flow taking an initial frame as a reference, 4 . •Button to calibrate the platform ( 5 ). This operation consists of storing a reference frame to track the movement referred to it. •Arrows buttons, 6 , to force the platform movement in different directions, i.e., South, North, East, and West. These functions are useful to set an initial camera position in the center of the visual scene. •Tracking mode button, 7 . In this operation mode, the platforms track moving objects within the visual scene through the optical flow computation. •Several items, labeled as 8 that display the numerical value of the optical flow computation. IX. EXPERIMENTAL RESULTS The entire system was characterized. Table II summarizes the main system specifications. A photograph of the final system implementation is shown in Figure 13. The entire system is powered with a power supply source operating at 12 V with a power of 60 W. The system tracking capability was tested in our laboratories. The system can easily track walking people from a distance of 1.5 meters or higher from the platform. Also, its applicability to track bright light sources and walking people was verified. Fig. 13. Photograph showing the final moving platform implementation. TABLE II TRACKING PLATFORM MAIN FEATURES. Functionality Optical flow computation and object tracking Microcontroller for data processing STM32L476RG, ARM Cortex-M4 @ 64 MHz Microcontroller for motor positioning STorm32 BGC Gimbal, ARM 32-bit Cortex-M3 @ 72 MHz Motors Three 2208 brushless DC motors, 90 KV, 14 poles Dimensions 160 mm ×125 mm Image sensor resolution 18 ×18 pixels Field of view 100° Latency 450 ms Frame rate 5 fps Power consumption 270 mA@12 V Degrees of freedom Three (Pitch,Roll, and Yaw) Stereo vision Yes Scalability Possible to add multiple image sensors Optional frame postprocessing Yes 0 200 400 600 800 1000 1200 1400 0 0.25 0.5 0.75 1 Degrees Pitch 0 200 400 600 800 1000 1200 1400 0 0.25 0.5 0.75 1 Degrees Roll 0 200 400 600 800 1000 1200 1400 Time (ms) 0 0.25 0.5 0.75 1 Degrees Yaw Fig. 14. Experimental motors’ unit step-response.