Hybrid CNN+LSTM for face recognition in videos

Departamento de Inform´atica Universidad T´ ecnica Federico Santa Mar´ ıa Proyecto de Tesis Mag´ıster en Ciencias de la Ingen´ıeria Inform´atica T´ıtulo del Proyecto de Tesis: “Hybrid CNN+LSTM for Face Recognition in Videos” Nombre del Alumno: Bellantonio Marco N´umero Tel´efono - Celular: +56 9 6565 1648 E-Mail: mad.b[email protected] Fecha de Ingreso al Programa: 1st August 2016 Pregrado: Bachelor Degree in Computer Engeneering (T´ıtulo o Grado, Instituci´on, A˜no) University of Bologna, 2014 Profesor Gu´ıa de Tesis: Prof. Ricardo Na˜nculef, Prof. Sergio Escalera Fecha Presentaci´on Tema de Tesis: 16th December 2016 Fecha Aprobaci´on Tema de Tesis: Fecha tentativa de T´ermino: Comisi´on interna de graduaci´on: Contents RESUMEN 2 ABSTRACT (Ingl´es) 4 1 FORMULACI ´ ON GENERAL DE LA PROBLEM´ ATICA Y PROPUESTA DE TESIS 6 1.1 Introduction....................................... 6 1.2 Deep Learning Methods for Video Face Recognition . . . . . . . . . . . . . . . . 7 CNN-basedMethods.................................. 7 Temporal Deep Learning Models . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.3 ProposedMethod.................................... 9 2 HIP´ OTESIS DE TRABAJO 11 3 OBJETIVOS 12 3.1 ObjetivosGenerales .................................. 12 3.2 ObjetivosEspec´ıficos.................................. 13 4 METODOLOG´ IA Y PLAN DE TRABAJO 14 4.1 Video Databases for Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . 14 4.2 ChosenDatasets .................................... 15 CMU Motion of Body (MoBo) Database . . . . . . . . . . . . . . . . . . . . . . . 15 YouTube Face (YTF) Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 UNBC-McMaster Shoulder Pain Expression Archive Database . . . . . . . . . . . 16 4.3 ImagePre-processing.................................. 17 4.4 Architecture Selection and Implementation Details . . . . . . . . . . . . . . . . . 17 Convolutional Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Long-ShortTermMemory............................... 18 4.5 Fine-tuning CaffeNet pretrained model . . . . . . . . . . . . . . . . . . . . . . . . 19 4.6 Dimensionality Reduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4.7 WorkPlan........................................ 21 5 RESULTADOS 22 5.1 Aportes y Resultados Esperados . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 5.2 FormasdeValidaci´on.................................. 22 6 RECURSOS 23 6.1 RECURSOSDISPONIBLES ............................. 23 6.2 RECURSOSSOLICITADOS ............................. 23 1 RESUMEN Debe ser suficientemente informativo, y contener una s´ıntesis del proyecto, sus objetivos, resultados esperados y palabras claves. (2 paginas) El reconocimiento de caras, junto con la identificaci´on de las acciones y gestos humanos, es en la actualidad una de las aplicaciones inform´aticas, m´as exitosas de anlisis automatizado del comportamiento humano. Durante los ´ultimos diez a˜nos aproximadamente, se ha convertido en un ´area muy popular de la investigaci´on en computer vision y ha recibido mucha atenci´on por parte de las organizaciones internacionales (Thumos, ChaLearn, etc). [1] El sistema de reconocimiento facial es una aplicaci´on creada para identificar o verificar una persona a partir de una imagen digital o un fotograma de v´ıdeo. ´ Estos, Verificaci´on eidentificaci´on son dos problemas muy distintos en el reconocimiento de los rostros. Los sistemas de verificaci´on tratan de responder a la pregunta “Es esta persona la que dice de ser?”. En este sistema, un individuo se presenta a s´ı mismo como una persona espec´ıfica, y el problema de verificaci´on se describe generalmente como un mapeo 1-a-1, donde el sistema intenta comparar la presencia de un individuo contra una informaci´on espec´ıfica del mismo individuo ya presente en el sistema. Un sistema de identificaci´on, por otro lado, tratan de responder a la pregunta “Qui´en es esta persona?”, Y su objetivo es identificar a una persona desconocida, comparando la informaci´on individual con la que ya est´an en el sistema de todos los dem´as. En otras palabras: la identificaci´on es un problema de clasificaci´on multiple descrito como un mapeo 1-a-n (donde n es el n´umero total de individuos en el sistema), mientras la verificaci´on, es una tarea de clasificaci´on binaria con par de ejemplos. En este proyecto se aborda el problema de la identificaci´on facial mediante t´ecnicas de aprendizaje profundo. El aprendizaje profundo es un ´area de estudio del aprendizaje autom´atico, estrictamente relacionado con las redes neuronales artificiales, cuya aproximaci´on al problema consiste en aprender representaciones de alto nivel de los datos mediante la parametrizaci´on de m´ultiples capas de procesamiento no lineal. Estas t´ecnicas permiten la extracci´on autom´atica de caractersticas en escenarios supervisados y no supervisados tanto para tareas de clasificaci´on como de regresi´on [29] [?]. Diferentes arquitecturas de aprendizaje profundo han sido utilizadas con ´exito en el reconocimiento de rostros [19, 26, 33, 50], en el reconocimiento de expresiones faciales [24,53] y en la detecci´on des las emociones [21,23,34]. Al igual que en muchas otras tareas de visi´on artificial, los datos de entrada para el reconocimiento facial pueden ser muy diferentes, incluyendo im´agenes, videos, mapas de profundidad [49] [30], im´agenes t´ermicas [48] [37], modelos 3D de la cara [5], entre otros. Por supuesto, el tipo de datos de entrada plantea diferentes limitaciones y oportunidades a nivel de modelaci´on. En el caso en que los datos de entrada son videos, parece natural que la informaci´on temporal deba ser explotado para realizar tareas de reconocimiento. De hecho, trabajos recientes confirman las ventajas de utilizar modelos temporales como Redes Neuronales Recurrentes (RNN) o Long-Short Term Memory (LSTM) para problemas de an´alisis de caras humanas, como la detecci´on y seguimiento de los rostros [52], el reconocimiento de la expresi´on facial [3] y el reconocimiento de emociones [11] [7]. Sin embargo, despu´es de una revisi´on exhaustiva de las fuentes bibliogr´aficas, llegamos a la conclusi´on que muy pocos trabajos han abordado el problema del reconocimiento facial usando modelos neuronales temporales, y ninguno de ellos se ha ocupado de reconocimiento de caras en los videos. En nuestra opini´on, esto representa una oportunidad interesante de investigaci´on con espacio para contribuciones originales. En esta tesis, se propone de abordar el problema de dise˜nar modelos de aprendizaje profundos adaptados para explotar la informaci´on temporal contenida en los videos, para el reconocimiento de rostros. En concreto, nos proponemos estudiar una arquitectura basada en la CNN-LSTM, utilizada con ´exito para otras tareas de an´alisis de v´ıdeo como el reconocimiento y 2 la descripci´on de objetos (image captioning) [10] [46], an´alisis de sentimiento [47] y clasificaci´on del texto [54], y comparar los resultados obtenidos con otros m´etodos de reconocimiento facial en estado del arte [16] [50] [42] [6]. Este trabajo se organiza en diferentes etapas. En primer lugar, se llevar´a a cabo una revisi´on exhaustiva de los trabajos m´as recientes en el campo de computer vision en relaci´on con los modelos de aprendizaje profundo, para el reconocimiento de caras en videos. En segundo lugar, queremos realizar un an´alisis preciso de los m´etodos m´as recientes y eficaces, junto con el estudio de los resultados observados hasta la fecha.Una vez reunida la informaci´on necesaria para estar informado sobre el estado de la arte, el siguiente paso importante ser´a la definici´on de las arquitecturas implicadas, Red Neuronal Convolutivas y Long-Short Term Memory, junto con la elecci´on de las bases de datos a utilizar para la validaci´on experimental. La disponibilidad de datos para el reconocimiento facial en v´ıdeos es grande. La m´as utilizada (y tambi´en la m´as dif´ıcil) es sin ninguna duda la colecci´on Youtube Face (YTF). Sin embargo, en este trabajo se propone tambi´en construir una nueva base de datos a partir de la conocida Motion of Body database (MoBo). La colecci´on MoBo DB est´a destinada a ser utilizada en tareas de detecci´on y reconocimiento de movimientos. Por lo tanto, las im´agenes de las que se compone son fotos de cuerpo entero de varios temas. En nuestro proyecto aplicamos t´ecnicas de procesamiento de im´agenes para detectar el rostro, recortar la regi´on de la cara y almacenar la imagen resultante en un formato adecuado. La nueva base de datos ser´ıa una contribuci´on adicional importante de este trabajo. Al dise´no de la arquitectura y la elecci´on de las bases de datos seguir´a la aplicaci´on y un conjunto de experimentos. 3 ABSTRACT (Ingl´es) Debe ser suficientemente informativo, y contener una s´ıntesis del proyecto, sus objetivos, resultados esperados y palabras claves. Debe ser equivalente al RESUMEN. (1 pagina) Face recognition, along with human action and gesture recognition, is nowadays one of the most successful application of automated human behaviour analysis. Over the last ten years or so, it has become a very popular area of research in computer vision and has received a lot of attention from international organizations (THUMOS, ChaLearn, etc) [1]. A facial recognition system is a computer application capable of identifying or verifying a person from a digital image or a video frame from a video source. These tasks, verification and identification, are two very distinct problems in face recognition. Verification systems seek to answer the question “Is this person who they say they are?”. Under a verification system, an individual presents himself or herself as a specific person, and the verification problem is generally described as a 1-to-1 matching where the system tries to match the presence of the individual against a specific information of the same individual already present in the system. Identification systems, on the other hand, seek to answer the questions “Who is this person?”, and aim to identify an unknown person by checking the individual information against all others already in the system. In this project, we address the problem of face identification with the use of a deep learning framework. Recent advances in facial analysis using deep learning frameworks such as Convolutional Neural Networks (CNN) or Deep Belief Networks (DBN) allow recognition even from low-level features such as pixels by the composition and parametrization of many levels of data processing [34]. These architectures have been widely used in face recognition [19, 26, 33, 50], facial expression recognition [24,53], emotion detection [21,23,34]. As in many other computer vision tasks, input data for face recognition can be very different, including raw images, videos, depth maps [49] [30], thermal images [48] [37], 3D face models [5], etc. Of course, the type of input data pose different constraints and opportunities at the modelling level. Specifically in videos, it may be apparent that temporal information should be exploited to perform recognition tasks. Indeed, recent successful works confirm the advantage of using temporal models such as Recurrent Neural Networks (RNN) and Long-Short Term Memory models (LSTM) for human face analysis problems, such as face detection and tracking [52], facial expression recognition [3] and emotion recognition [11] [7]. However, after an intensive literature review, we conclude that very few works have addressed the problem of face recognition using temporal neural models and none of them dealt with face recognition in videos. In our opinion, this represents an interesting research opportunity for original contributions. In this thesis we propose to address the problem of designing deep learning models tailored to exploit the temporal information contained in videos to perform face recognition. Concretely, we propose to study a CNN-LSTM based architecture successfully used for other video analysis tasks, such as object recognition and description (image captioning) [10] [46], sentiment analysis [47] and text classification [54] to mention few, and to compare the obtained results with other state-of-the-art face recognition methods [16] [50] [42] [6]. This work will be organized in different phases. First of all, an exhaustive review of recent papers and works in the field of computer vision related to deep models for face recognition in videos will be performed. Secondly, we plan to prepare a precise analysis of the most recent and efficient methods along with the study of the performances reported and the databases used. After having gathered the information necessary to be informed and aware of the state of the art, the next important step will be the definition of the architectures more directly involved in this research, namely Convolution Neural Network and Long-Short Term Memory, along with the choice of the databases for validating the proposal. The availability of data for video face recognition is big. However, in this work we contribute also by building a novel face database from the well known Motion of Body (MoBo) database. 4 After the design of the architecture and the choice of the databases, the implementation and a set of experiments would follow. 5 1 FORMULACI´ ON GENERAL DE LA PROBLEM´ ATICA Y PROPUESTA DE TESIS Debe contener la exposici´on general del problema, identificando claramente qu´e aspectos relacionados con la inform´atica son los m´as relevantes. Adem´as, deber´a contener el marco te´orico, la discusi´on bibliogr´afica con sus referencias y, finalmente, su propuesta de tesis. (La extensi´on m´axima de esta secci´on es de hasta 5 p´aginas. En hojas adicionales incluya la lista de referencias bibliogr´aficas citadas) 1.1 Introduction Accurately identifying people has always been a very human process. It is a task that we perform routinely and effortlessly in our daily lives. In the past 30 years, the wide availability of powerful and low-cost computers have aroused an enormous interest in automatic processing of digital images, in a variety of applications, including human-computer interaction, surveillance, biometric authentication, multimedia management, and so on and so forth. Research and development in automatic face recognition have followed naturally. As one of the most successful applications of image analysis and understanding, face recognition has recently gained significant attention and achieve astounding performances. Recent technologies i.e. the Facebook AI Lab FR systems are able to recognize face with an incredible accuracy of more than 97%. In computer vision, one of the most widely used and successful methods for image processing are Convolutional Neural Network (CNN). CNNs are biologically-inspired variants of Multi Layer Perceptrons proposed by Yann LeCun in 1998 [25]. Inspired by the biological functioning of the visual system, CNNs exploit spatiallylocal correlation by enforcing local connectivity pattern between units (neurons) of adjacent layers. In CNN each filter is replicated and locally shares the parametrization (weights). From a procedural point of view, CNN outputs (heatmaps) are obtained by ”convolving” the input (images or previous layers) with a linear filter, adding a bias term and then applying a non-linear function. Input pixels xij are multiplied by filter components (weights ω) and summed up. Finally, a nonlinear function σis applied. A formalization of this process is shown in equation 1 underneath: yij =σ m−1 X a=0 m−1 X b=0 ωabx(i+a)(j+b)!(1) Where aand bare parameters which determine the size of the convolution filters. In the field of computer vision, CNNs are fed with input images and have units (neurons) arranged in 3 dimensions, respectively width, height and depth. In order to extend CNNs to the video domain and to capture temporal information, the most widely used approach consist in extending the convolution along the temporal axis in what is well known as a 3D Convolutional neural network. The 3D convolution captures discriminative features along both spatial and temporal dimensions and are nowadays the most used by many approaches for human behaviour analysis where temporal information is available. Generally speaking, one of the most used network for temporal analysis are Recurrent Neural Networks (RNN). RNNs can take into account the temporal information by using recurrent connections in hidden layers and can deal with sequences of variable length by defining a recurrent relation over timesteps according to the formula St=σ(WxXt+WrSt−1) (2) where Stand Xtare respectively the state and the input at time t,St−1represents the state at the previous timestep, Wris the so-called transition matrix and Wxare the weights parameters 6 in feed-forward networks. The weights matrices Wxand Wrare filters that determine how much importance both the present input and the past hidden state have. The final output of the network Ytat a certain time step tis typically computed from one or more states St−1..St+1. The recent revived interest on RNN is mainly attributed to its recent success in many practical applications such as language modeling [35], speech recognition [4] [15], machine translation [43] [20] and conversation modeling [38], to name a few. But although RNN can be trained through time to learn and memorize what happened in the past, they are characterized by a very short-term memory, which is insufficient for real long-term world applications. To solve this problem (as well as the problem of the vanishine or exploding of the gradient) Long Short-Term Memory (LSTM) [12] was proposed. LSTMs are a particular implementation of Recurrent Neural Network proposed in 1997 by German researchers S. Hochreiter and J. Schmidhuber, usually used as a hidden layer of RNN. But unlike most RNNs, LSTM networks are well-suited to learn to classify, process ad predict time series from very long time windows (up to 1000 discrete time steps), generally of unknown size. LSTMs contain information outside the normal flow of the recurrent network using gated cells. Information can be stored in, written to, or read from a cell, similarly to how data are treated in a computers memory. The cell makes decisions about what to store and when to allow reads, writes and erasures, via gates that open and close. Those gates are called input gate, forget gate and output gate. This work aims to exploit the temporal information contained in consecutive video frames by using a LSTM. More specifically, the input of the LSTM network are not directly video frames, rather features extracted using a CNN. This process would hopefully lead to good results in a face recognition problem and also improve the performances of the CNN alone, providing an interesting case study for future extensions. 1.2 Deep Learning Methods for Video Face Recognition This section examines in more detail how deep learning methods have been used to address the challenges of face recognition, spanning from image quality to unconstrained scenario, from change in pose to occlusions. Li et al. [27] propose a deep hierarchical version of the PEP model, called Hierarchical Probabilistic Elastic Part-Model, to approach unconstrained face recognition problems. In order to build pose-invariant part-based face representations, faces are decomposed into parts using PEP model hierarchically. From top-down in the hierarchy, the H-PEP model builds pose-invariant face representation for both images and videos. Following in the hierarchy from bottom to up, face part representations are stacked at each layer. By aggregating FPR layer by layer, the method is able to build compact and pose invariant face representations. In 2014 Goswami et al. [14] presented a memorability based frame selection algorithm that enables automatic selection of memorable frames for facial feature extraction and matching. A deep learning algorithm was proposed to utilizes a stack of denoising Autoencoders and deep Boltzmann Machines and perform face recognition using the most memorable frames. This work provided the idea to use Autoencoders in order to perform dimensionality reduction of the method presented in this work. Further details will be presented in section 4.6. CNN-based Methods Many recent studies have reported the success of using deep CNN in face related tasks. The already cited work by Taigman et al. [44] called DeepFace is based on a very deep CNN architecture together with an alignment technique. The authors revisited face alignment and representation by employing explicit 3D face modeling in order to apply piece-wire affine transformation and derive a face representation from a 9-layer deep neural network. A particularity of the network is that it involves more than 120M parameters using several locally connected 7 layers without shared weights, rather that the standard convolutional neural network. Figure ?? illustrate the alignment pipeline process, whereas figure ?? shows the 9-layer architecture. Inspired by GoogLeNet, Sun et al. [40] used a very deep CNN network with multiple levels of supervision, called Deep hidden IDentity features (DeepID), which reaches human-level face verification performance in the LFW dataset by achieving 97.45% accuracy. DeepID features are built on top of the feature extraction hierarchy of a deep CNN (last hidden layer neuron activations). The proposed features are extracted from various face regions to form complementary and over-complete representations. Recently in 2016, Yang et al. [51] presented a Neural Aggregation Network (NAN) for video face recognition which takes a face video or face image set of a person with variable number of face frames as its input, and produces a compact and fixed dimension visual representation of that person. The whole network is composed of two modules. The feature embedding module is a CNN which maps each face frame into a feature representation. The neural aggregation module is composed of two content-based attention blocks which are driven by a memory storing all the features extracted from the face video through the feature embedding module. The output of the first attention block adapts the second, whose output is adopted as the aggregated representation of the video faces. Due to the attention mechanism, this representation is invariant to the order of the face frames. Important the work of Parkhi et al. [33], in which they made two important contributions: first, they designed a procedure to assemble a large scale dataset; secondly, they trained a deep CNN achieving results comparable to state-of-the-art methods. Sun et al. in 2015 [42] proposed to learn high-performance deep ConvNets with sparse neural connections called Sparse ConvNets. Sparse ConvNets are learned in an iterative way, each time one additional layer is “sparsified” and the entire model is re-trained given the initial weights learned in previous iterations. Important novelty is a new neural correlation-based weight selection criterion which empirically verifies its effectiveness in selecting informative connections from previously learned models at each iteration. Temporal Deep Learning Models Among many variants of RNNs, Long Short-Term Memory (LSTM) is arguably one of the most widely used. Other than supervised learning, LSTM is also used in recent work in image generation [45] [16], demonstrating its capability of modeling statistical dependencies of imagery data. LSTM are also widely applied to time series prediction, speech and handwriting recognition, music composition and human action recognition. In the literature we can find few methods which use RNN and LSTM for many different problems involving human faces. Yoo et al. [52] presented a new robust algorithm that improved face detection and tracking in video sequences by using geometrical facial information and a recurrent neural verifier. In particular they defined a new method called Three-Face Reference Model (TFRM) which brings the advantage of a better match process. Other authors such as Graves et al. [3] proposed a new approach for facial expression recognition which combines state-of-the-art techniques for model-based image interpretation and sequence labeling. The Candide-3 face model is used in conjunction with a learned objective function for face model fitting and then the resulting sequence of model parameters are presented to a Long-Short Term Memory for classification. The classification algorithm is explicitly designed to consider sequences of data as well as the temporal dynamics of facial expressions. Recently in 2016, Ebrahimi et al. [11] proposed an hybrid CNN-RNN architecture for facial expression analysis and emotion recognition in videos. The authors assert that spatio-temporal evolution of facial features is one of the strongest cues for emotion recognition. The proposed approach uses temporal averaging for aggregation and outperforms other modalities. Again in 2016, Chao et al. [7] presented a multi-modal (Audio-visual-physiology) approach to dimensional emotion recognition with a LSTM-RNN architecture. In their work they investigate -insensitive loss function (instead of squared loss) and temporal pooling. From their work we know that -insensitive loss function is more robust 8 ND-Flip-QO, UMD ComCast10, ESOGU Face Videos , MAHNOB-HCI, MMSE-HR and Trailed Face Dataset. Most of the aforementioned databases do not contain videos or are defined for specific problems. For this reason they are not suitable for the purpose of this research. 4.2 Chosen Datasets CMU Motion of Body (MoBo) Database The MoBo database contains 25 individuals walking on a treadmill in the CMU 3D room. The subjects perform 4 different activities: slow walk, fast walk, incline walk and walking with a ball. All subjects are captured using 6 high resolution color cameras distributed evenly around the treadmill. The database contains a total of 600 videos, 340 frames each makes 204,000 video frames. The dataset is challenging for its profile and semi-lateral camera views, where the face is partially visible due to the tilt of the head. In order to evaluate and fairly compare the proposed model, we gather all the papers which use the MoBo data set for face recognition. In table 3, for each method we report the reference, the face region (if extracted), the splits of the database used to evaluate the method and the accuracy along with the specific metric. The protocol follows always the same idea: one activity for training and the remaining three activities for testing. Only in one method the authors split the database into 2 subset without taking into account the number of activities contained in the split. Table 3: MoBo methods Paper Face Region Protocol Accuracy Towards Large-Scale Face Recognition Based on Videos - 1 train / 3 test 98.1% (CR) Learning Personal Specific Facial Dynamics for Face Recognition From Videos 40x40 1 2train / 1 2test 97.9% Joint sparse representation for video-based face recognition 30x30 1 train / 3 test 96.5% (IR) Face Recognition Based on Image Sets 40x40 1 train / 3 test 95.3, 98.1(CR) From Still Image to Video-Based Face Recognition: An Experimental Analysis 40x40 1 train / 3 test 92.3% (RR) Notes: RR: Recognition Rate, IR: Identification Rate, CR: Classification Rat The Motion of body database was designed to be used for motion detection and recognition problems, thus it contains full body pictures of the subjects. In order to extract the face region from each frame, a pre-processing step is necessary. Mobo DB pre-processing will be presented in section 4.3. YouTube Face (YTF) Database YoutTube Face is a database of face videos designed for studying the problem of unconstrained face recognition in videos. It contains 3425 videos of 1595 people (average of 2.15 videos for each subject). Considering that the video clip lengths vary from 48 to 6070 frames (average of 181.3 frames/video), we have approximately 620,000 frames. From the formal definition, YTF is a verification dataset. The standard verification protocol from main reference is described as follow: 1. Randomly collect 5000 videos pairs, half are pairs of videos of the same person, half of different people. 15 2. Pairs are divided into 10 splits. Each split contains 250 same and 250 not-same pairs. Pairs are divided ensuring that the split is subject-mutually exclusive. Subject appears in one split does not appear in anyone else. 3. 9 splits for training and 1 for testing. The Youtube Face Database contains a large number of subjects and the actions performed are naturally varied (as opposed to performing prescribed actions). It is easier to acquire, thus allowing the baselines to be used by the research community at large. All subjects also have still images available in the Labeled Faces in the Wild (LFW) database [18], thus allowing baselines to be compared to the video to still image matching scenario. The main challenging part is the low image quality: frames sequences of YouTube videos are generally worse than web photos, mainly because of motion blur or viewing distance. As for the MoBo database, we collected in table 4 the methods which use YTF to perform face recognition. In the table we report the work, the protocol used, the evaluation metric used to evaluate their method along with the obtained results. Table 4: YTF methods Paper Protocol Metric Result DeepID2+ [41] Standard protocol ACC 93.2% (VR) 95% (IR) DeepFace [44] Standard protocol (unrestricted) ACC 91.4% (CR) 100%-EER 92.5% Eigen-PEP for video face recognition [28] Standard protocol ACC 85.4% Face Recognition in Movie Trailers via Mean Square Sparse Representation-based Classification [31] Standard protocol ACC 75.3%, AUC 82.9% EER 25.3% Hierarchical-PEP model for real-world face recognition [26] Not specifically defined ACC 87% MDLFace [14] 3M face images of 50K identities ACC 97.9% Neural Aggregations Networks [50] 100 frames for each video ACC 96.5% (IR) AUC 98.7% Sparsifying Neural Network Connections [42] Train: 290K faces; Val: 47K faces; Test: 5K pairs of faces ACC 93.5% (RR) Unconstrained Face Recognition [6] Own gallery (YTF+LFW) + fusion ACC 79% Notes: ACC: Accuracy, AUC: area under the curve, EER: Equal Error Rate, RR: Recognition Rate, IR: Identification VR: Verification , Rate, CR: Classification Rate UNBC-McMaster Shoulder Pain Expression Archive Database UNBC-McMaster is a pain expression database collected by researchers at McMaster University and University of Northern British Columbia. The database contains facial video sequences of participants who had been suffering from shoulder pain and were performing a series of active and passive range of motion tests to their affected and unaffected limbs on multiple occasions. The database was originally created by capturing facial videos from 129 participants (63 males and 66 females). The participant had a wide variety of occupations and ages. During data capturing the participants underwent eight standard range-of-motion tests: abduction, flexion, and internal and external rotation of each arm. At present, the UNBC-McMaster database contains 200 video sequences of 25 subjects. As the description suggests, the database was thought for pain detection or estimation, therefore it is really challenging because of the changing 16 of the face expression due to the shoulder pain. Additionally, it also provides enough materials to perform face recognition. In our work it will be used as a additional dataset, given that in the literature there are no methods which use it to perform face recognition. If successful, we may consider this work as a baseline for future comparison. 4.3 Image Pre-processing In order to adapt the Motion of Body (MoBo) dataset to our problem, a pre-processing phase is necessary. From each frame the face is detected using a state-of-the-art face detector. Moreover, the face is cropped using the relative coordinates of the detected face. In some cases the face detector fails due to the tilt of the head. In those cases the face region is interpolate from the previuos frame. Each cropped face region is finally saved as new JPG image. The face detector (available at http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html) is called dlib Real-Time Face Pose Estimation, implementation of an excellent paper from the 2014 CVPR Converence [22]. In the following we will show an example of face detection and face region extraction, with the final storing of the resulting image. > python face detector.py im02 19451807.jpg processing file: im02 19451807.jpg number of faces detected: 1 detection position left,top,right,bottom: 232 122 275 166 > convert im02 01444804.jpg -crop $position -resize 224x224 im02 01444804 cropped.jpg 4.4 Architecture Selection and Implementation Details The objective of this work is to understand whether a hybrid LSTM+CNN network can improve the performances of a CNN-only based model. In this section we present the choice of the two architectures and the implementation details. Convolutional Neural Network Convolutional networks (CNNs) currently set the state of the art in visual recognition. The design of the CNN is based on one of the recent best-performing models , namely VGG-VeryDeep-16 CNN (VGG-16) [39]. From its formal definition, the VGG-16 inputs are fixed-size 17 224 ×224 RGB images. Figure 2 illustrates the VGG-16 network architecture, with precise information about the convolutional and pooling layers. Figure 2: VGG-16 CNN Here we can notice that the input images have to be of size 244 ×244, and that the last fully connected layer has a dimensionality of 4096 elements. Therefore the input of the LSTM would be a 1 ×4096 vector. For this work we chose a pre-trained model, trained from scratch using 2.6 Million images of celebrities collected from the web. The CNN descriptors are computed as described in [33]. For our purpose we need to use the CNN fully connected layer as input for a LSTM, therefore no further modification of the CNN classifier is necessary. In order to improve the performances, a fine-tunning phase will also be performed and explained in section 4.5. CNN Implementation Details The code and the VGG-16 pre-trained model is publicly available from the University of Toronto website [2]. The main reference offers the face descriptor source code and the models for Matlab, Torch and Caffe. Using Caffe, the code to obtain the output of a pre-trained mode is really straightforward. In order to read the deploy file and the already precomputed weights, caffe offers the function caffe.Net(model, weights). The network is fed with each image and a forward step is performed: net.blobs[’data’].data[...] = transformer.preprocess(’data’, img) out = net.forward() The output could be finally stored in a HDF5 file with outputs.append(h5py.File(outputFile + ’.h5’, ’w’)) Long-Short Term Memory The basic LSTM model, originally proposed by Hochreiter and Schmidhuber [17], is called Vanilla LSTM. As obvious, in the literature we can find different versions of LSTM, accordingly defined for specific needs. One popular variation, introduced by Gers & Schmidhuber in 2000 [13], is built by adding “peephole connections”, allowing the gate layers to look at the cell state. Otte et al. (2014) [32] improved the convergence speed of the LSTM by adding recurrent connections between the gates of a single block (but not between the blocks) in what they call a Dynamic Cortex Memory (DCM). Always in 2014, Sak et al. [36] introduced a linear projection layer that projects the output of the LSTM layer down before recurrent and forward connections in order to reduce the amount of parameters for LSTM networks with many blocks. A more drastic variation of the basic LSTM is the Gated Recurrent Unit (GRU) introduced by Cho, et al. (2014) [8]. This model combines the forget and input gates into a single update gate, and 18 it also merges the cell state and hidden state, with some other minor changes. The resulting model is simpler than standard LSTM models and its popularity has been growing increasingly in these past two years. Figure 3 shows the main differences between a LSTM block and a GRU block. Figure 3: CNN+LSTM system As we can see, the input gate i, forget gate fand output gate opresent in the LSTM are replaced by the reset gate rand the update gate zin the GRU block. It is also relevant to mention other few notable LSTM variants, such as Depth Gated RNNs by Yao, et al. (2015) and Clockwork RNNs by Koutnik, et al. (2014). In order to understand the differences in such a great number of RNN variations, Greff, et al. (2015) did an exhaustive comparison of the most popular ones, finding that they are all about the same. Also Jozefowicz, et al. (2015) tested many variation of RNN architectures, finding that some work better than LSTMs on certain tasks. From both researches we can conclude that: 1. The basic Vanilla LSTM is generally more efficient than any normal RNNs. 2. Other variations, built for specific problems, are not worth for our ojective. 3. Dropout is necessary and it often improves performances. 4. Learning rate and network size are the most crucial tunable LSTM hyperparameters. The LSTM used in this project is one-layer Basic Vanilla LSTM. We already know from the CNN definition that the input of the LSTM are 4096 ×1 vector. LSTM Implementation Details There are several possible frameworks for the implementation of the LSTM. The most used and known are Caffe, Keras, Lasagne, TensorFlow, Theano, Torch. A detailed research and final comparison of the best and most efficient frameworks for the desing of the LSTM is important. Table 5 shows a detailed comparison of the five most used frameworks for deep learning. For each framework we report the base language, the GPU support availability, the recurrent neural network design-ability and the compilation time efficiency. Table 5: Frameworks comparison Framework API GPU RNN fit Compile time Theano Python+Numpy Yes* Good Slow for large models Torch Lua Yes Not good Acceptable TensorFlow Python & C++ Yes Good Slow Caffe Python & C++ Yes Not good Slow Keras Python Yes Good Acceptable * No multi-gpu by default 4.5 Fine-tuning CaffeNet pretrained model In order to improve the performances of the CNN from its original pre-trained model, a fine-tuning phase is necessary. Fine-tuning takes an already learned model, adapts the architecture and resumes training 19 from the already learned model weights on a different dataset. First of all we need to download the train val and solver prototxt files provided by the author of the same pre-trained model, in our case the VGG16. These files contain information about the architecture with setup parameters useful for the finetuning process, i.e. learning rate multiplier, dropout probability, momentum, etc. By modifying the aforementioned configuration file, we replace the last layer of the CNN by a randomly initialized fully-connected layer with the correct number of face labels to recognize. Moreover, we set the learning rate of the fully connected layer as ten times the learning rate of the rest of the CNN and we set the global learning rate to one tenth of the original one. From the train val prototxt we notice that the input dataset is in Lightning Memory-mapped Database (LMDB) format file. For this reason, a function to convert our images into a LMDB file is necessary. After setting up the solver and the caffe prototxt, the model needs to be trained for few epochs, until the convergence of the losses is reached. The final fine-tune command is: caffe train --solver=$SOLVER --weights=$CAFFEMODEL Once fine-tuned, the new model is used to extract the features of the fc7 layer for each input images and feed the Long-Short Term Memory (LSTM) Recurrent Neural Network (RNN). 4.6 Dimensionality Reduction The fully connected fc7 layer of the VGG-16 network produces a 4096 dimensional vector. This vector is input of the LSTM. One may claim that a 4096 feature vector is too big to be efficiently treated by a LSTM. To tackle this point, some extension of the system are proposed to reduce the feature dimensionality. Figure 4: Extended system with Dimensionality Reduction (RD) Block In particular we analysed: 1. Principal Component Analysis (PCA): is a multivariate statistical procedure that is often useful in identifying patterns in high-dimensional data or in reducing dimensionality. PCA can be usefully used to convert the fc7 feature vector in a lower dimensional vector, saving space and computational time. However, PCA does not take into account class information when calculating the principal components. Therefore, especially in cases when the differentiating characteristics of the classes are not reflected in the variance of the variables, PCA may not be a good choice of data processing. 20 2. Multi Layer Perceptron: a small ANN module may takes the fc7 feature vector (output of the CNN VGG-16) and produce a new (smaller) feature representation, which would be the new input of the LSTM. It would be trained to classify the video sequences frame-by-frame and, as for the pre-trained CNN, feature vectors would be extracted from the last fully connected layer, where the feature abstraction is higher. 3. Convolutional Autoencoders: Autoencoders are artificial neural networks used in unsupervised learning to produce input data representations (encoding), typically for the purpose of dimensionality reduction. An useful version of autoencoders is called Convolutional Autoencoders, which extends the encoding process to two dimensions. Here the standard steps are: input →convolution →pooling →unpooling →deconvolution →error mearure. 4.7 Work Plan Figure 5 shows a temporal organization of the work flow that I propose to follow in order to accomplish the specific tasks of which my project is composed. Some of those tasks have been already achieved and are presented in this work. Figure 5: Work plan 21 5 RESULTADOS 5.1 Aportes y Resultados Esperados The aim of this work is to investigate the performances of a hybrid Recurrent-based deep method for video face recognition. In addition, we investigate whether a LSTM network can improve the performances of a CNN architecture. Thus, the contributions to the community would be a detailed analysis of the performances of a new hybrid deep temporal framework for video face recognition, along with an exhaustive investigation about how and in which measure temporal information can improve the performances of a CNN model. 5.2 Formas de Validaci´on In order to validate our method and to compare it with other state-of-the-art method, the choice of the evaluation metric and validation strategy is important. Table 6 show the most widely-used evaluation metrics for face recognition. Table 6: Evaluation Metrics Metric Definition Usage Error Rate # of misclassifications # samples in val set General accuracy evaluation F1 Score 2×true positive (2×true positive)+false negative+false positive Used to give a summary of the Precision-Recall (PR) curve. ROC / PR curve Precision = true positive true positive+false positive Recall = true positive true positive+false negative Used to show the overall performances of an algorithm as its discrimination threshold is varied. In order to evaluate the performances of the system we decided to calculate the Error Rate and the F1 score, which give us respectively an estimation of the accuracy and an overall knowledge of the precisionrecall curve. To calculate the F1 score, the computation of the confusion matrix is necessary. The confusion matrix is a specific table that allows the visualization of the performance of an algorithm for a supervised learning. Each column of the matrix represents the predicted classes while each row represents the actual class according to the ground truth. Table 6 shows the calculation of the true positive and true negative from the confusion matrix. Figure 6: confusion Matrix 22 6 RECURSOS 6.1 RECURSOS DISPONIBLES Se˜nale medios y recursos con que cuenta el Departamento de Inform´atica de la UTFSM, para realizar el proyecto de tesis (libros, software, laboratorios, etc.). This research project will be carried out at the University of Catalunya in Spain. So nothing is requested to the Federico Santa Maria University 6.2 RECURSOS SOLICITADOS Se˜nale medios y recursos no disponibles en el Departamento de Inform´atica de la UTFSM, necesarios para realizar el proyecto de tesis (libros, software, laboratorios, etc. ). Su extensi´on no debe exceder el espacio disponible This research project will be carried out at the University of Catalunya in Spain. So nothing is requested to the Federico Santa Maria University 23 References [1] Face recognition homepage. http://www.face-rec.org/general-info/. [2] Vgg face descriptor - university of oxford. http://www.robots.ox.ac.uk/ vgg/software/vggface/. [3] M. W. J. S. Alex Graves, Christoph Mayer and B. Radig,Facial expression recognition with recurrent neural networks, International Workshop on Cognition for Technical Systems, (2008). [4] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio,End-to-end attentionbased large vocabulary speech recognition, CoRR, abs/1508.04395 (2015). [5] B. Ben Amor, K. Ouji, M. Ardabilian, and L. Chen,3D Face recognition by ICP-based shape matching, in The second International Conference on Machine Intelligence (ACIDCA-ICMI’2005), Nov. 2005. [6] L. Best-Rowden, H. Han, C. Otto, B. F. Klare, and A. K. Jain,Unconstrained face recognition: Identifying a person of interest from a media collection, IEEE Transactions on Information Forensics and Security, 9 (2014), pp. 2144–2157. [7] L. Chao, J. Tao, M. Yang, Y. Li, and Z. Wen,Long short term memory recurrent neural network based multimodal dimensional emotion recognition, in Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, AVEC ’15, New York, NY, USA, 2015, ACM, pp. 65–72. [8] K. Cho, B. van Merrienboer, C¸ . G¨ ulc¸ehre, F. Bougares, H. Schwenk, and Y. Bengio,Learning phrase representations using RNN encoder-decoder for statistical machine translation, CoRR, abs/1406.1078 (2014). [9] D. C. Corra, D. H. P. Salvadeo, R. L. M. Levada, J. H. Saito, N. D. A, E. Moreira, and S. Carlos,Using lstm network in face classification problems. [10] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell,Long-term recurrent convolutional networks for visual recognition and description, CoRR, abs/1411.4389 (2014). [11] S. Ebrahimi Kahou, V. Michalski, K. Konda, R. Memisevic, and C. Pal,Recurrent neural networks for emotion recognition in video, in Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI ’15, New York, NY, USA, 2015, ACM, pp. 467–474. [12] F. A. Gers, N. N. Schraudolph, and J. Schmidhuber,Learning precise timing with lstm recurrent networks, JMLR, 3 (2002), pp. 115–143. [13] F. A. Gers, N. N. Schraudolph, and J. Schmidhuber,Learning precise timing with lstm recurrent networks, J. Mach. Learn. Res., 3 (2003), pp. 115–143. [14] G. Goswami, R. Bhardwaj, R. Singh, and M. Vatsa,Mdlface: Memorability augmented deep learning for video face recognition, in Biometrics (IJCB), 2014 IEEE International Joint Conference on, IEEE, 2014, pp. 1–7. [15] A. Graves, A. Mohamed, and G. E. Hinton,Speech recognition with deep recurrent neural networks, CoRR, abs/1303.5778 (2013). [16] K. Gregor, I. Danihelka, A. Graves, and D. Wierstra,DRAW: A recurrent neural network for image generation, CoRR, abs/1502.04623 (2015). [17] S. Hochreiter and J. Schmidhuber,Long short-term memory, Neural computation, 9 (1997), pp. 1735–1780. [18] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller,Labeled Faces in the Wild: A Database forStudying Face Recognition in Unconstrained Environments, in Workshop on Faces in ’Real-Life’ Images: Detection, Alignment, and Recognition, Marseille, France, Oct. 2008, Erik Learned-Miller and Andras Ferencz and Fr´ed´eric Jurie. [19] Z. Huang, R. Wang, S. Shan, and X. Chen,Face recognition on large-scale video in the wild with hybrid euclidean-and-riemannian metric learning, Pattern Recognition, 48 (2015), pp. 3113–3124. [20] S. Jean, K. Cho, R. Memisevic, and Y. Bengio,On using very large target vocabulary for neural machine translation, CoRR, abs/1412.2007 (2014). 24