Deep learning based face beauty prediction via dynamic robust losses and ensemble regression

Knowledge-Based Systems 242 (2022) 108246 Contents lists available at ScienceDirect Knowledge-Based Systems journal homepage: www.elsevier.com/locate/knosys Deep learning based face beauty prediction via dynamic robust losses and ensemble regression F. Bougourzi b, F. Dornaika a,c,d,∗, A. Taleb-Ahmed e aSchool of Computer and Information Engineering, Henan University, Kaifeng, China bInstitute of Applied Sciences and Intelligent Systems, National Research Council of Italy, 73100 Lecce, Italy cUniversity of the Basque Country UPV/EHU, San Sebastian, Spain dIKERBASQUE, Basque Foundation for Science, Bilbao, Spain eUniv. Polytechnique Hauts-de-France, Univ. Lille, CNRS, Centrale Lille, UMR 8520 - IEMN, F-59313 Valenciennes, France article info Article history: Received 12 October 2021 Received in revised form 20 December 2021 Accepted 19 January 2022 Available online 31 January 2022 Keywords: Facial beauty prediction Convolutional neural network Deep learning Ensemble regression Robust loss functions abstract In the last decade, several studies have shown that facial attractiveness can be learned by machines. In this paper, we address Facial Beauty Prediction from static images. The paper contains three main contributions. First, we propose a two-branch architecture (REX-INCEP) based on merging the architecture of two already trained networks to deal with the complicated high-level features associated with the FBP problem. Second, we introduce the use of a dynamic law to control the behaviour of the following robust loss functions during training: ParamSmoothL1, Huber and Tukey. Third, we propose an ensemble regression based on Convolutional Neural Networks (CNNs). In this ensemble, we use both the basic networks and our proposed network (REX-INCEP). The proposed individual CNN regressors are trained with different loss functions, namely MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey. Our approach is evaluated on the SCUT-FBP5500 database using the two evaluation scenarios provided by the database creators: 60%–40% split and five-fold cross-validation. In both evaluation scenarios, our approach outperforms the state of the art on several metrics. These comparisons highlight the effectiveness of the proposed solutions for FBP. They also show that the proposed dynamic robust losses lead to more flexible and accurate estimators. ©2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). 1. Introduction For centuries philosophers, artists, and scientists have tried to discover the mystery of beauty [1]. In fact, the beauty of the face is gaining more and more interest due to the rapid development of plastic surgery and cosmetic industry [2]. In recent years, facial beauty estimation and classification has become an interesting research topic in computer vision and machine learning due to its growing applications [3,4]. Applications for facial beauty estimation and prediction include: Cosmetic recommendations [5], scheduling of aesthetic surgeries [6], facial beautification [7], and Social Networks Services (SNS) (such as Facebook, Instagram, and dating websites) [8]. In addition, automatic facial beauty prediction (FBP) may find application when attractiveness is a basic requirement, such as in advertising, magazine covers, and screening applicants for certain jobs, such as entertainment and modelling [6]. Despite the considerable progress in estimating and predicting the beauty of faces, more labelled data is needed for training deep ∗Corresponding author at: University of the Basque Country UPV/EHU, San Sebastian, Spain. E-mail address: [email protected] (F. Dornaika). CNNs. To deal with the data limitation, some researchers used active data augmentation. In addition, others used pre-trained models trained on the ImageNet database [9]. These pre-trained models are capable of extracting high-level features. In this paper, we propose a system that exploits the diversity of learners. We present two main proposals. First, we propose to combine two different CNN architectures into a single architecture (called the two-branch architecture) that is trained end-to-end. Second, we propose to build an ensemble of regressions where the final prediction is given by the average of all predictions. The latter solution does not need to be trained on new validation sets. More specifically, we propose ensemble regressions using one-branch architectures (ResneXt-50 and Inception-v3) and our proposed two-branch architecture (REX-INCEP) trained with different loss functions. Four loss functions are used in our approach, namely MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey. In summary, the main contributions of this paper are as follows: •We propose ParamSmoothL1 regression loss function ParamSmoothL1. Moreover, we introduce a dynamic law that changes the parameters of the robust loss function during training. For this purpose, we use the cosine law https://doi.org/10.1016/j.knosys.2022.108246 0950-7051/©2022 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/bync-nd/4.0/). F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 with the following robust loss functions: ParamSmoothL1, Huber and Tukey. This can solve the problem of complexity in finding the best loss function parameter. •We propose two branches network (REX-INCEP) for face beauty estimation based on ResneXt-50 and Inception-v3 architectures. The main advantage of our REX-INCEP architecture is its ability to learn FBP features at a high level by using ResneXt and Inception blocks simultaneously, which proved its efficiency compared to seven CNN architectures. Moreover, our REX-INCEP architecture provides the right tradeoff between the performance and the number of parameters for facial beauty prediction. •We propose an ensemble regression for facial beauty estimation by merging the predicted scores of networks with one branch networks (ResneXt-50 and Inception-v3) and two branches network (REX-INCEP) trained with four loss functions (MSE, dynamic ParamSmoothL1, dynamic Huber, and dynamic Tukey). Although the individual regression models are trained with the same fixed hyperparameters, the resulting ensemble regression provides the most accurate estimates compared to the individual models and to state-of-the-art solutions. We have made our codes and preprocessed faces publicly available using our face alignment scheme at https://github.com/faresbougourzi/CNN-ER_for_ FBP. (Last accessed on November, 29th 2021) This paper is organized as follows: Section 2presents some related work on facial beauty prediction. In Section 3, we illustrate the backbone CNN architectures used, the proposed approach, and the proposed dynamic robust losses. Section 4includes: the description of the database and evaluation metrics used, the experimental settings, and the 60%–40% split and five fold crossvalidation experiments. Section 4is concluded by the comparison with state-of-the-art methods. Finally, Section 5concludes the paper. 2. Related work Image-based estimation of the beauty of faces is a new problem in computer vision. The first database that treats FBP as a regression task is from 2015 [10]. The methods used in the literature to predict and estimate facial beauty are either handcrafted methods [11–18] or deep learning methods [12,19–21]. The hand-crafted methods can be classified as geometry-based or appearance-based methods [16]. In [15], P. Aarabi et al. developed an automatic facial beauty rating system based on the relationships between facial features (face, eyes, eyebrows and mouth) with the K-nearest neighbour algorithm to learn a beauty mapping. D. Zhang et al. used tens of thousands of female and male faces and assigned them to a human face shape subspace. They then used a quantitative method to analyse the effect of geometric facial features on human facial beauty using a similarity transformation invariant shape distance measure. In [16], H. Yan proposed a new CSOR (Cost-Sensitive Ordinal Regression) to measure the importance of samples in different classes. They applied their CSOR to four types of features, namely intensity, LBP [22], SIFT [23], and LE [24]. L. Liang et al. [12] used geometric features (extracted 18-dimensional ratio features from faces) and appearance features (40 Gabor feature maps) with shallow predictors, which are linear regression (LR), Gaussian regression (GR), and support vector regression (SVR). These methods were tested using the SCUT-FBP5500 database. In recent years, deep learning architectures have been widely used to evaluate the beauty of faces. In [12], L. Liang et al. presented their face beauty database (SCUT-FBP5500) with two evaluation protocols (60%–40% split and 5-fold cross validation). They tested three CNN architectures (Alexnet [25], Resnet-18 [26] and ResneXt-50 [27]). Their results show that the ResneXt-50 architecture outperformed the other two deep architectures (Alexnet and Resnet-18). Moreover, the deep architectures performed better than the hand-crafted features they used with different shallow regressors. K. Cao et al. used a residual-in-residual (RIR) block to build a deeper network with multi-level skip connections to produce better gradient transmission flow. In addition, they used both channel-wise and space-wise attention mechanisms to find the inherent correlation between feature maps. Their approach was tested on the SCUT -FBP5500 [12] database and showed good performance. In [21], L. Lin et al. propose an R3CNN architecture consisting of two main components. The first component is a regression component that contains two identical regression subnets that consistently map each face image to a beauty value. The second component is a ranking component that uses the Siamese network to learn a pairwise ranking to guide the regression of the beauty prediction. Their architecture showed promising results on SCUT-FBP [10] and the SCUT-FBP5500 [12] databases. In addition to supervised learning, semi-supervised learning shows promising results for face beauty estimation [28, 29]. In [29], F. Dornaika et al. presented a multi-layered local discriminant embedding algorithm that integrates feature selection as the main step. Feature selection captures the most relevant and discriminative features of an input face image or face descriptor. 3. Methodology In this section, we will present the used CNN architectures and our proposed approach and the proposed dynamic robust losses. 3.1. Backbone CNN architectures Since deep learning architecture ‘‘Alexnet’’ [25] won the ImageNet challenge in 2012, numerous CNN architectures have been proposed. In our work, we will use two popular architectures (ResneXt-50 [27] and Inception-v3 [30]) as building blocks for our solution. It is worth noting that other backbone architectures can also be used. In our proposal, we use the above pretrained models trained on the ImageNet challenge database [9]. To keep the paper self-contained, this section briefly introduces the CNNs ResneXt-50 and Inception-v3, which were used as backbone architectures in our proposed solution. ResneXt-50 architecture. The architecture of ResneXt-50 is presented in [27], which is based on the ResneXt module (Fig. 1). The ResneXt module performs a series of transformations, each based on a low-dimensional embedding and sharing the same topology. The results of all transformations are combined by summation. Inception-v3 architecture. The Inception-v3 architecture is presented in [30], which is based on the Inception module presented in [31]. The main idea of the inception architecture is to combine different convolutional layers with different kernel sizes and pooling layers in one inception module, as shown in Fig. 2. 3.2. Our approach Our global approach is shown in Fig. 3. The output score is the average of multiple scores, which means that we use an ensemble of multiple regression models. In our implementation, we use six models. There are two main contributions in this ensemble: (i) the deep network with two branches (REX-INCEP) (see Section 3.4) and (ii) the dynamic robust loss functions (see Section 3.5). The first two scores are predicted by the trained deep networks ResneXt-50 and Inception-v3 using the MSE loss function 2 F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 Fig. 1. A ResneXt Module with cardinality = 32, with roughly the same complexity. A layer is shown as (# in channels, filter size, # out channels) [27]. Fig. 2. The Third Inception Module used in Inception-v3 architecture [30]. and the dynamic Huber loss function, respectively. The remaining four scores are estimated after training the proposed two-branch deep network (REX-INCEP) using four loss functions: MSE, dynamic ParamSmoothL1, dynamic Huber, and dynamic Tukey. The two-branch deep network consists of ResneXt-50 and inceptionv3, which are merged into a single architecture. As will be seen in the experimental section, using the two contributions without the ensemble results in performance that is better than that of the state-of-the-art methods. Using the ensemble shown in Fig. 3 will further improve the results. 3.3. Face preprocessing In the face preprocessing phase, we used the scheme proposed in [32,33]. The process is shown in Fig. 4. There are three steps to crop the face region from the raw face image. First, we use the provided face features to align the eyes by performing a 2D rotation of the face based on the eye coordinates. After this 2D rotation of the image and the detected points, the three farthest points in the left, right and bottom directions are selected as the three boundaries of the face. We denote the distance from the lower boundary to the position of the eyes as d1. The upper boundary of the face is set with a distance d2from the eyes, which is set as d2= 0.6 d1. Finally, obtain the face ROI, by cropping the face using the four boundaries and resizing the obtained box image to a fixed size that depends on the input size of the corresponding network. 3.4. Two branches architecture In recent years, many successful deep architectures have been proposed for many computer vision tasks. To train two architectures simultaneously, we propose two branches architecture to exploit the different capabilities of the networks. Since FBP data is limited, we propose to exploit the low-level and high-level feature extraction capability of two powerful architectures simultaneously. Fig. 5 summarizes our proposed architecture with two branches. The first and second branches are the ResneXt-50 and Inception-v3 architectures, respectively, with the decision levels removed. In our proposed architecture with two branches, we added the layer FC1 that maps the output of the ResneXt-50 branch (vector of dimension 2048) to 1024 neurons. Similarly, we added layer FC2, which maps the output of the Inception-v3 branch (vector of dimension 2048) to 1024 neurons. FC1 and FC2 were concatenated into a single vector FC, which is followed by the FC2 layer that performs the regression. Note that the weights of both branches are the weights of the pre-trained ResneXt-50 and Inception-v3 models (trained on the ImageNet challenge database [9].), while the FC1, FC2 and FC3 layers are randomly initialized. Our proposed network with two branches is called REX-INCEP architecture. In the training phase, we will fine-tune this architecture for FBP. 3.5. Loss functions: the use of dynamic robust losses During convolutional network training, the loss function measures the error (the loss) between the ground truth and the estimated values. The CNNs aim to minimize the loss based on the gradients of the loss function used to update the weights of the network. In this section, we will describe the loss functions used in our experiments. We emphasize that three of them are robust loss functions. We will also introduce a dynamic law that adjusts the parameters of the robust losses during training. The losses are computed for the batch size N,yidenotes the ground truth score of the ith image, and ˆ yidenotes the estimated value corresponding to the ith image. 3.5.1. L1Loss function L1is one of the most commonly used loss functions. The most important property of the L1loss function is its robustness to outliers. For Nbatch size, L1loss function is defined by: LL1=1 N N ∑ i=1 |yi−ˆ yi|(1) 3.5.2. Mean Squared Error (MSE) loss function MSE is also known as L2loss function, it is more sensitive to outliers compared to L1. The MSE loss function should be used when the target data are normally distributed around a mean and when it is important to penalize outliers particularly heavily. For Npredictions, the MSE loss function is defined by: LMSE =1 N N ∑ i=1 (yi−ˆ yi)2(2) 3.5.3. Dynamic parameterized SmoothL1 (ParamSmoothL1) loss function The loss function SmoothL1 creates a criterion that uses a quadratic term if the absolute element-wise error falls below 1, and an L1term otherwise. It is less sensitive to outliers than the MSE loss function, and in some cases prevents exploding gradients [34]. The SmoothL1 loss function of Nimages is defined by: LSmoothL1=1 N N ∑ i=1 zi(3) 3 F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 Fig. 3. General structure of the proposed approach (CNN-ER). Fig. 4. Face Region of Interest. The left image is an original image from the database SCUT-FBP5500 [12]. The second image is the rotated face with its 86 detected landmarks used to estimate the three face boundaries (right, left and bottom). These boundaries correspond to the three points ∗marked in blue. The third image shows how the upper boundary of the face is determined. It is located at a distance d2=0.6d1from the vertical position of the eyes. The fourth image shows the cropped and rescaled face image with 224 pixels. Note that the distances D1and D2are constant for all cropped faces. Fig. 5. Our proposed two branches network REX-INCEP. where Nis the batch size and ziis given by: zi={0.5 (yi−ˆ yi)2,if |yi−ˆ yi|<1 |yi−ˆ yi| − 0.5,otherwise (4) Since the threshold may vary from one task to another, we proposed a Parameterized SmoothL1 loss function defined as follows: LPara_SmoothL1=1 N N ∑ i=1 zi(5) where Nis the batch size and ziis given by: zi={0.5 (yi−ˆ yi)2,if |yi−ˆ yi| ≤ α |yi−ˆ yi| + 0.5α2−α, otherwise (6) where αis tunable parameter. Our proposed dynamic robust loss functions are based on the following observation. During the training of ConvNets, the robust loss functions can be adjusted as the training progresses. Namely, during training, the model evolves and the outlier examples may vary. In the early stages of training, the model is usually neither very stable nor accurate enough to handle the outlier examples. Therefore, it is recommended to use the quadratic function of loss. At the end of the training, the model may be more or less accurate to deal with the outliers. Therefore, it is recommended to use the robust loss function where the range of non-outlier errors is relatively small. This means that the parameter of the robust loss function starts with a maximum value and decreases monotonically as the training progresses. From a practical point of view, it is extremely difficult to know the best value for αin advance. However, the variation interval [αmin, αmax]can be known in advance. Therefore, to make the robust loss function more adaptive to the training progress, we propose a dynamic parameter α. This parameter follows a cosine law as a function of the epoch number. The current value of αis given by: αcur =αmin +1 2(αmax −αmin)(1+cos (ecur ne π))(7) where αcur is the value of αat the current epoch (ecur ). The latter varies between 1 and the total number of epochs (ne). αmax and αmin are the maximum and minimum of the αvalue. In this paper, we denote the proposed dynamic Parameterized SmoothL1 by dynamic ParamSmoothL1. Fig. 6 shows the values of αusing the proposed law (Eq. (7)) as a function of epoch number. Here αmax and αmin are fixed at 0.7 and 0.3, respectively. Our dynamic law was inspired by the dynamic law used to control the learning rate in stochastic gradient descent methods [35]. 3.5.4. Dynamic Huber loss function Similar to ParamSmoothL1, Huber is another loss function that is less sensitive to outliers in the data than the L2loss function L2. For Ntraining images, the Huber loss function is defined by [36]: LHuber =1 N N ∑ i=1 zi(8) where Nis the batch size and ziis defined by: zi={0.5 (yi−ˆ yi)2,if |yi−ˆ yi| ≤ β β|yi−ˆ yi| − 0.5β2,otherwise (9) 4 F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 Fig. 6. Dynamic ParamSmoothL1 with αthat decreases from 0.7 to 0.3. where βis a controlled parameter. Fig. 7, shows a visualization of the Huber loss function with four βvalues (0.7, 0.5, 0.3 and 0.1) and L2loss function. Similar to the ParamSmoothL1 loss, we propose to use the dynamic βgiven by the following the equation: βcur =βmin +1 2(βmax −βmin)(1+cos (ecur ne π))(10) where βcur is the value of βin the current epoch (ecur ), where ecur increases from 1 to the total number of epochs (ne). βmax and βmin are the maximum and minimum of the βvalue. 3.5.5. Dynamic Tukey loss function The Tukey loss function [37] has the property of suppressing the influence of outliers during backpropagation by reducing the magnitude of their gradient towards zero. Another interesting property of this loss function is the soft constraints it imposes between inliers and outliers [38]. The Tukey loss function is defined by: LTukey =1 N N ∑ i=1 zi(11) where Nis the batch size and ziis given by: zi={c2 6[1−(1 −(|yi−ˆ yi| c)2)3],if |yi−ˆ yi| ≤ c c2 6,otherwise (12) where cis an adjustable parameter. Similar to ParamSmoothL1 and Huber losses, we propose to use dynamic cduring training through the equation: ccur =cmin +1 2(cmax −cmin)(1+cos (ecur ne π))(13) where ccur is the value of cat the current epoch (ecur ), where ecur increases from 1 to the total number of epochs (ne). cmax and cmin are the maximum and minimum of the cvalue. 4. Performance evaluation 4.1. Database and evaluation protocols To evaluate the performance of our approach, we used the database SCUT-FBP5500 [12]. It consists of 5500 frontal faces of subjects with different attributes: Age (from 15 to 60), gender (male/female), and ethnicity (Asian/Caucasian). Each face image was given a beauty score in the range [1–5] by 60 volunteers. In addition, each face image contains 86 facial features. Figs. 8(a), 8(b),8(d) and 8(c) show some face samples with their corresponding beauty ratings. The creators of the SCUT-FBP5500 database provided two evaluation scenarios [12]. In the first scenario, the data were split into a training split and a test split (60%–40%). In the second scenario, the data was split into 5 folds to perform a five-fold cross-validation. In our analyses, we will use both scenarios. 4.2. Evaluation metrics To evaluate the performance of each model, four evaluation metrics are used, namely: mean absolute error (MAE), root mean square error (RMSE), Pearson correlation coefficient (PC) and the ϵerror. Consider Y=(y1,y2,...,yn) the ground-truth scores of the tested nimages and ˆ Y=(ˆ y1,ˆ y2,..., ˆ yn) as their corresponding estimated scores. Where nrepresents the number of face images tested. The evaluation metrics are defined as follows: Mean Absolute Error (MAE): MAE is defined by: MAE =1 n n ∑ i=1 |yi−ˆ yi|(14) MAE is a scale-dependent accuracy measurement, i.e. MAE uses the same scale as the data being measured. Root Mean Square Error (RMSE): RMSE is defined by: RMSE =1 n n ∑ i=1 (yi−ˆ yi)2(15) The RMSE is another scale-dependent accuracy measure. Unlike MAE, the effect of any error on the RMSE is proportional to the squared error; therefore, larger errors have a disproportionately large effect on the final RMSE. Consequently, the RMSE is sensitive to outliers. Pearson Correlation coefficient (PC): PC was developed by Karl Pearson [39] and it is defined by: PC =∑n i=1(yi−yi) ( ˆ yi−ˆ yi) √∑n i=1(yi−yi)2√∑n i=1(ˆ yi−ˆ yi)2 (16) where yiand ˆ yiare the means of the ground-truth scores and the estimated scores, respectively. PC has a value between +1 and -1, it is a statistic that measures the linear correlation between two variables Yand ˆ Y. A value of +1 means a completely positive linear correlation, 0 means no linear correlation, and −1 means a completely negative linear correlation. ϵ-error: ϵ-error is defined by: ϵ-error =1 n n ∑ i=1(1−exp ((yi−ˆ yi)2 2σ2 i)) (17) where σiis the standard deviation of the scores of all raters of image i. The value of the ϵerror is the accumulation of the errors of the individual images ibased on the term ϵ-errori= 1−exp ((yi−ˆ yi)2 2σ2 i ). When the absolute error of image iapproaches zero (i.e., yi=ˆ yi), ϵ-erroriis zero. On the other hand, when the absolute error is large, ϵ-error takes into account the uncertainty of the rate represented by σ2 i. More precisely, the division by the term σ2 icontributes less to the value of the ϵerror when the uncertainty of the rate is large and vice versa. 5 F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 Fig. 7. Visualization of two loss functions: L2and Huber with four βvalues (0.7, 0.5, 0.3 and 0.1). Fig. 8. Facial beauty samples from the SCUT-FBP5500 database, (a) Female Asian samples and the corresponding scores are (from left to right): 1.88, 3.00, 3.93, and 4.28. (b) Male Asian samples and the corresponding scores are (from left to right): 1.73, 2.48, 3.53, and 4.43. (c) Female Caucasian samples and the corresponding scores are (from left to right): 1.93, 2.87, 3.63, and 4.7. (d) Male Caucasian samples and the corresponding scores are (from left to right): 1.88, 2.67, 3.27, and 4.43. 4.3. Experimental setup All experiments were performed on Pytorch [40] with an NVIDIA Geforce GTX 1060 6 GB GPU. All networks are trained for 40 epochs using Adam optimizer [41] and batch size of 15. The initial learning rate is 1e−4 for 20 epochs, then the learning rate decreases to 1e−5 for the next 10 epochs, and for the last 10 epochs the learning rate decreases to 1e−6. Active data augmentation is performed by rotating the input face by an angle between [−5, 5]. For all experiments, the reported results correspond to the best PC of the test data during the training/testing of the 40 epochs. 4.4. Experimental results on the 60%–40% split scenario In this section, we limit the study to the provided 60%–40% split. 4.4.1. Raw input vs the proposed face preprocessing To investigate the effectiveness of our proposed face preprocessing method, we used ResneXt-50 and Inception-v3 with loss function MSE to train FBP on two input image scenarios (the raw image and the cropped face with our preprocessing method). The obtained results are summarized in Table 1. From these results, it can be seen that our preprocessing scheme improves the results for both ResneXt-50 and Inception-v3 architectures. In other words, our proposed face alignment scheme can support the training of CNN architectures by discarding the background features and prioritizing the face features. 4.4.2. FBP using CNN architectures In this section, we compare the performance of seven CNN architectures (VGG-16 [42], Resnet-50 [44], Resnet-101 [44], Resnet-152 [44], Wide-Resnet [43], Inception-v3 [30], ResneXt50 [27]) using the standard MSE loss function. The results are summarized in Table 2. Based on these results, we can conclude that Inception-v3 and ResneXt-50 perform the best compared to the other CNN architectures. Moreover, these two CNN architectures have a smaller number of parameters than VGG16, Resnet-101, Resnet-152, Wide-Resnet and similar to Resnet50, which proves the ability of Inception-v3 and ResneXt-50 to learn high-level features for FBP with an intermediate number of parameters. Given the observed efficiency of the Inception-v3 and ResneXt50 architectures in both performance and number of trainable parameters (Table 2), we propose to combine these two CNN architectures (REX-INCEP). From Table 2, we conclude that our proposed REX-INCEP achieves better performance than all CNN architectures. Moreover, the number of trainable parameters of our proposed REX-INCEP is similar to Resnet-101, Resnet-152 and Wide-Resnet and less than VGG-16. These advantages prove the 6 F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 Table 1 Face beauty prediction using ResneXt-50 and Inception-v3 networks with MSE loss function and two input image scenarios (The raw image and the detected face with our preprocessing scheme). CNN architecture Pre-processing PC ↑MAE ↓RMSE ↓ϵ-error ↓ ResneXt-50 Raw image 0.9119 0.2126 0.2845 0.0853 ResneXt-50 Face detection 0.9146 0.2092 0.2763 0.0802 Inception-v3 Raw image 0.9108 0.2150 0.2831 0.0873 Inception-v3 Face detection 0.9112 0.2147 0.2814 0.0833 Table 2 Comparison between seven CNN architectures (VGG-16, Resnet-50, Resnet-101, Resnet-152, WideResnet, Inception-v3 and ResneXt-50) and our proposed REX-INCEP approach for Facial Beauty Prediction with MSE loss function. CNN architecture PC ↑MAE ↓RMSE ↓ϵ-error ↓N of params VGG-16 [42] 0.9025 0.2229 0.2932 0.0891 134 M Wide-Resnet [43] 0.9066 0.2176 0.2889 0.0851 66 M Resnet-50 [44] 0.9087 0.2155 0.2850 0.0849 23 M Resnet-152 [44] 0.9069 0.2182 0.2880 0.0860 58 M Resnet-101 [44] 0.9095 0.2157 0.2852 0.0848 44 M Inception-v3 [30] 0.9139 0.2125 0.2779 0.0819 25 M ResneXt-50 [27] 0.9146 0.2092 0.2763 0.0802 22 M REX-INCEP (Our architecture) 0.9159 0.2071 0.2739 0.0790 52 M efficiency of our proposed REX-INCEP for FBP compared to CNN architectures. As shown in Fig. 5, our proposed REX-INCEP has two branches. In the first branch, our proposed REX-INCEP architecture is able to learn high-level features for FBP by using a combination of splitting, transformation and aggregation mechanisms through the ResneXt block. In the second branch, our proposed REX-INCEP architecture is able to learn high-level features for FBP by combining different convolutional layers with different kernel sizes and pooling layers through the Inception blocks. The main advantage of our REX-INCEP architecture is its ability to learn high-level FBP features using ResneXt and Inception blocks simultaneously, which proved its efficiency compared to seven CNN architectures. From the results in Table 2, we conclude that our REX-INCEP architecture provides the right tradeoff between the performance and the number of parameters for facial beauty prediction. 4.4.3. Dynamic vs fixed loss parameter In this section, we compare the performance of Face Beauty Prediction with dynamic and fixed loss functions. In this set of experiments, we choose a CNN architecture and a parametric robust loss function. We then compare the performance of two variants of this parametric robust loss function: (i) a loss function that assumes a fixed parameter, and (ii) a loss function that assumes a dynamic parameter using the cosine law. Specifically, we use the ResneXt-50 network and the following parametric loss functions: ParamSmoothL1, Huber and Tukey loss functions. To provide a fair comparison, the interval of parameter variation associated with the dynamic scheme is also used by the fixed parameter loss function. This is achieved by repeating the training and testing with several fixed values from the same interval. We compare the performance obtained with the dynamic scheme with the average performance associated with the spanned fixed values within this interval. Table 3 summarizes the results obtained. For the loss of ParamSmoothL1, the interval of αis fixed to [0.7–0.3]. The fixed parameter scheme spans the following values {0.7,0.6,0.5,0.4,0.3}. Based on the results of the loss function ParamSmoothL1, we can see that the performance of the dynamic scheme is better than the average performance obtained with the fixed loss. Similar to ParamSmoothL1, the dynamic Huber loss function with a βparameter in the interval [0.7–0.3] achieved better performance than the mean performance obtained by the fixed βvalues {0.7,0.6,0.5,0.4,0.3}. For Tukey loss, the interval of parameter cwas set to [2-1.5] and the fixed cvalues are {2,1.7,1.5}. Similar to ParamSmoothL1 and the Huber loss function, the dynamic Tukey loss provided better performance than the Tukey loss with a fixed cvalue. Moreover, the dynamic Tukey loss function adopting the [2-1] achieved better performance than the dynamic Tukey loss function adopting the interval [2-1.5]. In our approach, the intervals for the α,βand c parameters of the three loss functions are [0.7–0.3] [0.7–0.3] and [2-1], respectively. 4.4.4. Two branches vs one branch using five loss functions In this section, we compare the performance of single-branch networks (ResneXt-50 and Inception-v3) and that of the proposed two branches network (REX-INCEP). Table 4 shows the performances obtained with the ResneXt-50 network when five loss functions were used. From the results, it can be seen that the loss function MSE gives the best performance. Table 5 contains the results for the Inception-v3 network when five loss functions were used. From these results, we can conclude that the dynamic Huber loss function gives the best performance. Table 6 shows the results of our proposed two branches network (REX-INCEP) when four loss functions (MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey losses) are used. Among the results obtained, dynamic ParamSmoothL1 achieved the best performance. Moreover, for a given loss function, the performance of the two-branch solution was better than that of the one-branch solution. The exception is the dynamic Huber loss. However, this difference in performance is very small. Comparing the results of one-branch networks and two-branch networks, we find that the two-branch network achieves high performance for all loss functions, with ParamSmoothL1 being the best. In contrast, the one-branch networks achieved competitive performance only for the MSE loss with the ResneXt-50 network and the dynamic Huber loss with the Inception-v3 network. This proves the effectiveness of the proposed REX-INCEP network, which effectively fuses and transforms the features generated by each architecture. 4.4.5. CNN ensemble To increase the performance of FBP, we will use an ensemble of trained CNN architectures and use different loss functions. In this scenario, the final score is set to the average of the facial beauty scores provided by different models. In this group of experiments, six models are adopted: the two trained networks with one branch and the best loss functions (ResneXt50 with MSE and Inception-v3 with dynamic Huber) and four 7 F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 Table 3 Comparison between dynamic and fixed parameters of loss functions ParamSmoothL1, Huber and Tukey using ResneXt-50 network. Loss function Parameter PC ↑MAE ↓RMSE ↓ϵ-error ↓ α= 0.7 0.9122 0.2127 0.2799 0.0814 α= 0.6 0.9141 0.2098 0.2772 0.0803 ParamSmoothL1 α= 0.5 0.9132 0.2110 0.2780 0.0815 α= 0.4 0.9101 0.2146 0.2825 0.0833 α= 0.3 0.9116 0.2150 0.2810 0.0831 Mean 0.9123 0.2126 0.2797 0.0819 dynamic α(0.7–0.3) 0.9140 0.2104 0.2777 0.0805 β= 0.7 0.9126 0.2114 0.2796 0.0812 β= 0.6 0.9130 0.2107 0.2780 0.0804 Huber β= 0.5 0.9144 0.2111 0.2770 0.0808 β= 0.4 0.9124 0.2122 0.2783 0.0839 β= 0.3 0.9110 0.2155 0.2811 0.0845 Mean 0.9127 0.2122 0.2788 0.0822 dynamic β(0.7–0.3) 0.9141 0.2116 0.2777 0.0814 c= 2.0 0.9128 0.2155 0.2810 0.0837 c= 1.7 0.9116 0.2133 0.2805 0.0821 Tukey c= 1.5 0.9126 0.2129 0.2808 0.0824 Mean 0.9123 0.2139 0.2808 0.0827 dynamic c(2–1.5) 0.9127 0.2120 0.2801 0.0819 dynamic c(2–1) 0.9133 0.2100 0.2780 0.0802 Table 4 Facial Beauty Prediction using ResneXt-50 Network with five loss functions (L1, MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey losses). Loss function PC ↑MAE ↓RMSE ↓ϵ-error ↓ L10.9126 0.2113 0.2783 0.0810 MSE 0.9146 0.2092 0.2763 0.0802 Dyn. ParamSmoothL1 (0.7–0.3) 0.9140 0.2104 0.2777 0.0805 Dyn. Huber (0.7–0.3) 0.9141 0.2116 0.2777 0.0814 Dyn. Tukey (2–1) 0.9133 0.2100 0.2780 0.0802 Table 5 Facial Beauty Prediction using Inception-v3 Network with five loss functions (L1, MSE, dynamic SmoothL1, dynamic Huber and dynamic Tukey). Loss function PC ↑L1↓RMSE ↓ϵ-error ↓ L10.9103 0.2152 0.2832 0.0848 MSE 0.9112 0.2147 0.2814 0.0833 Dyn. ParamSmoothL1 (0.7–0.3) 0.9118 0.2129 0.2805 0.0836 Dyn. Huber (0.7–0.3) 0.9139 0.2125 0.2779 0.0819 Dyn. Tukey (2–1) 0.9124 0.2138 0.2794 0.0802 Table 6 Facial Beauty Prediction using the proposed two branches Network (REX-INCEP) with four loss functions (MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey losses). Loss function PC ↑MAE ↓RMSE ↓ϵ-error ↓ MSE 0.9159 0.2071 0.2739 0.0790 Dyn. ParamSmoothL1 (0.7–0.3) 0.9165 0.2065 0.2736 0.0789 Dyn. Huber (0.7–0.3) 0.9138 0.2113 0.2785 0.0818 Dyn. Tukey (2–1) 0.9149 0.2105 0.2766 0.0806 trained two branches networks with four loss functions (REXINCEP with MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey). The solution based on the six models is called CNN Ensemble Regression (CNN-ER) and the fusion scheme refers to the averaging of the predicted scores from more than one model. For each model, we selected the final model after training it with 40 epochs. The results are summarized in Table 7. This table presents three different ensemble solutions. According to the results depicted in this table, we find the mean scores of the models with two branches perform better than the mean scores of the models with one branch. Moreover, the mean scores of the models with one and two branches (all six models) outperformed the mean scores of the models with one and two branches. We also note that the mean scores of the one branch networks perform better than one and two branches networks (Tables 4, 5, and 6). This proves the effectiveness of our proposed CNN ensemble method. It is worth noting that the individual models were trained only on the training set with a fixed number of epochs (40 epochs). 4.5. Experimental results using the five fold cross-validation scenario In this section, we will use the provided five folds to perform the cross-validation experiments. Table 8 contains the results obtained with each fold, as well as the average over the five folds using the networks with one branch (ResneXt-50 with MSE loss function and Inception-v3 with dynamic Huber loss function) and four networks with two branches (REX-INCEP with MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey loss function). It is worth noting that the presented result for each fold corresponds to the best result obtained by PC over the test 8 F. Bougourzi, F. Dornaika and A. Taleb-Ahmed Knowledge-Based Systems 242 (2022) 108246 Table 7 Facial Beauty Prediction using the proposed CNN ensemble of different trained models on 60%–40% data split. Fusion scheme PC ↑MAE ↓RMSE ↓ϵ-error ↓ One branch (Mixture 2 models) 0.9190 0.2054 0.2705 0.0777 Two branches (Mixture 4 models) 0.9194 0.2043 0.2699 0.0771 CNN-ER (Mixture 6 models) 0.9207 0.2032 0.2683 0.0764 Table 8 Five-fold cross-validation of facial beauty prediction using networks with one branch (ResneXt-50 with MSE loss and Inception-v3 with dynamic Huber loss) and two branches (REX-INCEP with MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey losses). Architecture Fold PC ↑MAE ↓RMSE ↓ϵ-error ↓ Fold 1 0.9149 0.2142 0.2779 0.0799 Inception-v3 Fold 2 0.9138 0.2074 0.2791 0.0798 with dynamic Fold 3 0.9203 0.2122 0.2801 0.0800 Huber loss Fold 4 0.9226 0.2072 0.2684 0.0783 function Fold 5 0.9202 0.2056 0.2706 0.0777 Mean 0.9184 0.2093 0.2752 0.0791 Fold 1 0.9176 0.2072 0.2744 0.0765 ResneXt-50 Fold 2 0.9114 0.2163 0.2865 0.0845 with MSE loss Fold 3 0.9204 0.2106 0.2759 0.0783 function Fold 4 0.9228 0.2066 0.2686 0.0767 Fold 5 0.9204 0.2053 0.2702 0.0760 Mean 0.9185 0.2092 0.2751 0.0784 Fold 1 0.9190 0.2081 0.2722 0.0772 REX-INCEP Fold 2 0.9172 0.2068 0.2755 0.0783 with MSE loss Fold 3 0.9212 0.2085 0.2748 0.0783 function Fold 4 0.9252 0.2045 0.2654 0.0767 Fold 5 0.9213 0.2049 0.2691 0.0756 Mean 0.9208 0.2066 0.2714 0.0772 Fold 1 0.9202 0.2070 0.2718 0.0763 REX-INCEP Fold 2 0.9167 0.2056 0.2766 0.0774 with dynamic Fold 3 0.9206 0.2095 0.2763 0.0780 ParamSmoothL1 Fold 4 0.9253 0.2053 0.2634 0.0759 loss function Fold 5 0.9238 0.2011 0.2639 0.0742 Mean 0.9213 0.2057 0.2704 0.0764 Fold 1 0.9196 0.2078 0.2722 0.0767 REX-INCEP Fold 2 0.9167 0.2033 0.2742 0.0771 with dynamic Fold 3 0.9238 0.2084 0.2717 0.0775 Huber loss Fold 4 0.9282 0.1992 0.2605 0.0720 function Fold 5 0.9227 0.2041 0.2666 0.0755 Mean 0.9222 0.2046 0.2690 0.0758 Fold 1 0.9178 0.2079 0.2738 0.0766 REX-INCEP Fold 2 0.9166 0.2082 0.2782 0.0790 with dynamic Fold 3 0.9222 0.2089 0.2722 0.0773 Tukey loss Fold 4 0.9242 0.2036 0.2648 0.0755 function Fold 5 0.9205 0.2076 0.2702 0.0770 Mean 0.9203 0.2072 0.2718 0.0771 data during the training of 40 epochs. The cross-validation results can provide a better comparison between the networks and the loss functions. Comparing the networks, we can find that the trained networks with two branches outperform the networks with one branch. This is consistent with the conclusion found in the 60%–40% split scenario. Moreover, we can observe that ResneXt-50 with the MSE loss function and Inception-v3 with the dynamic Huber loss function achieve similar results. Based on the results obtained with the two branch solutions, we can conclude that the dynamic Huber loss function achieves the best performance. On the other hand, the dynamic loss function ParamSmoothL1 outperforms the dynamic loss functions Tukey and MSE, which obtained similar results. Table 9 shows the performances achieved by CNN ensembles. Similar to the ensemble experiments in the previous scenario, we consider an ensemble of one branch networks, an ensemble of two branch networks, and the mixture ensemble of all networks. Comparing the results of the 8and 9, we can make the following observations: Table 9 Five folds cross-validation of Facial Beauty Prediction using the proposed CNN ensemble of different trained models. Fusion scheme Fold PC ↑MAE ↓RMSE ↓ϵ-error ↓ Fold 1 0.9205 0.2065 0.2703 0.0753 Fold 2 0.9170 0.2060 0.2763 0.0788 One branch Fold 3 0.9245 0.2029 0.2691 0.0745 (2 models) Fold 4 0.9263 0.2015 0.2630 0.0747 Fold 5 0.9234 0.2010 0.2652 0.0736 Mean 0.9223 0.2036 0.2688 0.0754 Fold 1 0.9223 0.2035 0.2683 0.0743 Fold 2 0.9202 0.2018 0.2713 0.0757 Two branches Fold 3 0.9252 0.2053 0.2698 0.0751 (4 models) Fold 4 0.9289 0.1995 0.2586 0.0730 Fold 5 0.9253 0.1994 0.2622 0.0725 Mean 0.9244 0.2019 0.2660 0.0741 Fold 1 0.9232 0.2026 0.2667 0.0735 Fold 2 0.9204 0.2016 0.2710 0.0756 CNN-ER Fold 3 0.9264 0.2029 0.2675 0.0738 (Mixture Fold 4 0.9292 0.1990 0.2583 0.0727 6 models) Fold 5 0.9257 0.1984 0.2615 0.0720 Mean 0.9250 0.2009 0.2650 0.0735 Table 10 Comparison with the state-of-the-arts methods using the 60%–40% split. Method PC ↑MAE ↓RMSE ↓ LR [12] 0.5948 0.4289 0.5531 GR [12] 0.6738 0.3914 0.5085 SVR [12] 0.6668 0.3898 0.5132 Alexnet [12] 0.8298 0.2938 0.3819 Resnet-18 [12] 0.8513 0.2818 0.3703 ResneXt-50 [12] 0.8777 0.2518 0.3325 CNN with SCA [19] 0.8780 0.2517 0.3320 Dynamic ParamSmoothL1* (Ours) 0.9165 0.2065 0.2736 CNN-ER (Ours) 0.9207 0.2032 0.2683 Dynamic ParamSmoothL* is our REX-INCEP network that was trained using the dynamic ParamSmoothL1 loss function. •From the results of the one-branch ensemble, it can be seen that the fusion scheme outperforms the individual onebranch networks (ResneXt-50 with MSE loss function and Inception-v3 with dynamic Huber loss function). •For the two-branch ensemble results, the fusion scheme outperforms all the results obtained by the single twobranch networks (REX-INCEP with MSE, dynamic ParamSmoothL1, dynamic Huber and dynamic Tukey loss functions). •From the results of the mixture ensemble (all six models), it is clear that the fusion scheme outperforms not only the oneand two-branch networks, but also their fused models. The above observations prove the effectiveness of the proposed fusion scheme. This also shows the efficiency of using two branch networks with different loss functions. 4.6. Comparison with state-of-the-art methods In this section, we compare our proposed methods with the state-of-the-art methods in both scenarios: 60%–40% split and five-fold cross-validation. Table 10 shows a comparison between our method and the state-of-the-art methods using the 60%–40% split. The comparison shows that our approach (CNN-ER) outperforms the state9