A cost-sensitive constrained Lasso

Advances in Data Analysis and Classification https://doi.org/10.1007/s11634-020-00389-5 REGULAR ARTICLE A cost-sensitive constrained Lasso Rafael Blanquero1,2 ·Emilio Carrizosa1,2 ·Pepa Ramírez-Cobo2,3 · M. Remedios Sillero-Denamiel1,2 Received: 16 April 2019 / Revised: 26 February 2020 / Accepted: 2 March 2020 © Springer-Verlag GmbH Germany, part of Springer Nature 2020 Abstract The Lasso has become a benchmark data analysis procedure, and numerous variants have been proposed in the literature. Although the Lasso formulations are stated so that overall prediction error is optimized, no full control over the accuracy prediction on certain individuals of interest is allowed. In this work we propose a novel version of the Lasso in which quadratic performance constraints are added to Lasso-based objective functions, in such a way that threshold values are set to bound the prediction errors in the different groups of interest (not necessarily disjoint). As a result, a constrained sparse regression model is defined by a nonlinear optimization problem. This cost-sensitive constrained Lasso has a direct application in heterogeneous samples where data are collected from distinct sources, as it is standard in many biomedical contexts. Both theoretical properties and empirical studies concerning the new method are explored in this paper. In addition, two illustrations of the method on biomedical and sociological contexts are considered. Keywords Performance constraints ·Cost-sensitive learning ·Sparse solutions · Sample average approximation ·Heterogeneity ·Lasso Mathematics Subject Classification 62-07 Data analysis ·62H12 Estimation in multivariate analysis ·62J07 Ridge regression; shrinkage estimators (Lasso) ·62P99 Applications of statistics 1 Introduction Let (Y,X)be a random vector, where X=(X1,...,Xp)is a vector of ppredictors and Yidentifies the response variable. Given the observed response vector BM. Remedios Sillero-Denamiel [email protected] 1Departamento de Estadística e Investigación Operativa, Universidad de Sevilla, Seville, Spain 2Instituto de Matemáticas de la Universidad de Sevilla (IMUS), Seville, Spain 3Departamento de Estadística e Investigación Operativa, Universidad de Cádiz, Cádiz, Spain 123 R. Blanquero et al. y=(y1,...,yn),n>p, and the related observed predictors, xj=(x1j,...,xnj), j=1,...,p, the linear regression model predicts yby ˆ y=ˆ β0+ˆ β1x1+···+ ˆ βpxp. Consider the well-known prostate database (Stamey et al. 1989), which consists of the measurements of p=8 predictors and one response variable (clinical measures) on n=97 men who were about to receive a radical prostatectomy. Further, assume that the dataset is divided into two groups: Group 1, corresponding to young individuals (aged less than 65) and Group 2, related to the population older than 65. If the goal is to minimize the overall mean squared error (MSE), the parameter vector β= (β0,β 1,...,βp)can be estimated by a fitting procedure as ordinary least squares (OLS), yielding ˆ βols . The results obtained under the OLS are shown in the first two rows of Table 1, where 3/4 of the total set has been used to fit the model (training set) and the remaining samples for the assessment of the generalization error of the resulting model (test set). The overall MSE, the prediction errors over the two groups as well as the number of coefficients involved in the model are presented. Once the model is fitted, there are two fundamental criteria for evaluating its performance: the accuracy of prediction and the identification of significant predictors, which provides a good interpretation of the solution. It is well-known that ˆ βols is not sparse, as can be observed from Table 1where the eight predictor variables have been used by the model. This fact has entailed new penalization techniques as ridge regression (Hoerl and Kennard 1970), which is a continuous shrinkage method that achieves its better accuracy prediction through a bias-variance trade-off. Nevertheless, ridge regression is known not to be able to render a parsimonious solution. In contrast, best-subset selection (Garside 1965) achieves more sparse solutions, but it suffers from high variability and computational difficulties (Fan and Li 2001). To overcome those shortcomings, Tibshirani (1996) proposed the Lasso regularization technique, which achieves both estimation and selection of relevant predictors simultaneously by construction. Given X=[1|x1|...|xp]the predictor matrix; then, the Lasso solution can be defined as ˆ βLasso(λ) =arg min β 1 ny−Xβ2+λ(β1,...,βp)1(1) Table 1 Results obtained using prostate dataset Method Overall MSE Group 1 MSE Group 2 MSE Non-zero coefficients OLS Training set 0.344 0.355 0.333 8 Test set 0.373 0.380 0.367 Lasso Training set 0.365 0.397 0.335 5 Test set 0.408 0.414 0.403 CSCLasso Training set 0.355 0.357 0.352 6 Test set 0.393 0.399 0.388 123 A cost-sensitive constrained Lasso where λ≥0 is a tuning parameter and .1is the l1norm. To visualize the effect of the penalty term in the Lasso formulation, consider the third and fourth rows in Table 1, which provide the results obtained under the Lasso for the prostate dataset. In this case, in comparison with OLS results, a sparser, and therefore a more interpretable solution, has been obtained at the expense of slightly worsening MSE values. One of the advantages of the Lasso is that the entire path of solutions can be found thanks to the LARS algorithm (Efron et al. 2004). In addition, it is well-known that, under some conditions, the Lasso enjoys good theoretical and statistical properties (Donoho et al. 1995; Friedman et al. 2001; Bühlmann and Van-De Geer 2011). However, the Lasso presents some limitations; in particular, the literature related to the Lasso has not undertaken the problem of fully controlling the accuracy prediction on certain individuals of interest. In the previous prostate database, assume for instance that we are interested in fitting a sparse regression model to the dataset where, apart from obtaining a small overall mean squared error, also the prediction error for the young individuals should not exceed a given threshold. In this paper we propose a Lasso-based model that allows for such aim, namely the cost-sensitive constrained Lasso, denoted from now on as CSCLasso. The results obtained for the prostate database under the CSCLasso, whose definition and main properties shall be discussed in Sects. 2and 3, are shown in the last two rows of Table 1. A threshold for the mean squared error over Group 1 is set equal to 0.357, which represents an improvement of 10% over the prediction error of the Lasso (0.397). Note that in the training set the MSE satisfies the imposed constraint, as expected. Also note that the improvement in Group 1 is at the expense of slightly increasing the prediction error over Group 2. In terms of sparsity, the CSCLasso model has needed an additional predictor variable comparing to Lasso in order to comply with the constraint. As it will be seen in Sect. 2, the novel approach is set up by adding convex quadratic constraints to the Lasso formulation. Other approaches have considered constrained versions of the Lasso before, see for example (James et al. 2019; Gaines et al. 2018; Torres-Barrán et al. 2018;Huetal.2015 and references therein). In such works, equality or/and inequality linear constraints are considered for imposing prior knowledge and structure onto the coefficient estimates. In our approach instead, quadratic convex constraints are formulated and thus, our approach and results generalize those previously obtained in the literature. Not only constrained versions of the Lasso can be found in the literature. Indeed, many different variants have been proposed. For example, in Zou (2006) adaptive weights for penalizing different coefficients in the l1penalty are included as a way for fitting sparser models under more general conditions. Moreover, in the presence of highly correlated predictor variables (as is usual in microarray studies) or when predictors are structurally grouped (e.g. dummy variables), the Lasso sometimes does not perform well and, as a consequence, the elastic net (Zou and Hastie 2005) and the group lasso (Yuan and Lin 2006; Simon et al. 2011) were proposed. They combine l2and l1penalties to try to select (or remove) the correlated or structured predictor variables together. Other extension is to consider ˆ βLasso(λ) =arg min β 1 ny−Xβ2+λAβ1.(2) 123 R. Blanquero et al. instead of (1), where Ais a fixed matrix (see Tibshirani and Taylor 2011). If A= 0|Ip×p, then the Lasso objective function is obtained; however, other forms of A different from the identity can be found in the literature, see for example (Ollier and Viallon 2017). In fact, various choices of Ain (2) define problems that are already well-known in the literature as the fused lasso (Tibshirani et al. 2005). See Hastie et al. (2015) for an extensive review about Lasso problem and generalizations. The motivation of this paper (controlling the performance measure on certain groups of interest) is not novel in the Data Analysis literature, and indeed, it has been exploited in classification contexts with the term cost-sensitive learning (Prati et al. 2015;He and Ma 2013). Many realworld problems, such as those found in medical diagnosis or credit card fraud detection, have asymmetric misclassification costs associated, since the consequences of wrong predictions across the classes may be very different. Therefore, for these problems, it is more important to achieve better classification rates for the individuals of interest (ill people, defaulting customers). See Carrizosa et al. (2008), Sun et al. (2009), Datta and Das (2015), Lee et al. (2017), Bradford et al. (1998) and Freitas et al. (2007) for more details and applications. Those methods are based on adapting the classifier construction or adding parameters, among others. As some examples, consider (Datta and Das 2015; Carrizosa et al. 2008) and Lee et al. (2017), which adapt the support vector machine (SVM) classifier. In Datta and Das (2015) the decision boundary shift is combined with unequal misclassification penalties. A biobjective problem is introduced in Carrizosa et al. (2008), which simultaneous minimizes the misclassification rates. In Lee et al. (2017), the authors propose a new weight adjustment factor that is applied to a weighted SVM. This paper is structured as follows. In Sect. 2, the cost-sensitive constrained Lasso (CSCLasso) is introduced and some key issues are discussed. Section 3considers theoretical properties of the CSCLasso, as the existence and uniqueness of solution, limit behaviour (in terms of the penalty parameter) and consistency. Section 4presents a detailed numerical analysis with both simulated and real datasets, and finally, some conclusions and extensions are provided in Sect. 5. Technical proofs are relegated to the Appendix. 2 The cost-sensitive constrained Lasso: definition and key aspects This section presents the cost-sensitive constrained Lasso, which, as will be seen, is defined through an optimization problem with constraints related to prediction errors for individuals of interest. In addition, some computational details, as well as different key aspects concerning the tuning parameters of our proposal, are presented. 2.1 Definition The proposed CSCLasso is a novel variant of the Lasso where we shall demand that the prediction errors for the groups of interest are below certain threshold values, 123 A cost-sensitive constrained Lasso min β 1 n0 y0−X0β2+λAβ1 s.t. 1 n1 y1−X1β2−f1≤0, . . . 1 nL yL−XLβ2−fL≤0. (3) In the previous formulation, (y0,X0)is the set of observations used to build the sparse model with overall minimum MSE, which can be the complete dataset (y,X),ora subset of smaller size. Additionally, let (yl,Xl),l=1,...,L, define groups of interest (not necessarily disjoint), where the MSE predictions are to be controlled. Then, nl is the number of instances related to group l. Finally, f=(f1,..., fL)contains the different threshold values for the MSE on the different groups. The solution of optimization problem (3) will be denoted by ˆ βCSCLasso(λ). From the formulation (3) it is natural to wonder whether running a Lasso on just the groups of interest is more advantageous. However, if a single Lasso is run on the groups of interest, dramatically bad predictions can be obtained when the resulting model is applied to new observations outside those groups, which is not the case for our approach. The same issue arises when a different Lasso model is built on each group of interest, but, in addition, new observations are not given with their group of origin. Contrary to what happens with our novel approach (3), the Lpredictions obtained through the L different estimated Lasso models may not be suitable to give a final prediction for such new samples. The proposed method can be formulated as a Lasso with weighted quadratic penalties in the objective function associated with the different groups, but finding real meaning to their parameters (one per group) to be chosen is not an easy task (see Carrizosa and Romero-Morales (2001) and the references therein) and the full control over the accuracy prediction on certain individuals of interest would disappear. However, the parameters f=(f1,..., fL)involved in our model have a clear interpretation and, in addition, this formulation enables us to bound the prediction errors in the different groups of interest. As an example, in the illustration of the method in Sect. 1related to the prostate dataset, whereas the training set was used in the objective function with A=0|I8×8, the prediction error over the young population of the training set, (y1,X1), is controlled through a performance constraint ( f1=0.357). In a real application, once the Lgroups of interest are selected by the user, threshold values f1,..., fLhave to be fixed. Note that these thresholds will depend directly on the dataset in question and the considered groups of interest. As a first option, they could be fixed by the user according to her demand, but therefore unfeasibility problems may appear when solving the CSCLasso problem (3). For that reason, in Sect.2.3 two procedures for determining such threshold values so that (3) is feasible are given. Next, some other aspects related to the formulation of the CSCLasso and its resolution will be discussed. 123 R. Blanquero et al. 2.2 Computational details The CSCLasso problem as defined by (3) is a non-differentiable convex optimization problem with quadratic and convex constraints. However, if we rewrite the nondifferentiable term in (3)as Aβ=u+−u−, where u+=(u+ 1,...,u+ p)and u−=(u− 1,...,u− p)are new vectors of positive auxiliary variables, a differentiable version for the CSCLasso problem (3) is obtained in a straightforward manner as min β,u+,u− 1 n0 y0−X0β2+λ p  j=1 u+ j+λ p  j=1 u− j s.t. 1 n1 y1−X1β2−f1≤0, . . . 1 nL yL−XLβ2−fL≤0, Aβ=u+−u−, u+,u−≥0. This previous smooth formulation for the CSCLasso eases its resolution notably, since efficient solvers for quadratically constrained programming problems, such as Gurobi Optimization (2018), are available. In particular, the Gurobi R interface will be used in this work to obtain all numerical results. Another remark concerning the formulation of the CSCLasso is that, instead of using the sum of squared deviations, least absolute deviations could have been considered. Then, (3) would be reduced to a regression problem under linear inequality constraints, as those described in James et al. (2019), Gaines et al. (2018) and Hu et al. (2015). Nevertheless, to cope the non-differentiability of the absolute value function, a huge number of constraints and new auxiliary variables, which would depend on n, should have been added. Consequently, these constrained approaches are likely to face severe numerical difficulties in practice for large datasets. 2.3 The choice of threshold values As commented in Sect. 2.1, threshold values f1,..., fLcould be fixed by the user. If the user is too demanding, imposing very low MSE threshold values for (some of) the different groups, the optimization problem may become unfeasible. Although a try- and-error procedure may be used, it would be very helpful to have strategies yielding feasible solutions. Here we propose two procedures for determining f1,..., fLin such a way that (3)isfeasible. 123 A cost-sensitive constrained Lasso First, we propose a choice of the threshold values so that they are close to the OLS results, fl=(1+τ)MSEl(ˆ βols ), l=1,...,L,(4) where MSEl(β)=1 nl yl−Xlβ2,l=1,...,Land τ≥0 is a small parameter whose meaning is the percentage of worsening with respect to the OLS prediction error. For the numerical example in Sect. 1, we could have imposed the threshold for the MSE over Group 1 equal to 0.352, which is a 10% (τ=0.1) more than MSE1(ˆ βols )=0.319. The choice (4) deals with the heterogeneity coming from the variability of the different groups (MSElis different across groups). Nevertheless, when heterogeneity related to the importance of each group is also considered, the parameter τcan be replaced in (4)byτl,l=1,...,L. Next, we shall compute the minimum value of τ,τmin,soasto(3) is feasible. That is, the minimum τso that there exists β∗satisfying max l MSEl(β∗) MSEl(ˆ βols )−1≤τ, and, therefore, τmin will be given as τmin =max l MSEl(β∗) MSEl(ˆ βols )−1. Such τmin can be found as the optimal value of the following linear problem with convex and quadratic constraints min β,zz s.t. z≥MSEl(β) MSEl(ˆ βols ) −1,∀l=1,...,L.(5) The feasible version of the CSCLasso optimization problem can be formulated as min β 1 n0 y0−X0β2+λAβ1 s.t. 1 n1 y1−X1β2−(1+τ)MSE1(ˆ βols )≤0, . . . 1 nL yL−XLβ2−(1+τ)MSEL(ˆ βols )≤0, (6) where τ≥τmin. 123 R. Blanquero et al. Finally, note that if τis big enough, then solving (6) is equivalent to solve the unconstrained problem. Indeed, it is possible to find the value of τ,τmax(λ), such that both the constrained and unconstrained problems are equivalent τmax(λ) =max l∈{1,...,L} MSEl(ˆ βLasso(λ)) MSEl(ˆ βols ) −1.(7) A second possible choice for the threshold values follows an analogous approach but, instead of considering the results of the OLS, we shall consider the mean squared error of the Lasso, as in the numerical example introduced in Sect. 1. For each l= 1,...,L, fl=(1−γ)MSEl(ˆ βLasso(λ)), l=1,...,L,(8) where γ≥0 is related to the desired percentage of improvement over the Lasso solution (γ=0.1 in the numerical example of Sect. 1). In this case, we will compute the maximum value of γ,γmax, in such a way that (3) is feasible under (8), and the linear problem associated with γmax is max β,zz s.t. 1 −MSEl(β) MSEl(ˆ βLasso(λ)) ≥z,∀l=1,...,L.(9) Thus, another possible feasible version of the CSCLasso optimization problem can be formulated as min β 1 n0 y0−X0β2+λAβ1 s.t. 1 n1 y1−X1β2−(1−γ)MSE1(ˆ βLasso(λ)) ≤0, . . . 1 nL yL−XLβ2−(1−γ)MSEL(ˆ βLasso(λ)) ≤0, (10) where γ≤γmax. Note that the two choices previously described for selecting the threshold values are not unique. Indeed, instead of using the MSE, another statistical measure as the R-squared can be considered. Further details about how they perform in numerical applications are described in Sects. 2.4 and 4. 123 A cost-sensitive constrained Lasso 2.4 The role of the tuning parameters The CSCLasso, as defined by (6)or(10), is stated in terms of two tuning parameters, λand τor λand γ, respectively. The first one, λ, is related to the sparsity of the solution, and the second one is linked to the user’s demanding level, since the degree of requirement increases as τ→τmin (or γ→γmax). In this section we investigate how the solution of the CSCLasso changes when λand τjointly vary (analogous results are obtained if λand γare analyzed instead). With this purpose, consider again the experimental setting as in the example of Sect. 1related to prostate dataset with A=0|Ip×p(Lasso objective function), but in this occasion assume that the prediction errors of both groups (the young and the elderly people) shall be controlled. The interval of variation of the parameter λis set to Iλ=[0,30]. Moreover, according to (5), the smallest value of τsuch that the CSCLasso optimization problem (6) is feasible is τmin =0.055. On the other hand, following (7), τmax = max λ∈[0,30]τmax(λ) =2.355, although we will enlarge the interval of variation of τto also visualize the unconstrained solution; such interval will be finally set as Iτ=[τmin,τ max +2]=[0.055,4.355]. Figure 1represents, via a heat map, the solution for ˆ βCSCLasso 1(λ) for the different values of (λ, τ ) in a grid contained in Iλ×Iτ. Some conclusions can be drawn from the figure. Consider first the cases where τand λare big enough. Since, in this case, τ≥τmax, then, as commented at the end of the previous section, solving (6) is equivalent to solving the Lasso. Therefore, ˆ βCSCLasso(λ) =ˆ βLasso(λ) =0will be the optimal solution, provided that λis big enough. Analogously, if τ≥τmax but λis small, then ˆ βCSCLasso(λ) =ˆ βLasso(λ), which will be equal to zero or not depending on the importance of the variable. When τis small, the constraints are demanding and, even for large values of λ, it might happen that ˆ βCSCLasso 1(λ) = ˆ βLasso 1(λ) =0, as it is the case. Figure 7(see Appendix for further results) represents the analogous heat maps concerning ˆ βCSCLasso 2(λ), . . . , ˆ βCSCLasso 8(λ). A similar discussion as with ˆ βCSCLasso 1(λ) is applicable to these figures. An interesting remark to be made concerns the importance of each variable: while variable 1 is the only one selected for the Lasso, the CSCLasso returns a less sparse solution in this case, since predictor variables 1, 2, 4 and 5 turn out to be significant. However, this is not the rule, since there are examples where the level of sparsity is higher for the CSCLasso, as will be shown in Sect. 4. 3 Theoretical properties In this section we discuss some theoretical results concerning the CSCLasso model. In Sect. 3.1, the existence of a unique optimal solution to Problem (3) is proven for a fixed value of λ≥0. Section 3.2 deals with the limit behavior of the solution when λapproaches infinity. Finally, some consistency properties of the CSCLasso solution are derived in Sect. 3.3 from the Sample Average Approximation theory (see Shapiro et al. 2009). 123 R. Blanquero et al. Theorem 2 Assume that E[X2]<∞,E[Y2]<∞,E[YX] <∞. Then, ˆνCSCLasso(λ) converges to νCSCLasso(λ) and ˆ βCSCLasso(λ) converges to βCSCLasso(λ) with probability one (w.p. 1). Finally, note that the theoretical results that have been studied in this work are also applicable to other versions of constrained Lasso as long as the feasible set is convex and closed, as is the case with the above-mentioned works (James et al. 2019; Gaines et al. 2018;Huetal.2015). 4 Numerical experiments In this section, the behaviour and performance of our approach is illustrated throughout an extensive empirical study. In particular, using both simulated and real datasets, the aim of the experiments shall be to improve the prediction errors of the Lasso in one or more groups of interest. Or in other words, threshold values shall be fixed as in (10). Since our proposal is a novel extension of the Lasso, we will also show the results under the Lasso, not only for those groups that are controlled (for which obviously, the Lasso performs worse) but also for the non-controlled groups. In this way the CSCLasso can be better inspected in comparison to the Lasso. Other aspects as the overall MSE and the percentage of non-zero coefficients in the regression model, among others, will be explored. Such measures will be estimated through median values using a fivefold cross validation approach. To this end, the dataset will be split at each fold into three sets: the so-called training, validation and test sets. The training set is used to fit the model, the validation set is used to estimate prediction error for model selection and the test set is used for assessment of the generalization error of the final chosen model. 4.1 A simulation study The generation of the synthetic datasets in this section follows that of Ollier and Viallon (2017), where an overparameterized regression model is considered to cope with stratified data. A number of groups K=20 is set and two different sample sizes per group are considered, nk={150,500},fork=1,...,K. The number of predictors pwill be chosen from {20,100,500}. The matrix of predictor values Xis generated according to a multivariate normal distribution with zero mean and covariance matrix being a Toeplitz matrix with element (i,j)equal to 0.5|i−j|. Regarding the response vector, a set of 20 predictors are randomly selected (with indexes included in a set P0), while the rest of predictors are noise (that is, βj=0 for j/∈P0). The coefficients of the significant 20 predictors are chosen as follows. First, consider 10 random predictors out of the 20 selected. For such predictors, if the group k>6 then βj=1 and, otherwise, βj=1+K1 2. For the other 10 predictors, βj=1ifk≤6 and βj=1+K1 2otherwise. In this way, the predictors behave differently depending on the group. Finally, the response vector for each group is generated according to the standard linear regression model with normal error. 123 A cost-sensitive constrained Lasso Once the synthetic dataset is built and its response and predictor variables have been standardized, the CSCLasso with A=(0|Ip×p)is run with constraints imposed over the first six groups. The choice of λwill change at each fold. A grid in λis built as in Sect. 3.2, and, the value of λwhich leads to the lower overall MSE in the validation set is selected. Table 2shows the median prediction errors per group k(MSEk), k=1,...,6, obtained by the Lasso (rows in bold values) and the corresponding values obtained under the CSCLasso for different threshold values f. In particular, the values of fhave been set as improvement percentages over the Lasso values, where the improvement levels are 3%, 5%, 7%, 10%, 15% and 20% (γequal to 0.03, 0.05, 0.07, 0.15 and 0.20, respectively). The results are obtained for different combinations (p,nk), where, as commented before, pis chosen from the set {20,100,500}and nkfrom {150,500}. For example, if nk=150 and p=20, the median of the mean squared error for the Lasso in Group 1 was equal to 1.084. If the goal is to achieve an improvement of 3%, fmust be chosen equal to 1.052. The median of the mean squared errors for the CSCLasso is equal to 0.933 in this case (results obtained on the test sample). It is important to remark that, for some levels of improvement, the CSCLasso problem is unfeasible due to the fact that γmax for such datasets is smaller than the required γ, and, therefore, those cases are represented as empty spaces in the table. It must be noted that γmax will also depend on each fold in the cross-validation because it is associated with the partition of the data, since the MSElin (9) depends on such partition (l=1,...,L). It is also worth mentioning that the constraints will always be satisfied on the training set but not necessarily on the test set, see, for example, the case k=3, nk=150, p=100 with improvement level equal to 15%. This phenomenon is particularly common as pincreases and nk decreases (see for example the values corresponding to p=500 and nk=150 in Table 2). We next investigate how the improvement in the prediction errors of the groups of interest affects the prediction errors in the rest of the groups, the overall prediction error and the sparsity level. Figure 4represents the percentage of non-zero (NZ) coefficients and the overall prediction error for different sample sizes, different levels of improvement and for p=100. Lasso results are also included. For instance, when nk=300 (black squares in Fig. 4), the NZ percentage for Lasso is 39.60 with an associated overall MSE of 0.734; whereas running the CSCLasso demanding a 3% of improvement over the first six groups, we achieve a NZ percentage of 38.61, and an overall MSE of 0.735. In general terms, it can be seen that the sparsity of the solution decreases with the improvement level: smaller squares, which represent smaller imposed improvement percentages, are on the left of bigger squares (which are associated with demanding percentage of improvement). Then, if the user is very demanding in predicting a specific group, this implies, in the majority of the cases, a less sparse solution. Notwhitstanding, when no level of improvement is imposed (Lasso problem), the solution can be less sparse than in the CSCLasso, as in the case of nk=300. This also occurs when p=500 and nk=150 (see the bottom graphic of Fig. 10 in Appendix: further results). Furthermore, the overall prediction error slightly worsens with the improvement level, due to the worsening of the predictions in the uncontrolled groups. 123 R. Blanquero et al. Table 2 Median errors over test sets for synthetic datasets pn k=150 nk=300 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 20 MSEk(Lasso)1.084 0.904 0.768 0.925 0.944 0.674 0.801 0.770 0.966 0.937 0.776 0.938 Improv. f 1.052 0.877 0.745 0.898 0.916 0.653 0.777 0.747 0.937 0.909 0.753 0.910 3% MSEk0.933 0.722 0.695 0.859 0.822 0.574 0.752 0.689 0.810 0.859 0.744 0.870 Improv. f 1.030 0.859 0.730 0.879 0.897 0.640 0.761 0.732 0.918 0.890 0.737 0.891 5% MSEk0.921 0.708 0.689 0.848 0.804 0.564 0.734 0.682 0.787 0.844 0.736 0.859 Improv. ––––––0.7450.7170.8990.8710.7220.873 7% 0.717 0.677 0.768 0.834 0.721 0.847 Improv. –––––––––––– 10% Improv. –––––––––––– 15% Improv. –––––––––––– 20% 100 MSEk(Lasso)1.491 1.139 0.875 1.496 1.151 0.759 1.104 1.151 0.932 0.981 1.311 1.142 Improv. f 1.447 1.105 0.848 1.451 1.116 0.736 1.070 1.117 0.904 0.951 1.272 1.108 3% MSEk1.234 0.948 0.781 1.322 0.945 0.708 1.044 1.145 0.911 0.963 1.306 1.139 Improv. f 1.417 1.082 0.831 1.421 1.093 0.721 1.048 1.094 0.885 0.932 1.245 1.085 5% MSEk1.226 0.941 0.777 1.317 0.937 0.706 1.025 1.123 0.897 0.947 1.311 1.142 Improv. f 1.387 1.059 0.813 1.391 1.070 0.706 1.026 1.071 0.866 0.912 1.219 1.062 7% MSEk1.219 0.933 0.773 1.311 0.929 0.704 1.005 1.111 0.883 0.934 1.309 1.141 Improv. f 1.342 1.025 0.787 1.346 1.035 0.683 0.993 1.036 0.838 0.883 1.180 1.028 10% MSEk1.207 0.921 0.767 1.303 0.917 0.702 0.978 1.094 0.864 0.915 1.311 1.131 Improv. f 1.268 0.968 0.743 1.271 0.978 0.645 0.938 0.979 0.792 0.834 1.114 0.971 123 A cost-sensitive constrained Lasso Table 2 continued pn k=150 nk=300 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 Group 1 Group 2 Group 3 Group 4 Group 5 Group 6 15% MSEk1.128 0.903 0.765 1.250 0.919 0.695 0.936 1.061 0.833 0.882 1.274 1.087 Improv. f 1.193 0.911 0.700 1.196 0.920 0.607 0.883 0.921 0.745 0.785 1.049 0.914 20% MSEk1.155 0.909 0.771 1.251 0.900 0.700 0.895 1.036 0.803 0.855 1.227 1.046 500 MSEk(Lasso)1.306 1.133 1.318 1.261 1.246 1.473 1.151 1.204 1.171 1.148 1.278 1.068 Improv. f 1.267 1.099 1.279 1.223 1.208 1.429 1.116 1.168 1.136 1.114 1.240 1.035 3% MSEk1.270 1.133 1.314 1.248 1.246 1.453 1.029 1.077 1.047 1.022 1.186 0.961 Improv. f 1.241 1.076 1.252 1.198 1.183 1.400 1.093 1.144 1.113 1.091 1.214 1.014 5% MSEk1.257 1.133 1.308 1.245 1.245 1.437 1.025 1.060 1.032 1.004 1.165 0.945 Improv. f 1.215 1.053 1.226 1.173 1.158 1.370 1.070 1.119 1.089 1.068 1.189 0.993 7% MSEk1.246 1.133 1.312 1.241 1.246 1.425 1.018 1.042 1.023 0.987 1.144 0.929 Improv. f 1.176 1.019 1.186 1.135 1.121 1.326 1.036 1.083 1.054 1.033 1.150 0.961 10% MSEk1.231 1.132 1.313 1.230 1.246 1.401 1.005 1.016 1.006 0.961 1.114 0.905 Improv. f 1.110 0.963 1.120 1.072 1.059 1.252 0.978 1.023 0.995 0.976 1.086 0.907 15% MSEk1.207 1.114 1.271 1.193 1.241 1.376 0.971 0.977 1.039 0.923 1.060 0.866 Improv. f 1.045 0.906 1.054 1.009 0.996 1.179 0.921 0.963 0.937 0.919 1.022 0.854 20% MSEk1.184 1.081 1.266 1.150 1.230 1.354 0.952 0.972 0.995 0.908 1.057 0.857 123 R. Blanquero et al. Fig. 4 Median overall MSE over the test sets and NZ percentage under the choice p=100 Figure 5represents the prediction errors over the groups that are not controlled as well as the overall mean squared error, for different improvement levels, nk=150 (top figure) and nk=300 (bottom) when p=100. In the figure, Lasso values are also shown. From the figure, it can be concluded that the Lasso performs better in the uncontrolled groups, since the prediction errors worsen under our proposal. However, the overall mean squared error remains almost constant (since the improved errors compensate the more deteriorated ones). Similar conclusions can be drawn under the choices p=20 and p=500 (see Figs. 8,9in Appendix: further results, respectively). Next, we test how the solution of the CSCLasso behaves with respect to other global performance measures as the l2distance, false positive and negative rates, which are defined not in terms of prediction errors, but on the correct fitting of the generator process (see Yu and Liu 2016). In particular, the l2distance is defined as ˆ β−β2, where βis the vector of coefficients that generated the datasets (described at the beginning of this section), and ˆ β(λ) are the estimators. In addition, the false positive rate (FPR) and false negative rate (FNR) are calculated as follows: FPR =|j:βj=0& ˆ βj(λ) =0| |j:βj=0|, FNR =|j:βj= 0& ˆ βj(λ) =0| |j:βj= 0|, where j=1,...,p. The median of these three measures as well as the median of the overall MSE (already shown in Figs. 4,5and 8,9and 10 in Appendix: further results), are presented in Table 3. For the choices where p=20, the FPR values 123 A cost-sensitive constrained Lasso Fig. 5 Median MSEkover the test sets for k=7,...,20 under p=100 features and, nk=150 (top) and nk=300 (bottom). Each subgraph represents one group and the Y-axis shows the different percentages of improvement 123 R. Blanquero et al. Fig. 6 A two-dimensional graph of the logarithm of the user times in seconds for nk=300 as pincreases are not given since all the predictors have associated non-zero coefficients when the datasets were created. From this table it can be deduced that similar or even better results, comparing with those of the Lasso, are obtained in the majority of the cases across the four different measures. A final remark concerning the computational cost of the CSCLasso when comparing with Lasso is as follows. The median user time required to solve the problem with the largest dataset considered in this study (nk=300 and p=500) is 0.85 s when the Lasso is run on Intel(R) Core(TM) i7-7500U CPU at 2.70 GHz 2.90 GHz with 8.0 GB of RAM; whereas the CSCLasso requires 6.60 s. Nevertheless, to better understand how the computation time behaves depending on pvalue, a grid in this parameter has been inspected, while nkis set to 300. Figure 6depicts the logarithm of the user times in seconds obtained under Lasso and CSCLasso models when nk=300 and pchanges. Then, under a reasonable computational cost, the desired results are achieved. Further analyses regarding the computational times are shown in the Appendix (Fig. 11). 4.2 Leukemia dataset: a gene expression dataset The real stratified dataset described in Kouno et al. (2013) is explored here. The data contain information related to myeloid monocytic leukemia cells undergoing differentiation to macrophages. In particular, the dataset is formed by expression levels of 45 transcription factors (response and predictor variables) measured at 8 distinct times (groups) of the differentiation process. As in Ollier and Viallon (2017), the aim is to predict the EGR2 transcription factor in terms of the other p=44 factors. The sample size per group is equal to 120. Similarly as in Sect. 4.1, the Lasso was run and the overall prediction errors, individual prediction errors per group and percentage of non-zero coefficients are recorded. The records in Group 1 yield the best MSE 123 A cost-sensitive constrained Lasso using Lasso model. Therefore, we may be interested in obtaining an even better fitting for such data. The CSCLasso problem is solved with threshold values smaller than the Lasso error, which turns out to be 0.370. Table 4shows the obtained median results for an assortment of improvement levels, namely, 5%, 7%, 10% and 15% or, equivalently, γis equal to 0.05, 0.05, 0.07 and 0.15, respectively. From that table, it can be seen how the prediction error of interest (corresponding to Group 1) decreases with the improvement level, as expected. Similarly as in Sect. 4.1, the overall mean squared error does not exhibit significant changes, while the prediction errors in the uncontrolled groups do not exhibit the same behaviour. Some of them slightly improve (Group 6), others slightly worsen (as the Group 5 and Group 8) and others remain constant (as Group 2). Finally, in regards to the sparsity of the solution, for this dataset, less sparse solutions are obtained by CSCLasso in comparison with the Lasso ones. 4.3 Communities and crime dataset In this section, a real dataset from the UCI Machine Learning Repository (Lichman 2013) will be analyzed. In particular, the so-called Communities and Crime Unnormalized Data Set shall be considered. The dataset is about communities within the United States and has already been inspected in the literature (see Redmond and Baveja 2002). This dataset combines crime information from the FBI databases (U.S. Department 1995) as well as socio-economic and law enforcement data from U.S. Department (1992) and U.S. Department (1992), respectively. The dataset is formed by p=124 predictors, 23 of which present missing values, and n=2215 instances, where the response variable measures the number of murders per 100K population. The predictor variables with missing values are not consider for the next experiments. As such, we finally consider p=101 predictors. Additionally, for each instance (community), the region from which it comes is known. Thus, if we were interested in obtaining a good prediction in a certain region, say Midwest, we could control these communities by including a performance constraint. Table 5shows the median errors over the test set for Group 1, formed by the communities of Midwest, and over the rest of communities (Group 2). In terms of overall MSE and MSE over the two groups, similar conclusions as in Sect. 4.2 are drawn. Whereas different improvement levels are imposed, the MSE of interest (MSE1) is getting smaller but the overall prediction error is almost not affected by the constraint. Lastly, regarding the sparsity of the solution, an analogous behaviour as that observed in the case of simulated data is obtained: the solution becomes less sparse with the improvement level. As previously commented, the groups of interest may overlap. As an illustration, assume that the interest is in controlling the prediction error in communities of Midwest or communities with a population density larger than or equal to the 75th percentile. Let Group 1 denote the communities from Midwest, while Group 2 represents the communities where the density of population is higher than the 75th percentile. For instance, if we aim to improve in a 7% the errors obtained by the Lasso model (equal to 0.513 and 0.442), then the CSCLasso results become 0.475 and 0.441, respectively. 123 R. Blanquero et al. Table 3 Median performance measures over test sets for synthetic datasets pn k=150 nk=300 Overall MSE l2distance FPR FNR Overall MSE l2distance FPR FNR 20 Lasso 0.667 4.149 – 0.050 0.666 4.146 – 0.000 Improv. 3% 0.691 4.108 – 0.000 0.673 4.099 - 0.000 Improv. 5% 0.695 4.104 – 0.000 0.683 4.089 – 0.000 Improv. 7% – – – – 0.693 4.077 – 0.000 Improv. 10% – – – – – – – – Improv. 15% – – – – – – – – Improv. 20% – – – – – – – – 100 Lasso 0.732 2.931 0.275 0.250 0.734 2.729 0.250 0.150 Improv. 3% 0.740 2.857 0.388 0.100 0.735 2.721 0.163 0.150 Improv. 5% 0.741 2.855 0.375 0.100 0.734 2.721 0.163 0.100 Improv. 7% 0.742 2.853 0.400 0.100 0.734 2.718 0.163 0.100 Improv. 10% 0.745 2.849 0.400 0.100 0.732 2.704 0.275 0.100 Improv. 15% 0.751 2.844 0.425 0.100 0.735 2.678 0.288 0.050 Improv. 20% 0.759 2.835 0.463 0.100 0.738 2.655 0.288 0.100 500 Lasso 0.772 2.778 0.135 0.300 0.744 2.750 0.065 0.300 Improv. 3% 0.772 2.779 0.040 0.350 0.744 2.658 0.063 0.250 Improv. 5% 0.772 2.775 0.050 0.300 0.745 2.653 0.075 0.250 Improv. 7% 0.772 2.769 0.042 0.350 0.746 2.647 0.077 0.250 Improv. 10% 0.772 2.760 0.056 0.350 0.748 2.632 0.104 0.250 Improv. 15% 0.771 2.754 0.063 0.300 0.752 2.640 0.129 0.250 Improv. 20% 0.774 2.735 0.069 0.350 0.760 2.607 0.148 0.200 123 A cost-sensitive constrained Lasso Table 4 Median errors over test set for gene expression dataset. Constraints imposed over Group 1 f1Overall MSE M SE1MSE2MSE3MSE4MSE5MSE6MSE7MSE8NZ Lasso – 0.620 0.370 0.417 0.902 0.496 0.480 0.535 0.496 0.685 53.33 Improv. 5% 0.352 0.623 0.357 0.417 0.918 0.500 0.555 0.512 0.497 0.719 73.33 Improv. 7% 0.344 0.626 0.348 0.418 0.915 0.504 0.565 0.514 0.497 0.722 66.67 Improv. 10% 0.333 0.630 0.335 0.418 0.911 0.510 0.574 0.513 0.498 0.728 66.67 Improv. 15% 0.315 0.636 0.331 0.415 0.903 0.523 0.591 0.512 0.502 0.737 73.33 123 R. Blanquero et al. Appendix: Further results See Figs. 7,8,9and 10. To fully understand how the computation time behaves depending on nkand p values, a grid in both parameters have been inspected. Figure 11 displays the logarithm of the user times in seconds obtained under Lasso and CSCLasso models when nkand pchange. The perspective drawn in the top left figure shows that Lasso model (bottom surface) is solved faster and in a smoother way. Besides, whereas smaller times are obtained for both methods when nkand pare small, the biggest times are associated to nk=300 and p=500. Fig. 7 Heat maps of ˆ βCSCLasso(λ) =(ˆ βCSCLasso 1(λ), . . . , ˆ βCSCLasso 8(λ)) using prostate dataset 123 A cost-sensitive constrained Lasso Fig. 8 Median MSEkover the test sets for k=7,...,20 under p=20 features and the two nkoptions. Each subgraph represents one group and the Y-axis shows the different percentages of improvement 123 R. Blanquero et al. Fig. 9 Median MSEkover the test sets for k=7,...,20 under p=500 features and the two nkoptions. Each subgraph represents one group and the Y-axis shows the different percentages of improvement 123 A cost-sensitive constrained Lasso Fig. 10 Median overall MSE over the test sets and NZ percentage under the choice p=20 (top) and p=500 (bottom) 123 R. Blanquero et al. Fig. 11 Four perspectives of the logarithm of the user times in seconds for Lasso (bottom surface in the four graphics) and CSCLasso (top surfaces) models across a grid in nkand p References Bradford JP, Kunz C, Kohavi R, Brunk C, Brodley CE (1998) Pruning decision trees with misclassification costs. In: Nédellec C, Rouveirol C (eds) Machine learning: ECML-98. Springer, Berlin, pp 131–136 Bühlmann P, Van-De Geer S (2011) Statistics for high-dimensional data. Springer, Berlin Carrizosa E, Romero-Morales D (2001) Combining minsum and minmax: a goal programming approach. Oper Res 49(1):169–174 Carrizosa E, Martín-Barragán B, Morales DR (2008) Multi-group support vector machines with measurement costs: a biobjective approach. Discrete Appl Math 156:950–966 Datta S, Das S (2015) Near-Bayesian support vector machines for imbalanced data classification with equal or unequal misclassification costs. Neural Netw 70:39–52 Donoho DL, Johnstone IM, Kerkyacharian G, Picard D (1995) Wavelet shrinkage: Asymptopia? J R Stat Soc Ser B (Methodol) 57(2):301–369 Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499 Fan J, Li R (2001) Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc 96(456):1348–1360 Freitas A, Costa-Pereira A, Brazdil P (2007) Cost-sensitive decision trees applied to medical data. In: Song IY, Eder J, Nguyen TM (eds) Data warehousing and knowledge discovery. Springer, Berlin, pp 303–312 Friedman J, Hastie T, Tibshirani R (2001) The elements of statistical learning. Springer, Heidelberg 123 A cost-sensitive constrained Lasso Gaines BR, Kim J, Zhou H (2018) Algorithms for fitting the constrained Lasso. J Comput Graph Stat 27(4):861–871 Garside MJ (1965) The best sub-set in multiple regression analysis. J R Stat Soc Ser C (Appl Stat) 14(2– 3):196–200 Gurobi Optimization L (2018) Gurobi optimizer reference manual. http://www.gurobi.com Hastie T, Tibshirani R, Wainwright M (2015) Statistical learning with sparsity. Chapman and Hall/CRC, New York He H, Ma Y (2013) Imbalanced learning: foundations, algorithms, and applications. Wiley, Hoboken Hoerl AE, Kennard RW (1970) Ridge regression: biased estimation for nonorthogonal problems. Technometrics 12(1):55–67 Hu Q, Zeng P, Lin L (2015) The dual and degrees of freedom of linearly constrained generalized lasso. Comput Stat Data Anal 86:13–26 James GM, Paulson C, Rusmevichientong P (2019) Penalized and constrained optimization: an application to high-dimensional website advertising. J Am Stat Assoc 1–31 Kouno T, de Hoon M, Mar JC, Tomaru Y, Kawano M, Carninci P, Suzuki H, Hayashizaki Y, Shin JW (2013) Temporal dynamics and transcriptional control using single-cell gene expression analysis. Genome Biol 14(10):R118 Lee W, Jun CH, Lee JS (2017) Instance categorization by support vector machines to adjust weights in adaboost for imbalanced data classification. Inf Sci 381(Supplement C):92–103 Lichman M (2013) UCI machine learning repository. http://archive.ics.uci.edu/ml Ollier E, Viallon V (2017) Regression modelling on stratified data with the lasso. Biometrika 104(1):83–96 Prati RC, Batista GEAPA, Silva DF (2015) Class imbalance revisited: a new experimental setup to assess the performance of treatment methods. Knowl Inf Syst 45(1):247–270 Redmond M, Baveja A (2002) A data-driven software tool for enabling cooperative information sharing among police departments. Eur J Oper Res 141(3):660–678 Rockafellar RT (1972) Convex analysis. Princeton University Press, Princeton Shapiro A, Dentcheva D, Ruszczy´nski A (2009) Lectures on stochastic programming: modeling and theory. SIAM, Philadelphia Simon N, Friedman J, Hastie T, Tibshirani R (2011) Regularization paths for Cox’s proportional hazards model via coordinate descent. J Stat Softw 39(5):1–13 Stamey TA, Kabalin JN, McNeal JE, Johnstone IM, Freiha F, Redwine EA, Yang N (1989) Prostate specific antigen in the diagnosis and treatment of adenocarcinoma of the prostate: II. Radical prostatectomy treated patients. J Urol 141(5):1076–1083 Sun Y, Wong AK, Kamel MS (2009) Classification of imbalanced data: a review. Int J Pattern Recognit Artif Intell 23:687–719 Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodol) 58(1):267–288 Tibshirani R, Saunders M, Rosset S, Zhu J, Knight K (2005) Sparsity and smoothness via the fused lasso. J R Stat Soc Ser B (Stat Methodol) 67(1):91–108 Tibshirani RJ, Taylor J (2011) The solution path of the generalized Lasso. Ann Stat 39(3):1335–1371 Torres-Barrán A, Alaíz CM, Dorronsoro JR (2018) ν-SVM solutions of constrained Lasso and elastic net. Neurocomputing 275:1921–1931 U.S. Department of Commerce, Bureau of the Census, Census of Population and Housing 1990 United States: Summary Tape File 1a & 3a (Computer Files), U.S. Department of Commerce, Bureau of the Census Producer, Washington, DC and Inter-university Consortium for Political and Social Research Ann Arbor, Michigan (1992) U.S. Department of Justice, Bureau of Justice Statistics, Law Enforcement Management and Administrative Statistics (Computer File) U.S. Department Of Commerce, Bureau of the Census Producer, Washington, DC and Inter-university Consortium for Political and Social Research Ann Arbor, Michigan (1992) U.S. Department of Justice, Federal Bureau of Investigation, Crime in the United States (Computer File) (1995) Yu G, Liu Y (2016) Sparse regression incorporating graphical structure among predictors. J Am Stat Assoc 111(514):707–720 Yuan M, Lin Y (2006) Model selection and estimation in regression with grouped variables. J R Stat Soc Ser B (Stat Methodol) 68(1):49–67 Zou H (2006) The adaptive lasso and its oracle properties. J Am Stat Assoc 101(476):1418–1429 123 R. Blanquero et al. Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc Ser B (Stat Methodol) 67(2):301–320 Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. 123