Model-assisted Multi-criteria Tuning of an Event Detection Software under Limited Budgets

Schriftenreihe CIplus, Band 2/2012 Thomas Bartz-Beielstein, Wolfgang Konen, Horst Stenzel, Boris Naujoks Model-assisted Multi-criteria Tuning of an Event Detection Software under Limited Budgets Martin Zaefferer, Thomas Bartz-Beielstein, Boris Naujoks, Tobias Wagner and Michael Emmerich Model-assisted Multi-criteria Tuning of an Event Detection Software under Limited Budgets Martin Zaefferer1, Thomas Bartz-Beielstein1, Boris Naujoks1, Tobias Wagner2, and Michael Emmerich3 1Faculty for Computer and Engineering Sciences Cologne University of Applied Sciences, 51643 Gummersbach, Germany [email protected] 2Institute of Maching Technology (ISF) TU Dortmund University, 44227 Dortmund, Germany [email protected] 3Leiden Institute for Advanced Computer Science Leiden University, The Netherlands, [email protected] Department of Computer Science, Cologne University of Applied Sciences, Germany. Schriftenreihe CIplus TR 2/2012. ISSN 2194-2870 Abstract. Formerly, multi-criteria optimization algorithms were often tested using tens of thousands function evaluations. In many real-world settings function evaluations are very costly or the available budget is very limited. Several methods were developed to solve these costextensive multi-criteria optimization problems by reducing the number of function evaluations by means of surrogate optimization. In this study, we apply different multi-criteria surrogate optimization methods to improve (tune) an event-detection software for water-quality monitoring. For tuning two important parameters of this software, four state-of-theart methods are compared: S-Metric-Selection Efficient Global Optimization (SMS-EGO), S-Metric-Expected Improvement for Efficient Global Optimization SExI-EGO, Euclidean Distance based Expected Improvement Euclid-EI (here referred to as MEI-SPOT due to its implementation in the Sequential Parameter Optimization Toolbox SPOT) and a multi-criteria approach based on SPO (MSPOT). Analyzing the performance of the different methods provides insight into the working-mechanisms of cutting-edge multi-criteria solvers. As one of the approaches, namely MSPOT, does not consider the prediction variance of the surrogate model, it is of interest whether this can lead to premature convergence on the practical tuning problem. Furthermore, all four approaches will be compared to a simple SMS-EMOA to validate that the use of surrogate models is justified on this problem. MCO tuning of Event Detection Software 3 1 Introduction The time required for a process feedback can play a crucial role in many fields of industrial optimization. Complex and expensive real-world processes or time consuming simulations lead to large evaluation times. This restricts optimization processes to only a very limited number of such evaluations. Moreover, almost all industrial optimization tasks feature more than one quality criterion. Techniques from multi-criteria decision making, evolutionary multi-criteria optimization (EMO) in particular, were developed during the last decade to solve such tasks. The necessity to combine EMO techniques and optimization methods such as EGO [17] or SPO [1], which require a very small number of function evaluations only, should be self-evident. The application of such methods to realworld problems in industrial optimization provides a reasonable way to assess their feasibility. In contrast to artificial test functions, it allows for an assessment of the practical relevance for these kinds of problems. In this paper, which is an extended version of a paper submitted to the Evolutionary Multi-Criterion Optimization EMO Conference 2013, we focus on four different tuning methods which are applied to tune an anomaly detection software for water quality management. This problem is usually handled by receiver operator characteristic (ROC) analysis. Due to specific limitations of the software concerned, this can not be applied in the classical way. Rather, the ROC curve should be approximated by MCO methods. That means, the ROC curve can be interpreted as a Pareto front. Interpreting ROC curves from the multicriteria optimization perspective is an established approach in computational intelligence, see, e.g., [22]. In Sec. 2, we will summarize the former work performed in relevant research fields. The specific problem is presented in Sec. 3. The tuning algorithms (based on different SPO and EGO implementations) are described in Sec. 4. Section 5 describes the experimental setup, whereas the analysis is presented in in Sec. 6. Finally, Sec. 7 gives a summary of findings. The paper concludes with an outlook presented in Sec. 8. 2 Former research Surrogate modeling is not a new topic in optimization. Jin [16] provides a comprehensive overview of single-objective optimization with surrogate models. While methods like EGO or SPO for single criteria optimization are well established, the application of surrogate modeling procedures for multiple objectives is more recent. 2.1 Surrogate modeling in multi-criteria optimization Several approaches employ surrogate modeling in MCO, like the well established ParEGO by Knowles [20]. An overview of surrogate modeling in MCO is given by Knowles and Nakayama [21]. To balance exploration and exploitation within 4 Zaefferer et al. a limited budget situation, several methods try to employ infill criteria based on the expected improvement (EI). Two things are required for defining such a criterion: the definition of the improvement and an algorithm for computing its expectation [30]. Thereby, negative improvements are not possible, dominated solutions should result in an improvement of zero. As large variances potentially result in big improvements and big deteriorations are not penalized,these criteria also focus on the exploration of uncovered areas of the search space. It is of interest to see if the additional exploration is desirable for the problem at hand, or if the exploration provided by the initial design is already sufficient. In particular, since exploration is enforced by the requirement to cover the whole Pareto front (or set). 2.2 ROC analysis ROC provides means to select a threshold of a classifier based on trade-off between its True Positive Rate (TPR) and False Positive Rate (FPR). In case of an event detection software like CANARY [14, 26]4, TPR is the hit rate which is based on the number of correctly recognized events. FPR on the other hand is the false alarm rate. False alarms occur whenever the algorithm detects an event when actually none exists. The ROC curve shows the trade-off between TPR and FPR. Usually, it is drawn based on the threshold value of the classifier. This means, depending on the chosen threshold value one receives different pairs of TPR/FPR values which can be connected to a curve. To evaluate the performance of a classifier, the Area Under Curve (AUC) can be used. The worst possible classifier will have an AUC of 0.5, since all pairs of TPR and FPR will be on the straight line between the two extreme points of the curve. This performance would be equal to random guessing. The best possible classifier will have an AUC of 1, which means there is a configuration where no false alarms occur, all events are identified (cf. Fig. 1). In the case of CANARY, this form of measuring the performance can not be used, since the threshold value used in CANARY cannot be chosen independently. Therefore, each different setting of the threshold has to be considered as a new classifier. The ROC curve can then be used to compare performance of the different classifiers. Consequently, the threshold value is one of the parameters to be optimized. 2.3 MCO in ROC Analysis The ROC curve can be interpreted as a Pareto front, although it would classically only represent the Pareto front of an MCO problem with one dimensional decision space (i.e. the decision threshold being the only decision variable). However, it is reasonable to apply MCO methods for other cases, for instance when 4For documentation, manuals and source code of CANARY see: https://software.sandia.gov/trac/canary MCO tuning of Event Detection Software 5 False Positive Rate 0 1 AUC>0.5 0 1 AUC=0.5 True Positive Rate 0 0 1 1 AUC=1 perfect good random False Positive Rate False Positive Rate Event Probabilities Event Threshold TP FN TN FP Distribution of True Events Distribution of Non-Events TP TN FN FP TP TN Fig. 1: Relationship between threshold and ROC curve in the classical case. The upper graphs show the distribution of the events for three different cases, the lower graphs show the ROC curves for the same cases. Thereby, TN is True Negative, TP is True Positive, FN is False Negative, FP is False Positive. Leftmost is the case of perfect classification, the rightmost is random guessing. different classifiers are to be compared, or the threshold is not independent of the classification process. This is the case in the problem described in this paper. Applying MCO for ROC analysis is not a new topic. Kupinski and Anastasio [22] considered performances of the solutions returned from a multi-criteria objective genetic optimization as series of optimal (sensitivity, specificity) pairs, which can be thought of as operating points on a ROC curve. ROC analysis has also been introduced to machine learning as described by Flach [10]. Recently, Wang et al. [31] consider the ROC convex hull (ROCCH). They use multi-criteria genetic programing to approximate the optimal ROCCH. A survey of MCO applications for ROC can be found in the work of Everson et al. [9]. 3 Problem Description The problem to be solved in this paper is the tuning of a software designed for anomaly detection in water quality management: CANARY. It was developed by the US Environmental Protection Agency EPA and Sandia National Laboratories to detect anomalies (or events) in water quality time series data. It implements several different algorithms for time series prediction, pattern matching, and outlier detection. The main concept is to employ a time series algorithm to predict the next time step, and afterwards to distinguish whether 6 Zaefferer et al. the real value deteriorates from the predicted value sufficiently to declare it an outlier or anomaly. CANARY can use additional methods to reduce false alarms. One method is to look at the frequency of outliers in a smaller time window, to make sure that isolated deviations due to noise do not raise an alarm. This was found to be not helpful, at least for the data sets used in this study, as the data is not that noisy on a smaller time frame. Another method to reduce false alarms would be to compare known non-event patterns with current data, to exclude events out of interest. which are not of interest. This is impossible with the data relevant for this paper as well, as pattern matching requires knowledge about the position of such non-events, which is unavailable for the data set used. We will tune the two relevant parameters window size and threshold value. The window size defines how many values are used for the prediction, while the threshold value defines how much deviation between measured and predicted value are sufficient to declare an outlier. Both parameters have previously been tuned in different ways. Firstly, they have been tuned by a step-by-step procedure [26] which unfortunately does not consider interactions between parameters. Secondly, another study [32] tuned them with model based optimization, considering interactions, but only used a single criteria approach, which basically combined the objectives False Alarm Rate and Hit Rate to a weighted sum. Usually, as described by Murray et al. [26], a classical ROC analysis would be performed. The AUC would be used as a single quality criterion. This approach is not perfectly viable in this case, as the threshold value is not independent of the prediction process. Therefore, it is a more reasonable approach to add the threshold to the list of tuned parameters and apply multi-criteria optimization. For this reason, we will mainly use MCO-terminology in the following (e.g., Pareto front instead of ROC curve). 4 Algorithm Description Four different tuning algorithms are in the focus of this study. Due to the similarity to the AUC, the hypervolume is applied as a criterion in all but one of these approaches. Two of them are based on R-code (SPOT package), two are MATLAB implementations (SMS-EGO and SExI-EGO). All four share the following basic workflow: 1. Evaluate an initial design of npoints on the target problem (CANARY) 2. Build models (here: Kriging) for each objective 3. Use models to determine the next design point to be evaluated 4. Evaluate design point and update non dominated set 5. Iterate 2-4 The four tuning algorithms differ in the type of the invoked infill criterion. Three algorithms use different multi-criteria EI. The fourth is a straightforward approach that, instead of aggregating the objective values from the models, tries to optimize these separately with common MCO methods. MCO tuning of Event Detection Software 7 4.1 MEI-SPOT This multi-criteria expected improvement approach is the only approach that does not use hypervolume as a criterion. The implementation is based on MATLAB code of Forrester et al. [12]. MEI-SPOT is based on the integration over the non-dominated area and an Euclidean distance to the next point on the front. While Forrester et al. use a dominating variant (e.g. improvement considers only points that dominate existing Pareto-optimal solutions), the implementation used here uses an augmenting variant (i.e. improvement is also reported when a point is added to the front, without dominating an existing Paretooptimal solution). The different formulations for this distinction are detailed by Keane [19]. This approach is time consuming due to the integration. It can also have issues with the scaling of different objectives, since it is based on the Euclidean distance. 4.2 SExI-EGO The S-Metric Expected Improvement [7] computes the expected increment in hypervolume for a point, given a non-dominated set. Its exact computation is described in [8]. It is differentiable, rewards high variances [8], and is continous over the whole search domain. A disadvantage is the high effort of its exact computation, in particular when more than two objectives are considered. 4.3 SMS-EGO SMS-EGO, as suggested by Ponweiser et al. [27], employs a hypervolume based infill criterion as well. Thereby, a potential solution is computed using the lower confidence bound ˆypot = ˆy−αˆs, where ˆyis the mean value predicted by the Kriging model, ˆsis the variance, αis a gain factor for the variance. This approach may also explore unvisited regions of the design space, but without requiring the tedious integration of the previous approaches. It thus scales better with increasing objective dimension. If the resulting ˆypot is -dominated or dominated, SMS-EGO will assign a penalty value. If it is non-dominated, the hypervolume contribution will be used. This approach avoids plateaus of the criterion, but integrates non differentiable parts. For more details see Ponweiser et al. [27] and Wagner et al. [30]. 4.4 MSPOT MSPOT is a multi-criteria approach based on the Sequential Parameter Optimization Toolbox SPOT (cf. Zaefferer et al. [33]). It does not employ any form of expected improvement, or other forms of using the variance for exploration. The surrogate models of the different objectives are exploited by using a multi-criteria optimization algorithm (for instance: SMS-EMOA or NSGA-II). This will yield a population of promising points. One or more points of these are chosen for evaluations on the real target function. This selection is based 8 Zaefferer et al. on non-dominated sorting and the individual hypervolume contribution. As the original approach [33] could lead to clustering of solutions in the objective space, the available points have to be considered when calculating the hypervolume contributions. For this purpose, the known points are reevaluated on the surrogate model. In contrast to the other approaches in the study, this one does not promote exploration as much, since the variance measure computed by the Kriging model will not be used. On the other hand, the approach is not limited to surrogate modeling methods that yield a variance for each candidate. Of course, the variance can easily be added to MSPOT, as well as be removed from SMS-EGO (α= 0) or the integration-based algorithms (ˆs= 0). The optimization process of MSPOT is not a completely new idea. Especially, two similar approaches suggested previously have to be mentioned. Firstly, Voutchkov and Keane [29] employed NSGA-II to generate promising solutions in a quite similar optimization loop. In contrast to MSPOT, they used Euclidean distance to ensure evenly spaced points on the front. Instead of considering distance to known points, they suggest a larger number points in each loop, which also ensures a wider spread on the final front. The second similar approach is presented by Jeong and Obayashi [15]. While they also optimize the objectives separately, they employ the single objective EI criterion for each objective, thus optimizing a vector of EI values. 4.5 SMS-EMOA In addition to the four approaches above, a simple SMS-EMOA will be considered (cf. Beume et al. [2]). The results from this optimizer are used as a baseline for the comparison. In general, surrogate optimization methods are expected to outperform a non-surrogate SMS-EMOA, particularly on small budgets. 5 Experimental Setup The following research questions are to be treated for the CANARY problem in this study. 1. Can multi-criteria methods produce a front of parameter settings that help an operator to choose parameters for the CANARY event detection software? 2. Which kind of tuner is recommendable? 3. What aspects of a tuner affect its performance? 4. Is the use of surrogate models advantageous? 5. Can previous findings about the tuners be confirmed? 6. How are Pareto optimal solutions spread in the design space? To answer these questions, several experiments were conducted. Their setup is described in the following. MCO tuning of Event Detection Software 9 5.1 Time Series Data Two different sets of raw data are used. The first set is used to train CANARY (i.e. to tune the parameters), the second is used for validation of the resulting settings on unseen data. Additionally, from each of those sets, 3 different instances are generated, where each contains simulated (i.e. superimposed) events to be detected by CANARY. The data sets considered are available within the CANARY software package. Training Data The data recorded over a first month at a specific measurement station is used as training data. Four different sensor values are used (pH-Value, Conductivity, Total Organic Carbon, Chlorine). The time interval between measurements is five minutes. This results in about 9 000 time steps for each of the four sensors. As the data-set contains no events known beforehand (which is a typical problem for any available real-world data), events have to be simulated and incorporated in the time series. Therefore, 3 data sets are created from the raw data, each containing superimposed square waves (with smoothed transition) of different event strengths: 0.5, 1, and 1.5. These strengths indicate the amplitude of the events, and are multiplied to the standard deviation of the original signal. Figure 2 presents raw data and data with events for two sensor value as an example. As can be seen from the left part of Fig. 2, the raw data (i.e. without events) is rather strongly affected by background changes. This particularly holds for the conductivity values (COND), which jump from a baseline value around 50 to a new baseline value at about 200 and back. In general, these background changes are irregularly distributed over time and always switch back and forth for each of the signals. Obviously, such changes make event detection extremely difficult. Validation Data The validation data is similar to the training data, as it is the second month of data from the same measurement station. As could be expected, it provides a very similar background behavior with some sudden jumps. These jumps, however, are more numerous than in the training data, which is expected to lead to higher false alarm rates on the validation data. 5.2 Optimization Problem Configuration As mentioned earlier, three different data sets are considered, each with a different event strength. Additionally, CANARY is tuned in 3 different configurations, where each configuration uses a different time series prediction algorithm. These are: Time Series Increment TSI, Linear Prediction Correction Filter LPCF and Multi-Variate Nearest Neighbor MVNN. For more details on these algorithms, which are implemented in CANARY, see the corresponding documentation [26] and the manual [14]. Therefore, 3 ×3 = 9 instances are to be optimized. The 16 Zaefferer et al. be considered that additional exploration is already inherent in the selection process as not one single optimum, but a set of points is demanded. To visualize the problem landscape, Fig. 6 shows contour plots of reference DACE-models for each objective. These models were built by combining the designs of all algorithms and selecting some representatives based on the distance to an optimized Latin hypercube design. Whereas, the models seem to have a rather unimodal shape, there are clusters of optimal solutions due to a slightly oscillating behavior in the plateau regions. This effect can be observed using the model predictions and the actual data. As a consequence, the approximation of the knee region with window sizes between 200 and 400 and a threshold between 1.0 and 1.5 should be easy, whereas the extreme ones might become a multimodal problem. MCO tuning of Event Detection Software 17 Fig. 6: Problem landscape of both objectives. Contours show DACE-model based on represantatives from all evaluations on this instance (Algorithm MVNN and event strength 1.5). Black dots show all real pareto optimal solutions found. Grey dots show pareto optimal solutions on the model, yielded with grid sampling. 18 Zaefferer et al. 7 Summary In this study, we tested different approaches based on surrogate optimization to tune an event detection software. Most of the analysis was focused on the results of training data, since the results on validation data mainly provided similar results as on the training data. The surrogate optimization approaches are mostly able to outperform a baseline SMS-EMOA. The MEI-SPOT approach proved to be the exception from this observation, which confirms earlier findings by Wagner et al. [30]. This approach of calculating the expected improvement for multiple criteria seems to be unfavorable. There was no decisive difference between the other tested approaches, regardless whether variance was used in the approach (SMS-EGO and SExI-EGO) or not (MSPOT and SMS-EGO with zero gain). Plots of the model structures seem to indicate an almost unimodal fitness landscape for both objectives. This indicates that the additional exploration by variance might not be needed here, since the fitness landscape is easy to approximate without additional exploration of the design space. This study showed that the problem of tuning CANARY can reasonably be solved by multi-criteria methods. The produced results yield reasonable FPR and TPR values, which are comparable to previous results achieved by singleobjective optimization. Here, however, the approximation of a Pareto front offers more flexibility for the operator in charge. 8 Outlook The following topics will be subject of future research. –Since the additional exploration by variance does not decrease performance significantly, it might be interesting to test the lower confidence bound in MSPOT for future experiments based on other problems. –It has to be noted that only points on the convex hull of the Pareto front can be considered to be optimal in some sense. This is due to the fact, that any point below that hull might be considered to be improvable [11]. Future work should investigate if concavities in the ROC curve can be repaired for the application described here. –The concentration on certain regions of a Pareto front might be a topic for future research as well. An operator might be more interested in the knee region of the Pareto front, and less on extreme values, which might cause intolerable numbers of false alarms. Focusing on a subset of the Pareto front might save further evaluations, thus reducing the required budget. –As can be seen in Fig. 7 the Kriging based approaches generate similar results, with one exception: The Maximum Likelihood Estimation Gaussian Processes (MLEGP) variant seems to outperformed by the other two variants. This demands further investigation. Therefore, we plan a more detailed comparison of different Kriging based approaches. MCO tuning of Event Detection Software 19 emax: 0.5 emax: 1 emax: 1.5 ● ● ● ● ● ● ● ● ●● ●●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● 0.75 0.80 0.85 0.75 0.80 0.85 0.90 0.80 0.84 0.88 0.92 atype: inc atype: lpcf atype: mvnn DACE Forrester MLEGP MLP QRNN eSVM kSVM MARS RF LM DACE Forrester MLEGP MLP QRNN eSVM kSVM MARS RF LM DACE Forrester MLEGP MLP QRNN eSVM kSVM MARS RF LM Tuner Hypervolume Fig. 7: Comparison of different models using the MSPOT approach. DACE is the Kriging model used in previous comparisons. Forrester is a Kriging model implemented in SPOT based on Matlab Code by Forrester et al. [12]. MLEGP is the Kriging implementation Maximum Likelihood Estimates of Gaussian Processes in the Rpackage mlegp [6]. MLP is a Multi-layer perceptron neural network [23] from the Rpackage monmlp. QRNN is Quantile Regression Neural Network from the qrnn package [28, 4]. eSVM is a Support Vector Machine implementation in the package e1071 using LIBSVM [5]. kSVM is a SVM in the kernlab package [18]. MARS is Multivariate Adaptive Regression Splines [13] provided by the earth Rpackage. RF is a Random Forest implementation from the Rpackage randomForest which is based on Breiman and Cutler’s original Fortran code for classification and regression [3]. LM uses a Linear Model fitted with the rsm package [24]. 20 Zaefferer et al. Acknowledgments This work has been kindly supported by the Federal Ministry of Education and Research (BMBF) under the grants MCIOP (FKZ 17N0311) and CIMO (FKZ 17002X11). In addition, the paper is based on investigations of project D5 “Synthesis and multi-objective model-based optimization of process chains for manufacturing parts with functionally graded properties” as part of the collaborative research center SFB/TR TRR 30, kindly supported by the Deutsche Forschungsgemeinschaft (DFG). References 1. T. Bartz-Beielstein, K. E. Parsopoulos, and M. N. Vrahatis. Design and analysis of optimization algorithms using computational statistics. Applied Numerical Analysis and Computational Mathematics (ANACM), 1(2):413–433, 2004. 2. N. Beume, B. Naujoks, and M. Emmerich. SMS-EMOA: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research, 181(3):1653–1669, 2007. 3. L. Breiman. Random forests. Machine Learning, 45(1):5 –32, 2001. 4. A. J. Cannon. Quantile regression neural networks: implementation in r and application to precipitation downscaling. Computers & Geosciences, 37:1277–1284. doi:10.1016/j.cageo.2010.07.005, 2011. 5. C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm. 6. G. M. Dancik and K. S. Dorman. mlegp: statistical analysis for computer models of biological systems using r. Bioinformatics, 24(17):1966–1967, 2008. 7. M. Emmerich. Singleand Multi-objective Evolutionary Design Optimization: Assisted by Gaussian Random Field Metamodels. PhD thesis, Universit¨at Dortmund, Germany, 2005. 8. M. Emmerich, A. Deutz, and J. Klinkenberg. Hypervolume-based expected improvement: Monotonicity properties and exact computation. In Evolutionary Computation (CEC), 2011 IEEE Congress on, pages 2147–2154. IEEE, 2011. 9. R. Everson and J. Fieldsend. Multi-objective optimisation for receiver operating characteristic analysis. In Y. Jin, editor, Multi-Objective Machine Learning, volume 16 of Studies in Computational Intelligence, pages 533–556. Springer Berlin / Heidelberg, 2006. 10. P. A. Flach. ROC Analysis, pages 869–875. Springer, December 2010. 11. P. A. Flach and S. Wu. Repairing concavities in roc curves. In Proceedings of the 19th international joint conference on Artificial intelligence, IJCAI’05, pages 702–707, San Francisco, CA, USA, 2005. Morgan Kaufmann Publishers Inc. 12. A. Forrester, A. Sobester, and A. Keane. Engineering Design via Surrogate Modelling. Wiley, 2008. 13. J. H. Friedman. Multivariate adaptive regression splines. Ann. Stat., 19(1):1–141, 1991. 14. D. B. Hart, K. A. Klise, E. D. Vugrin, S. A. McKenna, and M. P. Wilson. Canary user’s manual and software upgrades. Technical Report EPA/600/R-08/040A, U.S. Environmental Protection Agency, Washington, DC, 2009. 15. S. Jeong and S. Obayashi. Efficient global optimization (EGO) for multi-objective problem and data mining. In D. Corne et al., editors, IEEE Congress on Evolutionary Computation, pages 2138–2145. IEEE, 2005. MCO tuning of Event Detection Software 21 16. Y. Jin. A comprehensive survey of fitness approximation in evolutionary computation. Soft Computing, 9(1):3–12, 2005. 17. D. Jones, M. Schonlau, and W. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13:455–492, 1998. 18. A. Karatzoglou, A. Smola, K. Hornik, and A. Zeileis. kernlab – an S4 package for kernel methods in R. Journal of Statistical Software, 11(9):1–20, 2004. 19. A. Keane. Statistical improvement criteria for use in multiobjective design optimisation. AIAA Journal, 44(4):879–891, 2006. 20. J. Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation, 10(1):50–66, January 2006. 21. J. D. Knowles and H. Nakayama. Meta-modeling in multiobjective optimization. In Multiobjective Optimization, pages 245–284. Springer, 2008. 22. M. A. Kupinski and M. A. Anastasio. Multiobjective genetic optimization of diagnostic classifiers with implications for generating receiver operating characteristic curves. IEEE Transactions on Medical Imaging, 18:675–685, 1999. 23. B. Lang. Monotonic multi-layer perceptron networks as universal approximators. In W. Duch, J. Kacprzyk, E. Oja, and S. Zadrozny, editors, Artificial Neural Networks: Formal Models and Their Applications ICANN 2005, volume 3697 of Lecture Notes in Computer Science, pages 31–37. Springer Berlin Heidelberg, 2005. 24. R. V. Lenth. Response-surface methods in R using rsm (updated to version 1.40). Technical report, The University of Iowa, 2010. 25. S. Lophaven, H. Nielsen, and J. Søndergaard. DACE—A Matlab Kriging Toolbox. Technical Report IMM-REP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark, Copenhagen, Denmark, 2002. 26. R. Murray, T. Haxton, S. A. McKenna, D. B. Hart, K. Klise, M. Koch, E. D. Vugrin, S. Martin, M. Wilson, V. Cruz, and L. Cutler. Water quality event detection systems for drinking water contamination warning systems—development, testing, and application of CANARY. Technical Report EPA/600/R-10/036, National Homeland Security Research Center, May 2010. 27. W. Ponweiser, T. Wagner, D. Biermann, and M. Vincze. Multiobjective optimization on a limited budget of evaluations using model-assisted -metric selection. In PPSN, pages 784–794, 2008. 28. J. W. Taylor. A quantile regression neural network approach to estimating the conditional density of multiperiod returns. Journal of Forecasting, 19(4):299–311, 2000. 29. I. Voutchkov and A. Keane. Multiobjective optimization using surrogates. In Adaptive Computing in Design and Manufacture ACDM, pages 167–175, 2006. 30. T. Wagner, M. Emmerich, A. Deutz, and W. Ponweiser. On expected-improvement criteria for model-based multi-objective optimization. Parallel Problem Solving from Nature–PPSN XI, pages 718–727, 2010. 31. P. Wang, K. Tang, E. Tsang, and X. Yao. Multiobjective genetic programming for maximizing ROC performance. Neurocomputing, 2012. 32. M. Zaefferer. Optimization and empirical analysis of an event detection software for water quality monitoring. Master’s thesis, Cologne University of Applied Sciences, May 2012. 33. M. Zaefferer, T. Bartz-Beielstein, M. Friese, B. Naujoks, and O. Flasch. Multicriteria optimization for hard problems under limited budgets. In T. Soule et al., editors, GECCO Companion ’12: Proceedings of the fourteenth international conference on Genetic and evolutionary computation conference companion, pages 1451–1452, Philadelphia, Pennsylvania, USA, July 2012. ACM. Kontakt/Impressum Diese Ver¨offentlichungen erscheinen im Rahmen der Schriftenreihe ”CIplus”. Alle Ver¨offentlichungen dieser Reihe k¨onnen unter www.ciplus-research.de oder unter http://opus.bsz-bw.de/fhk/index.php?la=de abgerufen werden. K¨oln, Januar 2012 Herausgeber / Editorship Prof. Dr. Thomas Bartz-Beielstein, Prof. Dr. Wolfgang Konen, Prof. Dr. Horst Stenzel, Dr. Boris Naujoks Institute of Computer Science, Faculty of Computer Science and Engineering Science, Cologne University of Applied Sciences, Steinm¨ullerallee 1, 51643 Gummersbach url: www.ciplus-research.de Schriftleitung und Ansprechpartner/ Contact editors office Prof. Dr. Thomas Bartz-Beielstein, Institute of Computer Science, Faculty of Computer Science and Engineering Science, Cologne University of Applied Sciences, Steinm¨ullerallee 1, 51643 Gummersbach phone: +49 2261 8196 6391 url: http://www.gm.fh-koeln.de/~bartz/ eMail: [email protected] ISSN (online) 2194-2870