Metodología ensemble para clasificación ordinal y aplicación en el control de calidad del aceite de oliva

Metodología Ensemble para Clasificación Ordinal y Aplicación en el Control de Calidad del Aceite de Oliva TESIS DOCTORAL Belén Vega Márquez Departamento de Lenguajes y Sistemas Informáticos Escuela Técnica Superior de Ingeniería Informática Universidad de Sevilla Enero 2022 Metodología Ensemble para Clasificación Ordinal y Aplicación en el Control de Calidad del Aceite de Oliva Memoria que presenta para optar al título de Doctora en Informática Belén Vega Márquez Dirigida por las Doctoras Dra. Cristina Rubio Escudero Dra. Isabel de los Ángeles Nepomuceno Chamorro Departamento de Lenguajes y Sistemas Informáticos Escuela Técnica Superior de Ingeniería Informática Universidad de Sevilla Enero 2022 Copyright ©Belén Vega Márquez A mis seres queridos, porque sin ellos nunca hubiera llegado hasta aquí No, no creo en la suerte, pero sí en asignar valor a las cosas. John Nash (Una mente maravillosa) Tesis doctoral subvencionada por el Ministerio de Economía y Competitividad del Gobierno de España con los proyectos TIN2014-55894-C2-1-R y TIN2017-88209-C2-2-R, y por la Consejería de Economía, Conocimien- to, Empresas y Universidad de la Junta de Andalucía con el proyecto US- 1263341. ix xvi Abstract The need to develop this algorithm arises from analyzing the quality of olive oil. The quality of olive oil is determined by physicochemical factors that are translated into labels that have an order of relationship between them depending on the degree of quality of the samples. The results obtained were promising, demonstrating that this methodology is a good alternative for this particular problem. Given the importance of correct classification of such an essential product in the Spanish economy as olive oil, and taking into account the richness and variety of the data we had, we also decided to explore other artificial intelligence techniques, such as artificial neural networks, to address the same problem from different points of view, to obtain the best possible results. Finally, given the sensitivity and privacy of the data we were working with, we studied artificial intelligence techniques for the generation of synthetic data to share data with other research groups without compromising the original data. The technique used was generative adversarial neural networks (GANs), which successfully generated synthetic data. Keywords machine learning, ensembles, ordinal classification, deep learning Índice Agradecimientos xi Resumen xiii Abstract xv I Memoria 1 1. Introducción 3 1.1. Motivación de la investigación . . . . . . . . . . . . . . . . . . 3 1.2. Objetivos ............................. 4 1.3. Contribuciones .......................... 5 1.4. Estructura de la memoria de la tesis doctoral . . . . . . . . . 8 2. Contexto de la investigación 9 2.1. Técnicas de inteligencia artificial . . . . . . . . . . . . . . . . 9 2.2. Clasificación ordinal . . . . . . . . . . . . . . . . . . . . . . . 11 2.3. Metodologías ensemble . . . . . . . . . . . . . . . . . . . . . . 13 2.4. Técnicas más avanzadas de deep learning . . . . . . . . . . . 14 2.4.1. Aprendizaje supervisado con redes neuronales (MLP, CNN) ........................... 14 2.4.2. Generación de datos (GANs) . . . . . . . . . . . . . . 16 2.5. Contexto en la clasificación del aceite de oliva . . . . . . . . . 17 3. Discusión conjunta de los resultados 21 3.1. Clasificación ordinal basada en ensembles . . . . . . . . . . . 21 3.2. Feed-forward neural networks . . . . . . . . . . . . . . . . . . 23 3.3. Convolutional neural networks . . . . . . . . . . . . . . . . . . 23 3.4. Generación de datos sintéticos . . . . . . . . . . . . . . . . . . 24 xvii xviii Índice II Publicaciones 29 4. Resumen de trabajos de investigación seleccionados 31 5. Artículos de revista 33 5.1. OCEAn: Ordinal Classification with an Ensemble Approach . 33 5.2. Deep Learning Techniques to Improve the Performance of Oli- ve Oil Classification . . . . . . . . . . . . . . . . . . . . . . . 59 5.3. Generation of Synthetic Data with Conditional Generative Adversarial Networks . . . . . . . . . . . . . . . . . . . . . . . 73 6. Congresos internacionales 87 6.1. Convolutional Neural Networks for Olive Oil Classification . . 87 6.2. Creation of Synthetic Data with Conditional Generative AdversarialNetworks ........................ 99 7. Otras publicaciones 111 7.1. Use of Deep Learning Architectures for Day-Ahead Electricity Price Forecasting Over Different Time Periods in the Spanish Electricity Market . . . . . . . . . . . . . . . . . . . . . . . 112 7.2. Using Prior Knowledge in the Inference of Gene Association Networks .............................133 7.3. An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting ............................147 III Conclusiones y Trabajos Futuros 161 8. Conclusiones y trabajos futuros 163 8.1. Conclusiones ...........................163 8.2. Trabajo futuro . . . . . . . . . . . . . . . . . . . . . . . . . . 165 IV Apéndices 167 A. Curriculum 169 A.1. Revistas Indexadas JCR . . . . . . . . . . . . . . . . . . . . . 169 A.2. Conferencias Internacionales . . . . . . . . . . . . . . . . . . . 170 A.3. Proyectos I+D+i . . . . . . . . . . . . . . . . . . . . . . . . . 171 A.4. Contratos con empresas (Arts. 68/83 LOU) . . . . . . . . . . 172 A.5. Experiencia Docente . . . . . . . . . . . . . . . . . . . . . . . 172 A.6.Estancias .............................173 Índice xix Bibliography 175 Índice de figuras 2.1. Disciplinas englobadas dentro de la inteligencia artificial [5]. . 10 2.2. Clasificación de los algoritmos de aprendizaje dentro del aprendizajeautomático. ........................ 10 2.3. Estructura de una red neuronal multicapa. . . . . . . . . . . . 15 2.4. Estructura de una red neuronal convolucional. . . . . . . . . . 16 2.5. Estructura de una red red neuronal generativa adversarial (GAN)[34]............................. 17 2.6. Esquema genérico de la técnica de la espectrometría de movilidadiónica[39]. ......................... 18 2.7. Visualización obtenida tras aplicar la técnica GC-IMS [18]. En el eje X se observa el tiempo de deriva de los analitos en milisegundos y en el eje Y el tiempo de retención. . . . . . . . 19 3.1. Pasos seguidos en la metodología OCEAn [33]. . . . . . . . . 22 3.2. Índices de correlación de Pearson entre los datasets generados y los reales. Para cada dataset se realiza la correlación 1 a 1 entre las columnas de los datasets. . . . . . . . . . . . . . . . 25 3.3. Índices de correlación entre los datasets generados y los reales. Para cada dataset se realiza la correlación 1 a 1 entre las columnas de los datasets. Los resultados obtenidos muestran que la correlación es prácticamente cero. . . . . . . . . . . . . 26 xxi Índice de Tablas 1.1. Resumen de artículos de investigación presentados indexados enrevistasJCR. ......................... 7 1.2. Resumen de artículos de investigación presentados en congresos internacionales. . . . . . . . . . . . . . . . . . . . . . . . . 7 1.3. Correspondencia entre objetivos y publicaciones . . . . . . . . 7 3.1. Comparativa en términos de Accuracy entre la metodología propuesta y los resultados previos proporcionados por el Departamento de Química Analítica de Córdoba. Cada una de las muestras podía ser de tres categorías: Lampante (LOO), Virgen Extra (EVOO) o Virgen (VOO). Cada una de las filas de la tabla representa las diferentes casuísticas que tuvimos en cuenta, teniendo para el primer caso un problema de clasificación de tres clases, y para los demás de dos clases. . . . . . 23 3.2. Comparativa en términos de accuracy entre los resultados obtenidos mediante redes feed-forward y algoritmos de aprendizaje supervisado muy usados en la literatura. En negrita se destaca, para cada uno de los tres experimentos realizados, el algoritmo con el que se obtuvo el mejor resultado. . . . . . . . 23 3.3. Comparativa, en términos de accuracy, entre todas las metodologías usadas hasta el momento . . . . . . . . . . . . . . . . 24 3.4. Resultados obtenidos tras aplicar un algoritmo XGBoost a uno de los conjuntos de datos presentes en la publicación. Para cada conjunto de datos se calcularon varias métricas. En esta tabla se muestran accuracy yF1-Score. Se puede observar que los resultados obtenidos tanto para el conjunto de datos original, como para el conjunto de datos generado es muy parecido, además, en la última fila se pueden observar los resultados obtenidos haciendo una unión de ambos conjuntos de datos. ............................... 26 3.5. Resultados de clasificación obtenidos tras aplicar un algoritmo XGBoost a los datos originales, a los generados y a la unión deambos.............................. 27 xxiii xxiv Índice de tablas 5.1. Datos del artículo: OCEAn: Ordinal Classification with an EnsembleApproach ....................... 33 5.2. Datos del artículo: Deep Learning Techniques to Improve the Performance of Olive Oil Classification . . . . . . . . . . . . . 59 5.3. Datos del artículo: Generation of Synthetic Data with Conditional Generative Adversarial Networks . . . . . . . . . . . . . 73 6.1. Datos del artículo: Convolutional Neural Networks for Olive OilClassification ......................... 87 6.2. Datos del artículo: Creation of Synthetic Data with Conditional Generative Adversarial Networks . . . . . . . . . . . . . . 99 7.1. Datos del artículo: Use of deep learning architectures for dayahead electricity price forecasting over different time periods in the Spanish electricity market . . . . . . . . . . . . . . . . 112 7.2. Datos del artículo: Using Prior Knowledge in the Inference of Gene Association Networks . . . . . . . . . . . . . . . . . . . 133 7.3. Datos del artículo: An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting . . . . . . . . . . . . . . . . . . . 147 Parte I Memoria 8Capítulo 1. Introducción 1.4. Estructura de la memoria de la tesis doctoral La presente memoria de tesis doctoral por compendio de artículos está dividida en 3 partes. Para facilitar el seguimiento y lectura de la misma, se describe en esta sección su estructura: En la Parte I (Memoria) se detalla cuál ha sido el trabajo de esta tesis doctoral, dividiéndolo en tres capítulos. El primero detalla la introducción, dicha introducción expresa cuál ha sido la motivación que ha guiado esta tesis doctoral por compendio de artículos, así como los objetivos planteados. También se incluye un resumen de las publicaciones que han contribuido a lograr el cumplimiento de los objetivos propuestos. El segundo capítulo detalla el marco teórico en el que se ha encuadrado esta tesis doctoral, explicando los conceptos teóricos fundamentales de cada una de las técnicas utilizadas. Por último, en el tercero se realiza una discusión de los resultados obtenidos aplicando las técnicas descritas previamente. En la Parte II (Publicaciones) se presentan los artículos de investigación derivados de esta tesis doctoral siendo organizados por tipo de publicación. En la Parte III (Conclusiones y Trabajos Futuros) se exponen las conclusiones adquiridas tras la realización de la tesis y los trabajos futuros planteados. Capítulo 2 Contexto de la investigación Si supiese qué es lo que estoy haciendo, no lo llamaría investigación, ¿verdad?. Albert Einstein En esta sección se presenta el Marco Teórico sobre los fundamentos en los que se basan los trabajos que componen esta tesis doctoral. De manera más concreta, se hace una introducción sobre las técnicas de inteligencia artificial existentes, también se define qué es la clasificación ordinal y las metodologías ensemble, grueso de esta tesis doctoral. Posteriormente, se definen otras técnicas más avanzadas de aprendizaje profundo también englobadas dentro de la inteligencia artificial y por último se explica el contexto en el que se encuadra el problema a resolver, que es la clasificación de la calidad del aceite de oliva. 2.1. Técnicas de inteligencia artificial No hay una definición exacta para el término inteligencia artificial (IA) usándose este término en muchos ámbitos diferentes y con sentidos diversos [38]. Una definición que se repite en muchos medios es la de la habilidad de una máquina de presentar las mismas capacidades que los seres humanos, como el razonamiento, el aprendizaje, la creatividad y la capacidad de planear [27]. La IA es una disciplina que encuadra una gran variedad de técnicas y campos de aplicación, de ahí que pueda ser usada en la mayoría de ámbitos existentes. Estas técnicas quedan reflejadas en la Figura 2.1 y se detallarán a continuación. 9 10 Capítulo 2. Contexto de la investigación Figura 2.1: Disciplinas englobadas dentro de la inteligencia artificial [5]. Machine learning o aprendizaje automático es una disciplina cuyo objetivo principal es desarrollar algoritmos basados en instrumentos estadísticos que son usados para encontrar patrones dentro de una gran cantidad de datos que luego serán usados para realizar predicciones [11]. Estos algoritmos se llaman auto-adaptativos, ya que permiten a las máquinas aprender de forma independiente, sin personas interviniendo en su aprendizaje. Dentro de esta disciplina pueden diferenciarse dos tipos de aprendizaje, el aprendizaje supervisado y el no supervisado [14] tal como se puede observar en la figura 2.2: Figura 2.2: Clasificación de los algoritmos de aprendizaje dentro del aprendizaje automático. 2.2. Clasificación ordinal 11 Aprendizaje Supervisado: En este tipo de aprendizaje los datos están previamente etiquetados, es decir, el sistema de IA ya sabe la respuesta correcta. Esto permitirá al algoritmo poder aprender una función capaz de predecir el atributo objetivo o clase para un conjunto de datos nuevo. Dependiendo de si la variable objetivo es categórica o numérica podemos dividir los tipos de algoritmos entre algoritmos para clasificación y algoritmos para regresión, respectivamente. Aprendizaje No Supervisado: La diferencia principal con respecto al tipo de aprendizaje comentado anteriormente es que los datos no están previamente etiquetados. El objetivo de aprendizaje de este tipo de algoritmos es el de encontrar patrones dentro de los datos. Estos patrones son capaces de encontrar grupos dentro de los datos que muestran comportamientos similares o características similares. Esta tarea suele denominarse análisis de cluster, ya que cada grupo se denomina cluster. Otro tipo de tarea dentro de este aprendizaje es el de reducir la dimensionalidad de los datos, buscando atributos muy similares entre sí que no aporten información relevante. Por último, pero no menos importante hay que destacar el aprendizaje profundo o deep learning. El aprendizaje profundo está dentro del subconjunto de metodologías de aprendizaje automático y está compuesto por redes neuronales que tratan de imitar el comportamiento de las neuronas de un cerebro [1]. Este tipo de técnica puede procesar un rango más amplio de conjuntos de datos ya que requiere de menos intervención humana en la parte de preprocesamiento, y normalmente obtiene resultados mejores que con metodologías de aprendizaje profundo tradicionales. El término “profundo” hace referencia al gran número de capas de las redes neuronales. Todos estos conceptos mencionados serán explicados más en profundidad en la sección 2.4. 2.2. Clasificación ordinal Una de las subcategorías del aprendizaje supervisado es la clasificación, tal como se ha visto en la sección 2.1. La gran mayoría de algoritmos de clasificación predicen una variable categórica sin considerar una posible relación de orden entre las posibles etiquetas. Esto supone a veces un problema, ya que se obvian relaciones que podrían ayudar a dar mejores resultados en las predicciones. Las técnicas que estudia este tipo de problema se clasifican como clasificación ordinal. Estas técnicas tratan de resolver el problema considerando un conjunto de etiquetas ordenadas en las que cada instancia del conjunto de datos es un par clave-valor (x, y)∈X×Y, donde xes un vector de entrada que pertenece a un espacio X, e yes una etiqueta dentro de un espacio finito de etiquetas con orden Y={C1, C2, ..., Ck}[22]. 12 Capítulo 2. Contexto de la investigación En la literatura podemos encontrar una gran cantidad de algoritmos, y aunque en algunos casos no es trivial hacer una categorización de ellos, los algoritmos podrían enmarcarse dentro de la siguiente taxonomía [16]: Aproximaciones bayesianas: este tipo de algoritmos tratan de resolver el problema de clasificación ordinal desde un punto de vista de subdivisión de problemas. Dentro de este conjunto de algoritmos se observan tres categorías, que son la regresión, clasificación nominal y clasificación sensible al coste. En las aproximaciones por regresión la variable objetivo es transformada en una variable numérica que luego será predicha mediante un algoritmo de regresión. El problema en este tipo de aproximación es que la naturaleza ordinal se pierde. Un ejemplo de la aproximación por regresión se observa en el trabajo propuesto en[15], donde se implementó un modelo de regresión para descubrir la relación entre los tratamientos y la fertilidad en un total de 1218 mujeres. En el caso de la clasificación nominal, el orden entre las clases tampoco es tenido en cuenta, pero la variable objetivo no es transformada de ninguna forma, sino que se trata simplemente con una variable categórica a predecir. Un ejemplo de este tipo de aproximación se observa en [9], donde se usó un árbol de regresión para determinar la calidad en los servicios de transporte público de Granada en el año 2007. Finalmente, las técnicas de clasificación sensibles al coste sí tienen en cuenta la relación entre las etiquetas, estableciendo una matriz de costes entre las etiquetas reales y las predichas. Un ejemplo de esta categoría puede observarse en [20]. En este trabajo se propuso un clasificador bayesiano para predecir el incremento del índice-hde las revistas científicas. Aproximaciones basadas en descomposición: dentro de este grupo de algoritmos se encuentran aquellos que tratan el problema de clasificación ordinal desde el punto de vista de “divide y vencerás” ya que su principal objetivo es dividir la etiqueta ordinal en varias etiquetas binarias. Los modelos encuadrados dentro de esta categoría pueden dividirse en 2 subcategorías: modelos múltiples y modelos individuales de varias salidas. En el primero de los casos se entrena un modelo por cada uno de los subproblemas, mientras que para el segundo solamente se entrena un modelo para todos los subproblemas que surgen de dividir el problema principal. En [12] se desarrolló un modelo para clasificación ordinal que consistía en transformar un problema con k clases en k-1 problemas de clasificación binarios más fáciles de resolver. Para realizar la predicción final se usaba una combinación de las probabilidades obtenidas de los k-1 problemas, dando como resultado una predicción final. Como ejemplo de la segunda categoría se encuentra el trabajo propuesto en [25] donde se utilizó una red neuronal convolu- 2.3. Metodologías ensemble 13 cional profunda para predecir la edad de personas a partir de fotos de sus rostros; lo que se hizo para resolver este problema fue transformar un problema de regresión en subproblemas de clasificación binaria. Modelos basados en umbrales: este tipo de algoritmos son los que mejor rendimiento han demostrado para tratar el problema en cuestión, y este gran rendimiento viene dado por la idea que subyace bajo ellos. Esta idea consiste en tomar como premisa que hay una variable continua que puede explicar el comportamiento de la variable ordinal, por lo que el principal objetivo es encontrar una función que permita mapear el valor continuo predicho en etiquetas. Este grupo puede ser subdividido en 7 subgrupos: modelos de enlace acumulativo, máquinas de soporte vectorial, aprendizaje discriminativo, perceptrones, clasificación binaria aumentada, ensembles y procesos gaussianos. Como ejemplo de trabajos encuadrados dentro de estas 7 categorías puede destacarse el presentado en [40], donde se desarrolló un método basado en ensembles para la clasificación ordinal en el ámbito del transporte. Otro ejemplo es el algoritmo SVOREX [7], dicho algoritmo basado en la idea de las máquinas de soporte vectorial (SVM) fue usado para llevar a cabo una aproximación de clasificación ordinal que abordaba el comportamiento de remisión de los inmigrantes en función de sus características individuales. Por último, cabe destacar desde el punto de vista de los procesos gaussianos el trabajo propuesto en [6], donde un algoritmo de selección genética fue desarrollado para descubrir patrones de expresión genética asociados a fenotipos clínicos con cierta relación ordinal. 2.3. Metodologías ensemble Dada la gran dificultad que presentan algunos conjuntos de datos, hay ocasiones en las que un algoritmo de clasificación no es capaz de predecir la clase objetivo, por ello surge la idea del aprendizaje mediante ensembles. La idea principal del aprendizaje mediante la teoría de ensembles es la de combinar un conjunto de clasificadores base para realizar una mejor predicción de la que obtendría un clasificador individual [26]. Según [28], todo algoritmo ensemble consta de 4 bloques que determinarán el tipo al que pertenece, estos bloques se enumerarán a continuación: Conjunto de entrenamiento: conjunto de datos etiquetado usado para entrenar el ensemble. Inductor: algoritmo de inducción que obtiene un conjunto de entrenamiento y forma un clasificador que representa la relación generalizada entre los atributos de entrada y la variable objetivo. 14 Capítulo 2. Contexto de la investigación Generador: este componente es el responsable de generar los diferentes clasificadores. Combinador: responsable de combinar las predicciones de los clasificadores individuales. Uno de los aspectos más importantes a la hora de realizar un ensemble es precisamente el último bloque, que es la forma de combinar los algoritmos. Hay dos formas principalmente de combinar los diferentes algoritmos base, una es mediante votación y otra es mediante meta-aprendizaje. En el primero de los casos, a cada uno de los clasificadores base se le asigna un peso que determinará la forma de combinación final. Por ejemplo, en [42] se desarrolló un sistema de reconocimiento facial mediante un ensemble que utilizaba un método de votación simple. Por otro lado, el meta-aprendizaje consiste en aprender de los clasificadores producidos por los inductores y de las predicciones de estos clasificadores, un método muy conocido dentro de esta categoría es el stacking de modelos que consiste en utilizar la salida de alguno de los algoritmos como entrada de otros algoritmos, como por ejemplo ocurre en [4], donde se aplica para la predicción de interacción entre proteínas con unos resultados muy prometedores. 2.4. Técnicas más avanzadas de deep learning El auge de las tecnologías de la información y la gran capacidad hardware de la que disponemos actualmente hizo posible el uso de tarjetas gráficas dedicadas para hacer cálculo computacional con redes neuronales que antes no era posible. Un modelo deep learning consiste en un modelo basados en redes neuronales artificiales (ANN) con muchas capas entrenado de forma estratificada [24]. Dependiendo de la estructura de capas de la que disponga el modelo se hablará de un modelo o de otro. Según [29] todas las técnicas de aprendizaje profundo pueden agruparse en 3 categorías según la tarea de aprendizaje que acometan: aprendizaje supervisado, no supervisado y semi-supervisado. En esta tesis nos centraremos en las dos primeras categorías, ya que son las que se usarán. 2.4.1. Aprendizaje supervisado con redes neuronales (MLP, CNN) Los principales tipos de redes usadas en la literatura para realizar aprendizaje supervisado son las redes multicapa o perceptrón multicapa (MLP) que es un tipo de red neuronal artificial feed-forward. Este tipo de redes consiste básicamente en una capa de entrada que recibe los datos, una capa de salida que realiza la predicción y una o más capas ocultas que realizan las tareas de computación tal como se muestra en la Figura 2.3. Estas tareas 2.4. Técnicas más avanzadas de deep learning 15 de computación consiste en encontrar una asignación de pesos óptima para cada neurona que haga que el error de predicción sea el mínimo, este algoritmo es denominado algoritmo de retropropagación [17]. Un ejemplo de esta metodología aplicada a un ejemplo real se puede observar en [30] donde se usó una red neuronal multicapa para la predicción de arritmias cardiacas. Figura 2.3: Estructura de una red neuronal multicapa. Las redes neuronales anteriormente expuestas trabajan muy bien con datos en una dimensión, sin embargo, dado el gran número de problemas en los que intervienen imágenes, como son la clasificación de imágenes médicas o la segmentación de imágenes surgieron las redes neuronales convolucionales (CNN), más apropiadas para este tipo de problemas. Este tipo de redes se consideran como una variante de las MLP con una estructura especial [41]. Esta estructura se compone de tres tipos de capas, que son las capas de pooling o agrupación, capas convolucionales y capas densas (Figura 2.4). 16 Capítulo 2. Contexto de la investigación Figura 2.4: Estructura de una red neuronal convolucional. 2.4.2. Generación de datos (GANs) Las grandes leyes de protección de datos de las que disponemos en la sociedad ha hecho que la transmisión de información sensible entre entidades se haya convertido en una tarea imposible. Las redes neuronales generativas adversariales (GANs) fueron introducidas en 2014 en [19]. Este tipo de redes de modelado generativo son capaces de descubrir patrones de forma automática de los datos de entrada, para poder producir nuevos ejemplos a partir del conjunto de datos original. Estas redes se componen de dos estructuras fundamentales definidas a continuación (Véase Figura 2.5): Generador (G): red encargada de generar datos nuevos a partir de datos denominados “ruido”. Discriminador (D): esta red está encargada de predecir la probabilidad de que una muestra se extraiga de los datos reales y no de los datos proporcionados por el generador. El término “adversarial” viene dado porque el entrenamiento consiste en una competición entre el generador y el discriminador, de forma que mientras el generador intenta engañar al discriminador creando datos más realistas, el discriminador intenta distinguir entre los datos reales y los generados. 2.5. Contexto en la clasificación del aceite de oliva 17 Figura 2.5: Estructura de una red red neuronal generativa adversarial (GAN) [34]. Varios son los ejemplos que han utilizado esta nueva perspectiva para la generación de datos. En [2] se realizó una generación de datos sintéticos a partir de datos médicos de pacientes que se sometieron a una prueba para comprobar su nivel de presión arterial, estos datos generados permitieron que los datos pudiesen ser compartidos sin poner de manifesito la identidad de los participantes. Este tipo de redes funcionan incluso para la generación de imágenes con más resolución tal como se puede comprobar en [3, 23]. 2.5. Contexto en la clasificación del aceite de oliva El aceite de oliva es el oro líquido de la dieta mediterránea, no obstante es importante reconocer que más allá de las aceitunas empleadas para su elaboración, hay que tener en cuenta que dependiendo de la composición del producto y del proceso de obtención del mismo se distinguen ciertas variedades de aceite de oliva, no todas ellas aptas para el consumo humano. De forma objetiva, los diferentes tipos de aceite de oliva quedan reflejados en el Reglamento (CE) 640/2008 de la Comisión Europea y consiste en un análisis físico-químico junto con una evaluación sensorial realizada por un panel de catas. Estos tipos se basan en la calidad de la sustancia, diferenciándose así tres categorías principales de mayor a menor calidad: virgen extra (EVOO), virgen (VOO) y lampante (LOO) [13]. El método oficial para analizar la calidad de un aceite de oliva en concreto se basa en métodos físico químicos y sensoriales, siendo estos últimos los más comunes, denominados paneles de catas [8]. Tres son los atributos principales que se tienen en cuenta a la hora de definir la calidad del aceite de oliva, el color, el sabor y el aroma. Estos atributos son características del aceite que se reconocen mediante los sentidos por personas cualificadas que 24 Capítulo 3. Discusión conjunta de los resultados decir, para cada muestra de aceite teníamos una serie de puntos con unos valores de intensidad en concreto, sin embargo no habíamos trabajado con la huella espectral entera. Dado que estos datos también se nos proporcionaron, decidimos trabajar, dado su gran rendimiento con imágenes, con redes neuronales convolucionales. En [31] se realizó también una optimización de parámetros y concluyó con los mejores resultados obtenidos hasta la fecha, tal como se puede observar en la Tabla 3.3. resultados previos OCEAn FFNN CNN EVOO/VOO/LOO 0.7429 0.7942 0.8142 0.8286 EVOO/non-EVOO 0.8572 0.8842 0.9 0.8785 LOO/non-LOO 0.9071 0.9087 0.95 0.9457 Tabla 3.3: Comparativa, en términos de accuracy, entre todas las metodologías usadas hasta el momento 3.4. Generación de datos sintéticos El uso de datos procedentes de fuentes externas hace muchas veces que el intercambio de estos datos sea una tarea complicada de realizar. Debido a la idiosincrasia de los datos, a veces resulta imposible poder compartirlos dado su carácter confidencial. En el caso de los datos del aceite de oliva, debido a la cláusula de confidencialidad a la que estaban sometidos era imposible poder compartirlos. A pesar de ello creemos que es indispensable que este tipo de datos pueda compartirse con otros investigadores para sacar el máximo provecho de ellos. Es aquí cuando surge la idea de realizar un trabajo relacionado con la generación de datos sintéticos a partir de técnicas de inteligencia artificial, y más en concreto CGANs (Conditional Generati- ve Adversarial Networks). Este trabajo fue presentado en primera instancia en un congreso internacional [36], cuya experimentación fue posteriormente ampliada para su publicación en revista [34]. El trabajo consistió en la aplicación de una CGAN para la creación de datos sintéticos en 5 conjuntos de datos diferentes. La forma en la que validamos los resultados consistía en primer lugar en la comprobación de que la correlación entre los datasets reales y los generados fuese mínima, esta correlación se hacía como la media de las correlaciones entre cada variable. Obsérvese que lo óptimo es que todos los valores están cercanos a cero, ya que esto indica que la correlación es muy pobre, demostrando esto que a partir de unos datos generados es muy difícil intuir el conjunto de datos original. Por otro lado también comprobamos que las métricas obtenidas tras aplicarle algoritmos de clasificación eran similares para los generados como para los creados, esto nos permitía comprobar si a pesar de ser conjuntos diferentes, se comportaban igual ante algoritmos de aprendizaje. 3.4. Generación de datos sintéticos 25 En la figura 3.2 puede observarse, para cada dataset, cuáles fueron los índices de correlación de Pearson entre el dataset real y el generado. Figura 3.2: Índices de correlación de Pearson entre los datasets generados y los reales. Para cada dataset se realiza la correlación 1 a 1 entre las columnas de los datasets. En la Tabla 3.4 se muestran los resultados obtenidos para uno de los conjuntos de datos. Estos resultados se obtuvieron después de aplicar un algoritmo XGBoost a los datos generados por un lado, a los originales por otro y también a la unión de ambos. Además, para cada prueba se obtuvieron 2 métricas distintas (accuracy,F1-Score), para poder tener un marco comparativo lo más amplio posible. Los resultados muestran que las métricas son muy parecidas, lo que significa que los datos generados pueden ser usados en sustitución de los reales, pudiendo así resolver el problema de intercambio de los mismos. 26 Capítulo 3. Discusión conjunta de los resultados Accuracy F1-Score Credit Card Real 0.8204 0.4706 Fake 0.8519 0.6129 Real + Fake 0.8304 0.5121 Tabla 3.4: Resultados obtenidos tras aplicar un algoritmo XGBoost a uno de los conjuntos de datos presentes en la publicación. Para cada conjunto de datos se calcularon varias métricas. En esta tabla se muestran accuracy yF1- Score. Se puede observar que los resultados obtenidos tanto para el conjunto de datos original, como para el conjunto de datos generado es muy parecido, además, en la última fila se pueden observar los resultados obtenidos haciendo una unión de ambos conjuntos de datos. Una vez que se comprobó que la técnica funcionaba correctamente, se decidió aplicar dicha técnica a los datos del aceite de oliva. En primer lugar se comprobó que no había correlación entre los datos generados y los reales, también se calculó la correlación 1 a 1 entre todas las columnas del dataset generado y del real. En este caso se calcularon tres índices de correlación, el de Spearman, Pearson y la covarianza. Los resultados obtenidos pueden observarse en la Figura 3.3. Estos resultados vuelven a demostrar que la correlación entre los datos reales y generados es prácticamente nula, lo que significa que los datos obtenidos mediante la técnica propuesta no comprometen a los originales. Figura 3.3: Índices de correlación entre los datasets generados y los reales. Para cada dataset se realiza la correlación 1 a 1 entre las columnas de los datasets. Los resultados obtenidos muestran que la correlación es prácticamente cero. Además, En la Tabla 3.5 pueden observarse los resultados obtenidos para 3.4. Generación de datos sintéticos 27 este conjunto de datos en concreto. Los resultados muestran claramente cómo la generación de datos sintéticos supone un avance, ya que los resultados de clasificación son muy similares para las métricas probadas, en este caso accuracy yF1-Score. Accuracy F1-Score Olive Oil Real 0.8071 0.7365 Fake 0.7857 0.6555 Real + Fake 0.7964 0.6948 Tabla 3.5: Resultados de clasificación obtenidos tras aplicar un algoritmo XGBoost a los datos originales, a los generados y a la unión de ambos. Parte II Publicaciones Capítulo 4 Resumen de trabajos de investigación seleccionados Es mejor debatir una cuestión sin llegar a concluirla, que llegar a una conclusión sin debatirla. Joseph Joubert En esta parte de la memoria se detallan los artículos de investigación incluidos en esta tesis doctoral, presentada en la modalidad por compendio de artículos. También están incluidas publicaciones, que aunque no estaban incluidas en la temática principal de esta tesis doctoral, han sido resultados de estos años de trabajo. Las publicaciones se mostrarán ordenadas en función de si son revistas de impacto indexadas en JCR o congresos. Para cada una de estas publicaciones se mostrará un breve resumen de la publicación, así como el detalle de las referencias, número de citas y la revista, así como sus principales índices de impacto. A continuación se enumeran, de forma resumen, las publicaciones que se detallarán en las subsecciones siguientes: Artículos publicados en revista: •OCEAn: ordinal classification with an ensemble approach. Belén Vega-Márquez, Isabel A. Nepomuceno-Chamorro, Cristina Rubio-Escudero, José C. Riquelme. Information Sciences , Volumen: 580, 221-242, 2021. DOI: https://doi.org/10.1016/j. ins.2021.08.081. F.I.(JCR-2020): 6.795 (Q1). •Deep learning techniques to improve the performance of olive oil classification. Belén Vega-Márquez, Isabel A. Nepomuceno- Chamorro, Natividad Jurado-Campos, Cristina Rubio-Escudero. Frontiers in Chemistry, Volumen: 7, 929, 2020. DOI: https:// doi.org/10.3389/fchem.2019.00929. F.I.(JCR-2020): 5.221 (Q2). 31 32 Capítulo 4. Resumen de trabajos de investigación seleccionados •Generation of Synthetic Data with Conditional Generative Adversarial Networks. Belén Vega-Márquez, Cristina Rubio-Escudero, Isabel Nepomuceno-Chamorro. Logic Journal of the IGPL, 2020. DOI: https://doi.org/10.1093/jigpal/ jzaa059. F.I.(JCR-2020): 0.861 (Q1). Congresos internacionales: •Convolutional neural networks for olive oil classification. Belén Vega-Márquez, Andrea Carminati, Natividad Jurado-Campos, Andrés Martín-Gómez, Lourdes Arce-Jiménez, Cristina Rubio- Escudero, Isabel A. Nepomuceno-Chamorro. (2019, Junio). International Work-Conference on the Interplay Between Natural and Artificial Computation (pp. 137-145). Springer, Cham. F.I. (SJR-2019): 0.427 (Q2). •Creation of synthetic data with conditional generative adversarial networks. Belén Vega-Márquez, Cristina Rubio- Escudero, José C. Riquelme, Isabel A. Nepomuceno-Chamorro. (2019, Mayo). International Workshop on Soft Computing Models in Industrial and Environmental Applications (pp. 231-240). Springer, Cham. Otras publicaciones: •Use of deep learning architectures for day-ahead electricity price forecasting over different time periods in the Spanish electricity market. Belén Vega-Márquez, Cristina Rubio-Escudero, Isabel A. Nepomuceno-Chamorro, Ángel Arcos-Vargas. Applied Sciences, Volumen 11(13), 6097, 2021. DOI: https://doi.org/10.3390/app11136097 . F.I.(JCR-2020): 2.679 (Q2). •Using prior knowledge in the inference of gene association networks. Isabel A. Nepomuceno-Chamorro, Juan A. Nepomuceno, José Luis Galván-Rojas, Belén Vega-Márquez, Cristina Rubio-Escudero. Applied Intelligence, Volumen 50(11), 3882- 3893, 2020. DOI: https://doi.org/10.1007/s10489-020-01705-4 . F.I.(JCR-2020): 5.086 (Q2). •An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting. Belén Vega-Márquez, Javier Solís-García, Isabel A. Nepomuceno-Chamorro, Cristina Rubio-Escudero. (2021, Septiembre). International Workshop on Soft Computing Models in Industrial and Environmental Applications (pp. 675-684). Springer, Cham. Capítulo 5 Artículos de revista 5.1. OCEAn: Ordinal Classification with an Ensemble Approach Autores Belén Vega-Márquez, Isabel A. Nepomuceno-Chamorro, Cristina Rubio-Escudero, José C. Riquelme Revista Information Sciences Año 2021 Páginas 221-242 Volumen 580 DOI 10.1016/j.ins.2021.08.081 FI 6.795 (JCR 2020) Ranking Computer Science, Information Systems. 18/161 - Q1 Citas - Tabla 5.1: Datos del artículo: OCEAn: Ordinal Classification with an Ensemble Approach Resumen Normalmente, cuando se habla de problemas de aprendizaje supervisado y más en concreto en en el ámbito de los problemas de clasificación, las etiquetas que se usan para predecir suelen tener una relación intrínseca que no se tiene en cuenta, sin embargo hay problemas reales en el que los diferentes valores de la variable a predecir están relacionados entre sí mediante un orden. Por ejemplo, si tenemos un conjunto de datos de paciente de UCI cuyo objetivo es predecir si su estado será muy crítico,medio crítico ofavorable, no supone el mismo coste predecir un paciente con estado favorable cuando realmente es muy crítico, que predecir que será medio crítico cuando 33 In a formal way, the error function is calculated as follows: 1. Given a chromosome W, a prediction matrix Cand an instance e i for each of the possible labels in the dataset DS, the weight associated with each is calculated as follows: fðW;C;e i ;lÞ¼X t j¼1 w j C ij ¼¼ l  ;where C ij ¼¼ l  ¼1;if C ij ¼l 0;otherwise ð1Þ Then, given Wand C, the winning label L i is calculated for each instance e i as follows: L i ¼argmax l fðW;C;e i ;lÞ ð2Þ We illustrate an example in Fig. 4. Consider W¼ð3;1Þ; that is, the p 1 classifier has a weight of 3 and the classifier p 2 has a weight of 1. According to the C opt matrix, p 1 predicted label 0 for the instance e 2 and p 2 predicted label 1 for the same instance. As a consequence, the value for fðW;C opt ;e 2 ;0Þis 3, because classifier p 1 (with weight 3) predicted label 0. The value for fðW;C opt ;e 2 ;1Þis 1, because the classifier that predicted label 1 has a weight of 1 in W. No classifier predicted label 2 and therefore fðW;C opt ;e 2 ;2Þis 0. Thus, the winning label L 2 is 0. 2. Finally, the error function associated with a specific individual Wand a prediction matrix Cbelonging to a dataset DS arethe calculated as the sum of the costs, according the cost matrix M, for each example in DS: FðW;C;DSÞ¼X n i¼1 Ml i ;L i ðÞ ð3Þ where l i is the actual label for i th instance. An example can be seen in Fig. 5. In the figure, step 1 is applied to all rows in the prediction matrix C opt , giving the winning label for each of the instances. Then, taking into account the cost matrix Mand the real label of the 3 instances, the error function is calculated. The error function sums up, for each of the instances, the cost of predicting the resulting label in step 1 (L i ) when the real one is l i . Selection: The methodology used to make the selection is the deterministic tournament method. This technique runs several tournaments among a few chromosomes chosen randomly from the population. The number of chromosomes that participate in a tournament is called selection pressure, represented by sel press , which determines the probability that individuals with worse error take part in the tournament. In each tournament, the winning chromosome is the one with a lowest error value. Reproduction and crossover. Uniform crossover was used, because it is one of the most effective methods when looking for all possibilities that can arise when the parents are recombined. In this type of crossover, each gene in the offspring has the same probability of belonging to one or the other parent. The number of children generated by 2 fathers is called n h . Mutation. Nonuniform mutation was used in this paper. For each chromosome in the population, this mutation operator takes into account a mu press pressure, so that each individual in the new population has a probability of a gene mutation of mu press %. The new gene is randomly generated from a chosen interval. 3.3. Evaluation step Once the best weights are obtained after the optimization process, these weights are applied to the prediction matrix over the validation set. This allows us to obtain a prediction, which is used to calculate the validation measures, in this case, MAE and MZE. MAE is the average of the absolute differences between prediction and actual observation, and MZE is the error rate of the classifier: Fig. 4. Step 1 example. The winning label for the first row in C opt is calculated by taking into account the set of weights W. It can be seen that the winning label is 0, because it is the one with the maximum weight. Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 226 MAE ¼1 nX n j¼1 y j ^ y j ð4Þ MZE ¼1 nX n j¼1 y j –^ y j ¼1Accuracy ð5Þ where y i is the true label and ^ y i is the predicted label. It should be noted that, in ordinal classification, when the labels are not numeric, MAE and MZE are calculated by first converting the labels to numerical values based on the cost matrix. To make the evaluation step more reliable, our proposal took into account several splits of the data, which led to a number of iterations of the algorithm; that is, the OCEAn proposal was trained and validated with each of the subsets resulting from each partition, and the resulting measure is the average of the results obtained with each iteration. 3.4. Pseudocode The pseudocode for our proposal is given in Algorithm 1. Algorithm 1. OCEAn Algorithm Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 227 4. Results In this section, we describe the experiments that were carried out to analyze the performance of the proposal. First, selected datasets are described. Second, algorithm parameters are explained. Third, the software and hardware infrastructure used is specified. Fourth, the design of the comparison carried out is explained. Finally, a statistical analysis to test the effectiveness of our proposal is included. 4.1. Data description To show the power of the ensemble classifier, predictions were conducted based on the same datasets used in a previous work from Gutiérrez et al. [19] to which we compared our proposal. The datasets can be divided in 2 groups: real ordinal classification datasets extracted from 2 benchmark repositories (UCI [12] and the mldata.org repository [33]) and regression datasets retrieved from [7], which are converted into ordinal classification datasets using equal frequency discretization; that is, discretizing the target variable into different bins with the same number of instances in each bin. All of these data can be found at http://www.uco.es/grupos/ayrna/orreview, provided by the team that conducted the research in [19].It is necessary to point out that these sets of data do not belong to ordinal classification problems, they have a regression function, and they do not present the same characteristics as those related to ordinal classification, such as data unbalance. This allowed us to check how our approach behaved in both balanced and unbalanced datasets. In addition, we used a different number of bins for the same dataset, so we could see whether the size of the intervals influenced the final results. We used a total of 41 datasets, 17 of which were real ordinal classification datasets. The remaining 24 were those discretized from the 12 datasets found in [7]. Two different discretizations were made, with 5 and 10 target classes for each dataset. Tables 1 and 2 show the characteristics of each of the datasets; that is, the name, number of instances, number of attributes (total number of attributes, taking into account the class attribute as well), and number of classes. The first contains the real ordinal classification datasets, and the second the discretized datasets. 4.2. Algorithm settings One of the fundamental aspects in this research was the choice of the base algorithms to be used for the ensemble and of the parameters for the genetic algorithm. The choices made will be described below. Fig. 5. In this figure, the error function is calculated for C opt . Step 1 is repeated as many times as the number of rows in the prediction matrix, in this case 2 times. Once the winning label has been obtained for each row, the cost of predicting the winning label is calculated considering the cost matrix M. Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 228 4.2.1. Cost-sensitive base algorithms The main tool used to carry out this proposal was Weka, given its wide use in the literature [42] and its ease of use. Weka has a cost-sensitive classification algorithm, called ‘‘CostSensitiveClassifier”, which allows any classification algorithm (called base classifier) to be cost-sensitive. This can be done in 2 different ways: by re-weighting training instances according to the total cost assigned to each class or by predicting the target value with minimum expected misclassification cost [46]. In this case, the second option was used, because the intention was to maintain the same importance for each of the instances without the class interfering in it. This method also needs a cost matrix to be able to carry out ordinal classification, and the experiment took into account numerous datasets each with a different number of classes. Therefore, we opted for the simplest solution in terms of cost matrix; that is, we considered the one in which the position m ij of the matrix Mis equal to the Table 2 Discretized ordinal classification datasets. This group of datasets has the peculiarity that they are not datasets intended for ordinal classification, but are regression datasets. However, by means of a discretization process, the label can be transformed into Q different bins with equal frequency and they become suitable for ordinal classification problems. Specifically, for each of the original datasets, 2 binarizations were applied, one with 5 labels and the other with 10. The original data can be retrieved from [7]. Dataset name # Instances # Attributes # Classes pyrim5 (P5) 74 27 5 machine5 (M5) 209 7 5 housing5 (H5) 506 14 5 stock5 (S5) 950 10 5 abalone5 (A5) 4177 11 5 bank5 (B5) 8192 9 5 bank5’ (BB5) 8192 33 5 computer5 (C5) 8192 13 5 computer5’ (CC5) 8192 22 5 cal.housing5 (CH5) 20640 9 5 census5 (CE5) 22784 9 5 census5’ (CEE5) 22784 17 5 pyrim10 (P10) 74 27 10 machine10 (M10) 209 7 10 housing10 (H10) 506 14 10 stock10 (S10) 950 10 10 abalone10 (A10) 4177 11 10 bank10 (B10) 8192 9 10 bank10’ (BB10) 8192 33 10 computer10 (C10) 8192 13 10 computer10’ (CC10) 8192 22 10 cal.housing10 (CH10) 20640 9 10 census10 (CE10) 22784 9 10 census10’ (CEE10) 22784 17 10 Table 1 Real ordinal classification datasets. Although many of these datasets have not been used in the literature as ordinal classification problems, the fact that they have a categorical label can be used to establish an ordering relation among them. These datasets were extracted from UCI [12] and the mldata.org repository [33] Dataset name # Instances # Attributes # Classes contact-lenses (CL) 24 5 3 pasture (PA) 36 23 3 squash-stored (SS) 52 25 3 squash-ustored (SU) 52 24 3 tae (TA) 151 6 3 newthyroid (NT) 215 6 3 balance-scale (BS) 625 5 3 SWD (SW) 1000 11 4 car (CA) 1728 7 4 bondrate (BO) 57 11 5 toy (TO) 300 3 5 eucalyptus (EU) 736 20 5 LEV (LE) 1000 5 5 automobile (AU) 205 26 6 winequality-red (WR) 1599 12 6 ESL (ES) 488 5 9 ERA (ER) 1000 5 9 Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 229 difference between the categories represented by position ði;jÞ; that is, m ij ¼jijj.Table 3 shows 2 examples of cost matrices for 3- and 4-class classification problems. The base algorithms used for training CostSensitiveClassifier are enumerated in Table 4. These algorithms were chosen from Weka with the premise that they could handle multiclass labels, which is why we chose the 18 algorithms presented in the table below. For each category, we selected the best known methods. All were trained with the default parameters provided by the tool. 4.2.2. Weighting-vote ensemble optimization As previously noted, a genetic algorithm was the methodology used to establish the best weights in the weighting vote ensemble. There are 5 basic phases in genetic algorithms, as first stated in [21]: initialization, evaluation, selection, reproduction and crossover, and mutation. Because one of the fundamental aspects within genetic algorithms is the parameters used, a parameter search was conducted. First, we fixed the population size to 200 (N¼200), then we made a grid search (Table 5) for the 2 datasets with the fewest instances for both truly ordinal and discrete datasets; that is, contact lenses (CL) and pyrim5 (P5), and then we kept the parameters that provided better results. The parameters chosen for each of the phases are listed below and can be seen in Table 6. 1. Initialization: The initial population consisted of 200 individuals (N¼200), each of which was made up of 18 elements (one for each of the trained base algorithms) of random integers, as done in [40] between 1 and 10, in order to cover a wider spectrum of possible combinations among the 18 classifiers. This avoids the situation in which the dominant classifiers always obtained the best weights, with the goal that all algorithms take part in the optimization process. 2. Evaluation: The evaluation of individuals was carried out on the basis of an error function that calculates, for each instance of the dataset, what the cost would be of predicting a label, taking into account the vote carried out with the weights assigned by the individual in question, so that the error function would be the sum of these costs for each instance of the dataset. The goal is to minimize this error function. 3. Selection: The methodology used to make the selection was the deterministic tournament method. To be able to explore new regions in the search space, a pressure of 2 was considered; that is, in each tournament, 2 individuals participate ðsel press ¼2Þand the one with the lowest error is considered to be in the next generation. 4. Reproduction and crossover: Uniform crossover was used. The number of children generated from 2 parents is 2; that is, n children ¼2. Table 3 Example of different cost matrices for 3- and 4-class classification problems. 3-class cost matrix 4-class cost matrix 012 101 210 0 @1 A 0123 1012 2101 3210 0 B B @ 1 C C A Table 4 Base learners used in CostsensitiveClassification Weka algorithm. Algorithm Family Algorithm Name Functions Logistic (Log) MultilayerPerceptron (MLP) SMO Bayes NaivesBayes (NB) BayesNet (BN) Lazy KStar IBk Meta AdaBoostM1 (AdB) Bagging (Bag) LogitBoost (LB) ClassificationViaRegression (CVR) Trees J48 DecisionStump (DS) LMT RandomForest (RF) REPTree Rules PART JRip Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 230 5. Mutation: Mutation takes into account a mu press pressure of 0.4, so that each individual in the new population has a probability of a gene mutation of 40 %. The new gene will be randomly generated from the interval between 1 and 100. The stop criterion is the number of generations, G; in this case, we chose G¼40; that is, for each of the iterations, 40 repetitions of the genetic algorithm were conducted. We also took into account 10 different seeds to make up the ensemble, to reduce the influence of the initial population used to carry out the optimization process. For that purpose, the number of iterations was used as the initial seed in the generation of the initial population. All of the parameters chosen are listed in Table 6. 4.3. Software and experimental setting The cost-sensitive base classifiers used to carry out the ensemble were implemented with the Python Weka Wrapper, which allowed us to use Weka from within Python. It also provides a wrapper around the basic functionality of Weka. Weka is software for machine learning written in Java, developed at the University of Waikato. This is free software licensed under the GNU General Public License. The optimization process and the evaluation step were carried out with the Python programming language. Because of the large number of experiments, this work made intensive use of the High Performance Computing (HPC) cluster at Centro Informático Científico de Andalucía (CICA) formed by more than 60 nodes, with a total of almost 650 cores, with 32 calculation nodes (372 cores) connected to each other through Infiniband DDR at 20 Gbps. The source code with the different tests performed in this study can be found in https://github.com/bvegaus/OCEAn. 4.4. Experimental results In this subsection, we explain how the experimental setting was designed and the results obtained. In the survey carried out in [19], 16 algorithms were used to assess the performance obtained by each one of them. To make the comparison with our proposal, we decided to keep the algorithms that stood out in terms of error in the survey carried out in [19]; that is, SVC1V1 [22], SVMOP [13,43], SVORIM [9], and SVOREX [9], as well as a simple voting algorithm to serve as a baseline. In addition, for a more extensive and fair comparison, we added the results obtained with a model based on granular computation presented in [26]. To make the comparison and validation more reliable, we used the same methodology as in [19], where random splits were carried out: one part to train the algorithms and the remainder to test the classifiers. Because these splits only considered 2 partitions (D train ;D opt ) and we needed 3 (DS train ;DS opt ;DS v al ), we kept their D opt as our DS v al subset in both types of datasets, ordinal and discretized. For our DS train and DS opt subsets we needed to split their D train subset into 2 parts. Depending on whether the dataset was ordinal or discretized, we used different approaches. For ordinal regression datasets, 66% was considered for DS train and the remaining 34% for DS opt (in this way we had the same number of instances for optimization and validation). For the discretized regression datasets, 50% was considered for DS train and the remaining 50% for the DS opt subset. This decision was made because in some datasets, the number of instances for validation was higher than that for training, so if we kept the same proportion used in the ordinal regression datasets, only a few instances would be used for the training of classifiers and the genetic algorithm (steps 1 and 2 of the proposal). We carried out 5 random splits. The comparison between our proposal and [19] can be seen in Tables 7 and 8 for real ordinal datasets, and in Tables 9 and 10 for ordinal discretized datasets. For each dataset, we show the results achieved with SVC1V1, SVMOP, SVORIM, SVOREX, granular computing Model (GC), simple voting (SimpleVot) and results obtained with the OCEAn algorithm, for both MAE Table 6 Parameters used to conduct the genetic algorithm. Parameter Number N200 sel press 2 mu press 0.4 n children 2 G40 Table 5 Grid search for the training parameters of OCEAn. Parameter Grid Search sel press [2, 4, 6] mu press [0.2, 0.4, 0.6] G[40, 60, 80, 100] Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 231 and MZE, with the standard deviation in the 5 executions as sub-indexes. The best results obtained among the 7 algorithms are highlighted in bold. Tables 7 and 8 show the results obtained for real ordinal classification datasets, the first shows the results in terms of MAE, and the second in terms of MZE. In the case of MAE, our proposal obtained better results in 13 out of 17 experiments, and considering MZE, our proposal was better in 12 cases. Table 9 illustrates the MAE results obtained for the discretized ordinal classification datasets (considering 5 and 10 bins). In this case, our proposal performed better in 17 of the 24 datasets. The MZE results in Table 10 showed better results in 15 datasets with our approach. A graphical summary of these results can be seen in Figs. 6 and 7, where we can observe the better performance of our proposal compared with previous results. To obtain more information about the performance of our proposal, the confusion matrices of the CA dataset were added as an example (Table 11). In the table, 5 confusion matrices are shown, one for each of the iterations of each dataset (recall that each of the training steps for each dataset was composed of 5 iterations, each with a different division among training, optimization, and validation data). As can be seen, most of the values are on the diagonal of the matrices, which would be optimal, and there are no values at the bottom left and top right, which means that, for this dataset, our algorithm never predicted a label with a maximum cost. This phenomenon can be seen in the F1-Score measure, which has a value of 0.909 for the CA dataset. Table 8 MZE comparison for real ordinal datasets. MZE SVC1V1 SVMOP SVORIM SVOREX GC SimpleVot OCEAn CL 0:333 0:000 0:333 0:000 0:333 0:000 0:333 0:000 0:200 0:139 0:200 0:075 0:193 0:099 PA 0:533 0:049 0:622 0:230 0:666 0:000 0:577 0:092 0:533 0:197 0:267 0:100 0:268 0:105 SS 0:584 0:042 0:553 0:191 0:569 0:042 0:584 0:042 0:446 0:126 0:338 0:088 0:327 0:060 SU 0:353 0:116 0:338 0:042 0:538 0:000 0:384 0:143 0:400 0:147 0:262 0:069 0:249 0:046 TA 0:589 0:044 0:757 0:075 0:668 0:014 0:568 0:047 0:447 0:067 0:511 0:040 0:507 0:033 NT 0:296 0:000 0:270 0:057 0:296 0:000 0:296 0:000 0:059 0:035 0:033 0:009 0:030 0:007 BS 0:119 0:018 0:628 0:460 0:104 0:022 0:119 0:018 0:174 0:024 0:116 0:018 0:095 0:020 SW 0:518 0:025 0:662 0:059 0:557 0:036 0:523 0:044 0:434 0:029 0:416 0:025 0:416 0:019 CA 0:287 0:009 0:184 0:071 0:206 0:011 0:292 0:004 0:059 0:002 0:071 0:005 0:034 0:006 BO 0:413 0:029 0:480 0:073 0:413 0:029 0:413 0:029 0:693 0:076 0:453 0:12 0:437 0:094 TO 0:706 0:000 0:682 0:082 0:736 0:005 0:706 0:000 0:210 0:041 0:181 0:058 0:154 0:049 EU 0:641 0:044 0:825 0:125 0:490 0:034 0:672 0:033 0:470 0:048 0:341 0:024 0:337 0:023 LE 0:571 0:017 0:631 0:078 0:496 0:015 0:571 0:010 0:401 0:021 0:402 0:018 0:381 0:022 AU 0:680 0:010 0:596 0:067 0:734 0:008 0:680 0:010 0:565 0:072 0:262 0:067 0:253 0:042 WI 0:419 0:005 0:429 0:020 0:430 0:024 0:443 0:033 0:488 0:022 0:374 0:007 0:347 0:020 ES 0:434 0:021 0:303 0:028 0:331 0:014 0:406 0:017 0:345 0:006 0:318 0:018 0:319 0:014 ER 0:819 0:001 0:796 0:025 0:754 0:024 0:819 0:001 0:732 0:031 0:747 0:024 0:747 0:027 The best result obtained for each of the datasets is highlighted in bold type. Table 7 MAE comparison for real ordinal datasets. MAE SVC1V1 SVMOP SVORIM SVOREX GC SimpleVot OCEAn CL 0:500 0:000 0:500 0:000 0:500 0:000 0:500 0:000 0:300 0:247 0:300 0:218 0:303 0:179 PA 0:555 0:078 0:911 0:318 0:666 0:000 0:577 0:092 0:733 0:278 0:267 0:1 0:268 0:105 SS 0:738 0:042 0:707 0:323 0:569 0:042 0:738 0:042 0:553 0:100 0:385 0:095 0:373 0:064 SU 0:369 0:114 0:369 0:034 0:538 0:000 0:384 0:143 0:476 0:147 0:262 0:069 0:249 0:046 TA 0:684 0:110 1:005 0:143 0:668 0:014 0:626 0 :065 0:600 0:100 0:632 0:082 0:637 0:038 NT 0:296 0:000 0:385 0:130 0:296 0:000 0:296 0:000 0:077 0:065 0:033 0:009 0:030 0:007 BS 0:156 0:036 1:173 0:920 0:115 0:028 0:156 0:036 0:266 0:049 0:143 0:033 0:109 0:029 SW 0:534 0:021 0:804 0:154 0:580 0:044 0:543 0:049 0:460 0:033 0:430 0:025 0:425 0:019 CA 0:389 0:014 0:198 0:081 0:231 0:012 0:388 0:007 0:081 0:003 0:083 0:009 0:038 0:009 BO 0:600 0:094 0:693 0:111 0:600 0:094 0:600 0:094 1:080 0:098 0:587 0:179 0:558 0:151 TO 1:141 0:011 1:162 0:170 0:952 0:011 1:141 0:011 0:394 0:064 0:192 0:066 0:154 0:049 EU 1:136 0:137 1:768 0:471 0:572 0:062 1:233 0:104 0:842 0:125 0:362 0:033 0:359 0:026 LE 0:656 0:038 0:863 0:101 0:545 0:023 0:647 0:019 0:434 0:023 0:431 0:015 0:412 0:022 AU 1:115 0:023 0:850 0:097 1:015 0:016 1:115 0:023 1:584 0:229 0:346 0:087 0:330 0:061 WI 0:462 0:011 0:477 0:023 0:470 0:030 0:491 0:048 0:869 0:056 0:409 0:01 0:379 0:020 ES 0:519 0:017 0:309 0:031 0:375 0:008 0:490 0:014 0:547 0:076 0:341 0:013 0:340 0:009 ER 1:776 0:004 1:536 0:180 1:261 0:050 1:776 0:004 1:327 0:061 1:222 0:048 1:220 0:053 The best result obtained for each of the datasets is highlighted in bold type. Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 232 4.5. Comparison with base classifiers In addition to comparing our proposal with methods in the literature, we decided to see how well the base algorithms performed separately and confirm whether our ensemble exhibited improvement in the results. One of the advantages of using ensemble-based algorithms is that they can eliminate the difficult decision of which algorithm to choose when faced with a given problem. An ensemble algorithm brings together all the features of the base classifiers it uses to give a final prediction. As proof that our proposal indeed had an advantage over the base algorithms that composed it, we calculated Table 9 MAE comparison for discretized datasets. MAE SVC1V1 SVMOP SVORIM SVOREX GC SimpleVot OCEAn P5 1:075 0:313 1:658 0:459 1:166 0:000 1:116 0:194 1:316 0:234 1:042 0:213 0:964 0:116 M5 0:545 0:078 1:176 0:174 0:650 0:054 0:647 0:075 0:728 0:192 0:444 0:070 0:457 0:066 H5 0:625 0:053 1:627 0:291 0:626 0:045 1:018 0:068 0:907 0:119 0:399 0:076 0:411 0:058 S5 0:362 0:045 1:472 0:127 0:337 0:007 0:351 0:020 0:386 0:555 0:133 0:017 0:122 0:012 A5 0:852 0:055 1:393 0:093 0:782 0:019 1:061 0:026 1:101 0:072 0:702 0:008 0:685 0:006 B5 0:972 0:075 1:355 0:238 1:200 0:000 0:805 0:034 1:589 0:049 0:694 0:065 0:624 0:082 BB5 1:593 0:150 1:570 0:085 1:200 0:000 1:400 0:000 1:912 0:034 1:147 0:034 1:138 0:023 C5 0:588 0:058 1:549 0:417 0:731 0:066 0:748 0:100 1:368 0:091 0:573 0:028 0:716 0:022 CC5 0:723 0:027 1:385 0:116 0:645 0:006 0:779 0:042 1:530 0:129 0:492 0:036 0:765 0:040 CH5 1:007 0:083 1:449 0:182 1:173 0:030 1:002 0:112 1:197 0:106 0:748 0:022 0:719 0:036 CE5 1:040 0:033 1:645 0:082 1:014 0:125 1:013 0:092 1:245 0:057 0:724 0:047 0:716 0:020 CEE5 1:058 0:070 1:552 0:177 1:178 0:020 1:070 0:116 1:503 0:039 0:781 0:033 0:766 0:041 P10 2:050 0:240 3:358 0:323 2:566 0:086 2:608 0:133 3:033 0:555 2:367 0:336 2:185 0:346 M10 1:440 0:196 3:216 0:542 1:318 0:126 1:332 0:115 2:464 0:250 1:014 0:113 0:989 0:092 H10 1:395 0:135 2:866 0:144 1:106 0:059 2:293 0:273 3:018 0:140 0:949 0:076 0:929 0:087 S10 0:813 0:019 2:841 0:348 0:704 0:022 1:788 0:075 1:416 0:142 0:321 0:017 0:286 0:014 A10 1:861 0:141 2:804 0:248 1:529 0:032 2:601 0:087 2:919 0:064 1:481 0:028 1:431 0:021 B10 2:272 0:225 2:877 0:194 2:324 0:107 1:782 0:252 4:217 0:101 1:626 0:155 1:600 0:122 BB10 2:840 0:114 3:426 0:248 2:500 0:000 2:742 0:097 4:437 0:043 2:376 0:097 2:358 0:074 C10 1:289 0:065 2:622 0:346 1:337 0:055 1:490 0:177 3:848 0:120 1:208 0:060 1:509 0:038 CC10 2:304 0:181 2:797 0:542 1:271 0 :045 2:274 0:092 3:723 0:159 1:052 0:058 1:612 0:035 CH10 2:248 0:096 2:929 0:213 1:789 0:078 2:064 0:223 3:570 0:239 1:677 0:080 1:568 0:049 CE10 2:083 0:173 2:804 0:258 1:760 0:102 2:335 0:314 3:726 0:120 1:542 0:046 1:506 0:041 CEE10 2:164 0:069 3:001 0:198 1:820 0:091 2:257 0:203 4:203 0:102 1:667 0:069 1:615 0:039 The best result obtained for each of the datasets is highlighted in bold type. Table 10 MZE comparison for discretized datasets. MZE SVC1V1 SVMOP SVORIM SVOREX GC SimpleVot OCEAn P5 0:633 0:068 0:825 0:061 0:791 0:000 0:666 0:051 0:708 0:083 0:666 0:078 0:623 0:053 M5 0:474 0:033 0:749 0:030 0:606 0:052 0:467 0:048 0:508 0:047 0:407 0:053 0:409 0:055 H5 0:460 0:022 0:819 0:030 0:561 0:037 0:598 0:026 0:492 0:050 0:343 0:051 0:355 0:039 S5 0:307 0:043 0:800 0:027 0:336 0:006 0:346 0:018 0:235 0:029 0:133 0:017 0:122 0:012 A5 0:573 0:009 0:805 0:015 0:607 0:009 0:609 0:005 0:616 0:018 0:540 0:009 0:530 0:008 B5 0:652 0:036 0:768 0:029 0:800 0:000 0:627 0:046 0:706 0:010 0:56 0:042 0:522 0:054 BB5 0:751 0:024 0:780 0:015 0:800 0:000 0:800 0:000 0:792 0:003 0:759 0:013 0:755 0:011 C5 0:455 0:013 0:800 0:069 0:642 0:010 0:570 0:043 0:647 0:017 0:488 0:024 0:546 0:014 CC5 0:532 0:016 0:730 0:048 0:610 0:005 0:555 0:010 0:684 0:029 0:438 0:035 0:579 0:011 CH5 0:644 0:023 0:794 0:028 0:796 0:005 0:630 0:041 0:653 0:024 0:572 0:007 0:551 0:017 CE5 0:616 0:005 0:809 0:018 0:744 0:045 0:631 0:017 0:642 0:008 0:553 0:020 0:547 0:014 CEE5 0:636 0:011 0:811 0:022 0:794 0:006 0:663 0:013 0:702 0:011 0:592 0:009 0:580 0:012 P10 0:825 0:054 0:966 0:034 0:900 0:037 0:850 0:063 0:808 0:104 0:858 0:092 0:838 0:088 M10 0:752 0:019 0:928 0:032 0:769 0:045 0:718 0:061 0:738 0:039 0:641 0:023 0:635 0:009 H10 0:694 0:025 0:876 0:011 0:700 0:013 0:811 0:009 0:757 0:025 0:575 0:026 0:575 0:026 S10 0:512 0:017 0:910 0:020 0:619 0:014 0:745 0:013 0:457 0:007 0:297 0:020 0:268 0:017 A10 0:750 0:009 0:899 0:006 0:775 0:015 0:780 0:009 0:810 0:003 0:743 0:006 0:736 0:005 B10 0:859 0:015 0:902 0:011 0:890 0:007 0:805 0:021 0:882 0:007 0:783 0:023 0:779 0:023 BB10 0:861 0:006 0:9023 0:002 0:900 0:000 0:879 0:010 0:898 0:001 0:875 0:014 0:876 0:013 C10 0:679 0:007 0:876 0:029 0:787 0:011 0:731 0:010 0:847 0:013 0:687 0:013 0:759 0:013 CC10 0:838 0:013 0:878 0:047 0:767 0:016 0:858 0:010 0:836 0:008 0:641 0:016 0:787 0:004 CH10 0:848 0:006 0:906 0:004 0:848 0:011 0:834 0:012 0:850 0:016 0:782 0:008 0:759 0:007 CE10 0:799 0:009 0:890 0:012 0:833 0:015 0:813 0:012 0:862 0:004 0:765 0:017 0:759 0:013 CEE10 0:817 0:010 0:899 0:008 0:834 0:011 0:834 0:008 0:884 0:006 0:793 0:007 0:788 0:004 The best result obtained for each of the datasets is highlighted in bold type. Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 233 the metrics individually for the 18 algorithms that made up our ensemble, and then calculated, for each dataset, the average of the 18 classifiers and compared it to the results obtained by OCEAn, as shown in Figs. 8 and 9. The specific results obtained for each individual classifier and dataset can be found in the Appendix section. 4.6. Statistical test Because the development or modification of algorithms is a relatively easy task, there is a need for rigorous statistical analysis of published results [16]. Therefore, to test the performance of the OCEAn approach, we applied a statistical framework. The recommended approach for carrying out the statistical comparison of multiple algorithms over multiple datasets is the nonparametric Friedman test [3], along with the Holm post hoc procedure for detecting pairwise differences between 2 algorithms within a multiple comparison test [15]. This statistical analysis was carried out using the open-source platform StatService [32]. The Friedman test could be used to determine whether the MZE and MAE scores were significantly different from the mean rank expected under the null hypothesis [27]. If the Friedman test rejected the null hypothesis with an a ¼0:05, it means the measures’s average ranks were significantly different, so a post hoc test must be carried out to evaluate the relative performance of the proposed algorithms against a control algorithm (the one with the best value; in this case, the one with the minimum value). Table 12 shows the Friedman test rankings for every algorithm in the comparison, distinguishing between MAE and MZE. The ranking shows, in both cases, that OCEAn ranks first, with scores of 1.5122 and 1.6463 respectively. The Friedman statistic was 70.829 in terms of MAE and 49.002 if we consider MZE scores, with 4 degrees of freedom in both cases. As previously stated, the level of significance considered for the Friedman test was a ¼0:05, which means that if the pvalue for Friedman was lower than this, significant differences exist among the algorithms compared, because the null hypothesis, which states that they all behave in a similar way, is rejected. In both cases (MAE and MZE rankings), the pvalue was lower than 0.00001 indicating that significant differences existed among the approaches. Fig. 6. Comparison of MAE between previous results and our approach. All datasets, real and generated are included. Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 234 Fig. 7. Comparison of MZE between previous results and our approach. All datasets, real and generated are included. Table 11 Confusion matrices for the CA (Car) dataset. Each of the tables represents 1 of the 5 divisions of the datasets. Our methodology divided each of the datasets into 3 partitions to carry out both classification with the base algorithms and optimization of the genetic algorithm. These partitions and the entire methodology were done 5 times so that there was no relationship between the data partitions and the final results Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 235 Cristina Rubio-Escudero received her Ph.D. in computer science from the University of Granada, Spain in 2007. She is a tenured lecturer in the Department of Computing Systems and Languages, University of Sevilla, and a researcher in the Data Science & Big Data research lab. She has directed some regional and national research. She is co-author of more than 50 articles, book chapters, and conference papers. Her research interests include data science in general and clinical and environmental data mining in particular. José C. Riquelme received an M.Sc. degree in mathematics and a Ph.D. degree in computer science from the University of Seville, Spain. Since 1987, he has been with the Department of Computer Science, University of Seville, where he is currently a full professor. His primary research interests include data mining, machine learning techniques, and evolutionary computation. Belén Vega-Márquez, I.A. Nepomuceno-Chamorro, C. Rubio-Escudero et al. Information Sciences 580 (2021) 221–242 242 5.2. Deep Learning Techniques to Improve the Performance of Olive Oil Classification 59 5.2. Deep Learning Techniques to Improve the Performance of Olive Oil Classification Autores Belén Vega-Márquez, Isabel A. Nepomuceno-Chamorro, Natividad Jurado-Campos, Cristina Rubio-Escudero Revista Frontiers in Chemistry Año 2020 Páginas 929 Volumen 7 DOI 10.3389/fchem.2019.00929 FI 5.221 (JCR 2020) Ranking Chemistry, Multidisciplinary (53/178 - Q2) Citas 7 (Scopus); 9 (Google Scholar); 5 (Web of Science) Tabla 5.2: Datos del artículo: Deep Learning Techniques to Improve the Performance of Olive Oil Classification Resumen Este artículo de investigación presenta una nueva herramienta de apoyo para la determinación de la calidad del aceite de oliva basada en técnicas de aprendizaje profundo. Como es bien sabido, la economía Española depende mucho de la exportación de aceite de oliva al exterior (la producción española representa aproximadamente el 60% de producción de aceite de oliva en toda Europa y el 45 % a nivel mundial), por eso se hace indispensable la determinación de forma exhaustiva de la calidad de los mismos, para evitar posibles fraudes económicos, relacionados con la venta de productos de baja calidad y también evitar problemas nacionales de salud por la venta de productos no aptos para el consumo. El sistema que se utiliza ahora mismo para determinar la calidad del aceite de oliva es el conocido como “panel de catas” en el que los expertos son los que determinan, basándose en atributos sensoriales del aceite como el gusto o el olor, la calidad del mismo. Puesto que estos atributos sensoriales pueden ser traducidos a variables químicas, hay un gran interés en utilizar técnicas químicas para ayudar a los expertos en esta tarea. Dos de las técnicas químicas con más éxito para este objetivo son la cromatografía de gases en combinación con la espectrometría de masas. El objetivo de este artículo ha sido combinar estas técnicas químicas con tratamientos de aprendizaje profundo para clasificar de forma automática muestras de aceite de oliva de 2 cosechas distintas en sus 3 clases correspondientes según la calidad: Oliva Virgen Extra (EVOO), Oliva Virgen (VOO) y Lampante (LOO). Hemos de agradecer al Departamento de Química Analítica de la Universidad de Córdoba por ser los encargados de la extracción 60 Capítulo 5. Artículos de revista y análisis químicos de las muestras, teniendo un total de 701 ejemplos, que fueron obtenidos de dos campañas diferentes, la primera comprendida entre los años 2014 al 2015 y la segunda que comprende los años 2015-2016. Los datos que se han utilizado no son los de la huella espectral entera, sino de las intensidades de aquellos puntos más interesantes proporcionados por los químicos, llamados marcadores. Con el fin de obtener los mejores resultados se ha hecho un preprocesamiento de los datos, como es la estandarización de los mismos. También se ha realizado una optimización de parámetros de los modelos acorde a la naturaleza de los datos. Para poder hacer una valoración objetiva de nuestra propuesta, se han comparado los resultados con los previos obtenidos por el equipo de trabajo de la Universidad de Córdoba. Además también se han usado otros algoritmos conocidos en la literatura, como son las Máquinas de Soporte Vectorial (SVM), k-NN, Árboles de decisión, o XGBoost resultando en una mejora con respecto a los mismos. ORIGINAL RESEARCH published: 17 January 2020 doi: 10.3389/fchem.2019.00929 Frontiers in Chemistry | www.frontiersin.org 1January 2020 | Volume 7 | Article 929 Edited by: John C. Cancilla, Scintillon Institute, United States Reviewed by: Ricard Boqué, University of Rovira i Virgili, Spain Ramesh L. Gardas, Indian Institute of Technology Madras, India *Correspondence: Belén Vega-Márquez [email protected] Specialty section: This article was submitted to Theoretical and Computational Chemistry, a section of the journal Frontiers in Chemistry Received: 15 February 2019 Accepted: 20 December 2019 Published: 17 January 2020 Citation: Vega-Márquez B, Nepomuceno-Chamorro I, Jurado-Campos N and Rubio-Escudero C (2020) Deep Learning Techniques to Improve the Performance of Olive Oil Classification. Front. Chem. 7:929. doi: 10.3389/fchem.2019.00929 Deep Learning Techniques to Improve the Performance of Olive Oil Classification Belén Vega-Márquez1*, Isabel Nepomuceno-Chamorro1, Natividad Jurado-Campos2and Cristina Rubio-Escudero1 1Department of Computer Languages and Systems, University of Sevilla, Sevilla, Spain, 2Department of Analytical Chemistry, Institute of Fine Chemistry and Nanochemistry, International Agrifood Campus of Excellence (ceiA3), University of Córdoba, Córdoba, Spain The olive oil assessment involves the use of a standardized sensory analysis according to the “panel test” method. However, there is an important interest to design novel strategies based on the use of Gas Chromatography (GC) coupled to mass spectrometry (MS), or ion mobility spectrometry (IMS) together with a chemometric data treatment for olive oil classification. It is an essential task in an attempt to get the most robust model over time and, both to avoid fraud in the price and to know whether it is suitable for consumption or not. The aim of this paper is to combine chemical techniques and Deep Learning approaches to automatically classify olive oil samples from two different harvests in their three corresponding classes: extra virgin olive oil (EVOO), virgin olive oil (VOO), and lampante olive oil (LOO). Our Deep Learning model is built with 701 samples, which were obtained from two olive oil campaigns (2014–2015 and 2015–2016). The data from the two harvests are built from the selection of specific olive oil markers from the whole spectral fingerprint obtained with GC-IMS method. In order to obtain the best results we have configured the parameters of our model according to the nature of the data. The results obtained show that a deep learning approach applied to data obtained from chemical instrumental techniques is a good method when classifying oil samples in their corresponding categories, with higher success rates than those obtained in previous works. Keywords: olive oil classification, chemometric approaches, GC-IMS method, machine learning, deep learning, feed-forward neural network 1. INTRODUCTION Olive oil is a fatty substance which is obtained from the fruit of the olive tree Olea europea L.. There are three different olive oil categories that in descending order of quality are named as extra virgin olive oil (EVOO), virgin olive oil (VOO), and lampante olive oil (LOO). The first two are edible while the last one should be refined prior to be consumed. The EVOO flavor is characterized by a pleasant balanced flavor of green and fruity sensory characteristics. In the VOO and LOO, some negative attributes (chemical compounds associated to defects) can be detected in different proportions. The EVOO is the only non-defective olive oil and therefore it is the most appreciated and expensive. Moreover, selling lower quality olive oils as EVOO is one of the most common olive oil commercial frauds. The classification of olive oil depends on (i) chemical parameters such as free Vega-Márquez et al. Deep Learning for Olive Oil Classification acidity, peroxide value and absorbance (K270 and K232) defined by the current European Union Regulation (EEC, 1991) and (ii) a sensory assessment by trained tasters. The sensory assessment methodology is slow and expensive. Consequently, instrumental analytical measurements used in conjunction with chemometric methodologies represent an alternative for reducing costs in the task of differentiating between olive oil categories. Few studies (Borràs et al., 2015; Borràs et al., 2016; Garrido- Delgado et al., 2015; Sales et al., 2017; Contreras et al., 2019b) can be found to demonstrate the potential of analytical instruments in order to complement the sensorial analysis to classify olive oil samples as EVOO, VOO, and LOO. To demonstrate the usefulness of these methods, the amount of analyzed samples of different harvests should be high in order to obtain representative conclusions. Also, the accuracy of the classification models could be assessed by splitting the total number of analyzed samples in training and testing sets. And finally, the selection of the correct chemometric approaches would be a key point to offer a method which could classify olive oil with guarantee. Machine learning algorithms have been used in chemistry for several decades obtaining successful results (Svetnik et al., 2003; Du et al., 2008). The massive use of these algorithms has been due to the fact that they create intuitive models which transform complex input chemical data to an explainable output. However, in more sophisticated chemical problems, the relationships between input data and output solutions are not so easy to identify. Apart from that, some machine learning algorithms are not efficient enough in dealing with high-dimensional data when no dimension reduction is performed. Neural networks solve most of the problems that arise with the use of machine learning algorithms: firstly, they solve the problem of searching and identifying existing relationships, resulting black-box models that are not so interpretable, but with a high level of accuracy. Lastly, there is no problem with the amount of data, that’s why they can work efficiently with high-dimensional data. The use of artificial intelligence to detect the quality of gastronomic and agricultural products is not a new research field. In particular, Deep Learning techniques are being used for similar classification tasks with promising results, for example in the detection of different types of wine using taste sensors and neural networks (Riul et al., 2004) and in food classification (De¸bska and Guzowska-´ Swider, 2011). There are also several works with the objective of determining the quality of olive oil with artificial neural networks, as expressed in the review from Gonzalez- Fernandez et al. (2019), however, none of them distinguishes among the three currently existing categories (EVOO, VOO, and LOO), they only distinguish between two (EVOO/non EVOO, LOO/non LOO). Our aim in this study has been the application of Deep Learning techniques to a group of significant markers obtained by analytical instrumentation, specifically based on gas chromatography coupled to ion mobility spectrometry (GCIMS). This approach has been applied to 701 samples of the categories EVOO, VOO, and LOO, from two different olive oil harvests (2014–2015 and 2015–2016). The study has been divided in two parts: on the one hand we have studied the two crops covering the years 2014–2016 with the aim of improving the results obtained in a work related to the same dataset (Contreras et al., 2019b) and on the other hand we have applied well known algorithms in the literature to these same harvests in order to compare them with our methodology. The article is organized as follows: section 2 provides a detailed description about the technique used to obtain the data and the algorithm and methodologies applied to carry out the classification task. Section 3 shows the results obtained with the previous techniques, and finally, section 4 samples the conclusions that have been obtained after the study. 2. MATERIALS AND METHODS In this study, we aimed at providing a data mining approach based on Deep Learning techniques to classify olive oil samples based on chemical data. The main goal is to provide a computational methodology to help and complement the standardized sensory analysis according to the panel test method (Circi et al., 2017). The process followed is known as Knowledge Discovery in Databases (KDD). According to Lara Torralbo (2014), the KDD process pursues the automated extraction of non-trivial, implicit, previously unknown and potentially useful knowledge from large volumes of data. In summary, it can be said that KDD is a term that refers to the whole process of knowledge extraction encompassing certain phases or stages as can be seen in Figure 1. The stages can be summarized as follows: •Data acquisition and selection: In this phase, data from different sources are integrated into a single data repository, creating a target dataset with interesting variables or data samples, on which discovery is to be performed. •Preprocessing: It might not be possible to perform data mining on the data collected in the dataset, because the data may not be clean, may contain irrelevant attributes, etc. Different types of data selection, cleaning and transformation techniques are applied in this phase, e.g., feature selection, data cleaning. •Transformation: The data mining algorithms that will be used in the later phase sometimes need to have a specific data input format. The transformation phase is in charge of this task, with techniques such as normalization or auto-scaling. •Data Mining: this part of the process is in charge of solving the main problem presented, using classification, regression, among others. •Evaluation: After obtaining the data mining models, the last step of the KDD process consists of evaluating the quality of these models and interpreting them to obtain the desired knowledge. In general, in order to evaluate a model, a small subset of the data (test set) is reserved and used to validate the model built with the rest of the data (training set). This approach is known as simple validation. This process is not static, that is, it can vary depending on the problem, taking into account the nature of the data chosen to decide whether to follow all phases, add extra phases or just follow some of them. We have mainly carried out four stages: data acquisition, data visualization techniques, data Frontiers in Chemistry | www.frontiersin.org 2January 2020 | Volume 7 | Article 929 Vega-Márquez et al. Deep Learning for Olive Oil Classification FIGURE 1 | Steps in the data analysis methodology. preprocessing, classification models and finally a validation stage of the proposed model. 2.1. Data Acquisition 2.1.1. GC-IMS Analysis Analyses of olive oil samples were carried out with a GC-IMS commercial instrument (FlavourSpecr). The IMS module was equipped with a tritium radioactive ionization source of 6.5 KeV and a drift tube of 5 cm long (Gesellschaft für Analytische Sensorsysteme mbH, G.A.S., Dortmund, Germany). A non-polar column (94% methyl-5% phenyl-1% vinylsilicone) with 30 m of length, an internal diameter of 0.32 mm and 0.25 µm of film thickness (SE-54-CB of CS-Chromatographie Service GmbH, Düren, Germany) was coupled to the IMS device. In addition, an automatic sampler unit (CTC-PAL, CTC Analytics AG, Zwingen, Switzerland) was employed to improve the reproducibility of measurements. The GC-IMS method for olive oil analysis was obtained from a previous work by Contreras et al. (2019b). The sample introduction system employed was a headspace generated in a 20 mL glass vial closed with magnetic cap and silicone septum. Then, 1 g of olive oil was placed in that vial and the sample was heated at 60◦C for 8 min. The automatic injection of 200 µL of headspace was carried out with a heated syringe (80◦C) into the heated injector (80◦C). The injected headspace was driven into the GC column by using nitrogen 5.0 as carrier gas at 5 mL min−1the first 6 min and then it was increased to 25 mL min−1until the end of the analysis (23 min). Neutral analytes were separated at 40◦C. Later, this neutral volatiles were introduced into the IMS ionization chamber to generate their corresponding ions. The generation of ions of this IMS device takes place due to the presence of an excess reagent whose signal is called reactant ion peak (RIP) which is always registered in the measurements. In positive polarity, the RIP consist on hydrated protons generated due to the collision of primary electrons emitted by the tritium source with nitrogen, and a subsequent series of reactions. When one analyte (M) enters into the ionization chamber, the corresponding ion is formed due to the association of M to this hydrated proton resulting in the displacement of water molecules (Jurado-Campos et al., 2018). Then, the ions were separated in the drift tube working at a constant temperature and voltage of 55◦C and 400 V cm−1, respectively. A counter-current gas flow of nitrogen was also used (drift gas) at a 250 mL min−1rate. This flow is necessary to eliminate neutral molecules in the drift tube and influences the separation of ions in it. The values of different IMS parameters were set at: 32 for average of scans for each spectrum acquired, 100 µs for grid pulse width, 21 ms for repetition rate and 150 kHz for sampling frequency. Finally, two-dimensional GC-IMS data were acquired in positive mode, represented as topographic plots in LAV software (version 2.0.0) from G.A.S. So that, each individual signal or marker included in these 2D maps is characterized by the retention time of the neutral compound in the GC column, the drift time of the ion generated in the IMS (the time that the swarm of ions spend traveling along the drift tube) and its intensity value which depends on the concentration. The intensity of each marker can be automatically obtained from the topographic plots using LAV quantification module tool of the software. 2.1.2. Datasets We analyzed 292 olive oil samples from the 2014–2015 harvest and 409 samples from the 2015-2016 harvest, henceforth named datasets D1, and D2. For D1, the 292 olive oil samples are divided in 98 EVOO, 159 VOO and 35 LOO samples. D2 harvest was composed by 92 EVOO examples, 196 VOO and 121 LOO. The structure of the dataset for harvest D1 and D2 is the same, i.e., the datasets have a total of 118 attributes, with 113 being intensity of the markers (Contreras et al., 2019b) and the remaining others indicate the identifier of the sample Frontiers in Chemistry | www.frontiersin.org 3January 2020 | Volume 7 | Article 929 Vega-Márquez et al. Deep Learning for Olive Oil Classification FIGURE 2 | Number of instances for each olive oil class in harvests from 2014 to 2016. (“Name”), the class (EVOO, LOO, VOO) to which it belongs (“Class”), the base value (“Baseline”), the position of the RIP (“RIP Position”) and the maximum intensity of the RIP (“RIP Height”) respectively. 2.2. Visualization Before applying data analysis techniques it is important to know the nature of the data. The stage of visualization undertakes this task. In this section we provide some graphical information about the dataset analyzed. In particular, two different visualizations have been carried out: first, we show the proportion of each type of olive oil sample using pie charts and second, we reported results from principal component analysis to describe possible partitions in the dataset. Figure 2 reports the proportion of each type of olive oil in the different harvests using a pie plot graphic. It can be seen that the two harvests have very few instances of EVOO compared to the last. For this reason we decided to merge these harvests into one. This union serves to improve the classification algorithm results since the training set will have more instances. After this union the distribution of instances is 190 EVOO, 355 VOO, and 156 LOO. Furthermore, a principal component analysis (PCA) has been carried out. This study aims at a priori determination of the number of possible existing partitions. Figure 3 illustrates data distribution into the first two components of PCA-analysis for 2014–2016 harvest. According to these figures there is not an a priori clear separation among classes, and therefore we decide to apply Deep Learning techniques to this problem. Deep learning techniques are able to learn a meaningful latent space, i.e., find and represent relationships among attributes that are not known a priori and are suitable for the olive oil classification problem. 2.2.1. Preprocessing Two fundamental tasks were carried out in the preprocessing phase: the normalization of samples with respect to RIP Height FIGURE 3 | PCA for the 2014–2016 harvest. in order to reduce potential instrumental variations and autoscaling of markers that may improve the results obtained in the classification task. First, the normalization is made by dividing each of the values of markers for the maximum value of the RIP, in order to work with more homogeneous data. Second, after carrying out several tests, we found out that the auto-scaling (sometimes also called, standardization, or z-transformation) of markers resulted in slightly improved classification results. Thus, each column of the dataset was auto-scaled, i.e., numeric columns will have zero mean and unit variance. The equation used to do this task is the following: z1=xi− ¯x s(1) Frontiers in Chemistry | www.frontiersin.org 4January 2020 | Volume 7 | Article 929 Vega-Márquez et al. Deep Learning for Olive Oil Classification FIGURE 4 | An example of and architecture of two hidden layers for a dataset with three attributes and two possible classes. where: •z1: marker auto-scaled, •x1: marker we want to auto-scale, • ¯x: mean of the values for the marker, •s: sample standard deviation. 2.2.2. Classification Task For the classification task, a feed forward artificial neural network was used. An artificial neural network is a computational learning algorithm based on the architecture of the biological neural networks of the brain (Gibson and Patterson, 2016). These networks seek at finding a function that approximates data input into a desired output (DeepAI contributors, 2018). The architecture of an artificial neural network is determined by three main elements, nodes, connections between nodes and layers. Nodes are elements that try to model the neurons of the biological brains. The connections between nodes, such as synapses in brains, allow signals to be transmitted from one node to another. The combination of neurons are called a layer, the set of one or more layers constitutes the neural network. There are three types of layers: input, hidden and output. The input layer is composed of neurons that receive data of the problem that is under study. In this case, the input layer obtains the data of each of the features of the dataset, in our problem markers of the harvests. The hidden layers are those between the input and the output, so they do not have a direct connection to the environment. The output layer is the one that is responsible for providing the classification result obtained after applying the learning algorithm. Depending on the number of layers and the direction in which the information flows, several types of neural networks can be distinguished (Larranaga et al., 2019). A multilayer feed forward network (Gibson and Patterson, 2016) has been used to classify olive oils in our study. A multilayer feedforward network is a neural network with an input layer, one or more hidden layers, and an output layer where each layer has one or more artificial neurons as can be seen in Figure 4. Input layer. This is the first layer of a feed forward neural network. It receives the information of the problem, i.e., the input dataset. The number of neurons in this layer is usually the same as the number of attributes of the problem under study. Input layers in classical feed-forward neural networks are fully connected to the next hidden layer. Hidden layer. The number of hidden layer in a feed forward neural network depends on the problem. Hidden layers are in charge of encoding and transporting the information extracted from the dataset to the following layers. These layers are also the key that allow neural networks to model non-linear functions. Output layer. This layer is the one that allows to obtain the prediction of the model on the data. Depending on the nature of the problem, this prediction can be a real value (regression) or a set of probabilities (classification). To obtain these values, the corresponding activation function is chosen. In our case we have chosen the softmax function that represents the distribution of probability over Kdifferent outputs. In our example, the output is a vector with three values (or two values depending on whether the model is ternary or binary) that indicates the probability that an example belongs to one class or another. 2.2.3. Validation The previous study carried out on this same dataset (Contreras et al., 2019b) used the accuracy as the validation metric. In order to compare with the previous results we decided to take this measure to validate the generated model. Accuracy is defined as the percentage of correctly classified examples from the dataset. To calculate it, it is necessary to take a look at the confusion matrix. If we define two variables, P for the positive instances Frontiers in Chemistry | www.frontiersin.org 5January 2020 | Volume 7 | Article 929 5.3. Generation of Synthetic Data with Conditional Generative Adversarial Networks 73 5.3. Generation of Synthetic Data with Conditional Generative Adversarial Networks Autores Belén Vega-Márquez, Cristina Rubio-Escudero, Isabel A. Nepomuceno-Chamorro Revista Logic Journal of the IGPL Año 2020 Páginas - Volumen 0 (0) DOI 10.1093/jigpal/jzaa059 FI 0.861 (JCR 2020) Ranking Logic (2/21 - Q1); Mathematics (185/330 - Q3); Mathematics, Applied (210/265 - Q4) Citas 1 (Google Scholar) Tabla 5.3: Datos del artículo: Generation of Synthetic Data with Conditional Generative Adversarial Networks Resumen En este artículo se presenta una novedosa metodología para la generación de datos sintéticos mediante el uso de redes neuronales generativas adversariales, comúnmente conocida como GANs. Cada vez es mayor la preocupación en la sociedad por la privacidad de sus datos, dado el constante flujo de información que existe con las nuevas tecnologías. Esta preocupación hace que sean más las leyes de protección de datos que se instauran cada año, suponiendo esto un problema para el intercambio de información de cualquier entidad y sobre todo para el ámbito de investigación, en el que el uso de datos de fuentes externas y la reproducibilidad es muy importante. El uso de la inteligencia artificial está cada vez más presente en el entorno investigador dada su versatilidad y buenos resultados en diferentes estudios y no iba a ser menos en esta temática. Una de las técnicas más innovadoras para resolver la generación de datos sintéticos es el uso de redes neuronales con aprendizaje adversario. Este tipo de redes ha demostrado una gran capacidad para crear datos sintéticos a partir de los originales con una alta efectividad. El objetivo de la generación de datos sintéticos es crear un dataset que se comporte de forma similar al dataset original en las tareas de análisis, como puede ser clasificación o regresión. El problema de las redes neuronales generativas adversariales es que no toman en cuenta la etiqueta del conjunto de datos cuando se generan los nuevos, es tratado como otro atributo haciendo caso omiso de la importancia del mismo. En este trabajo se propone el uso de redes neuronales generativas adversariales condicionales (CGAN) para la 74 Capítulo 5. Artículos de revista creación de nuevos datos. Este tipo de redes tiene toda las características de las GANs pero tienen en cuenta la etiqueta del dataset para la generación de los mismos. Para poder medir la efectividad de nuestros resultados se han utilizado dos vías: la primera ha sido la comparación entre los resultados obtenidos en técnicas de clasificación con los datos generados y con los datos originales, suponiendo un valor similar una buena generación de los mismo, las métricas usadas han sido accuracy (precisión), F1-Score yAUC (área bajo la curva). Por otro lado, la segunda vía ha sido calcular la correlación entre los datos generados y los originales, observando que la correlación fuese mínima. Esta correlación mínima indicaría que la correspondencia entre los datos originales y los creados es prácticamente nula, demostrando esto que se podría conservar la privacidad sin poner en compromiso los datos originales. Generation of Synthetic Data with Conditional Generative Adversarial Networks BELÉN VEGA-MÁRQUEZ∗, Department of Computer Languages and Systems, University of Sevilla, 41012, Sevilla, Spain. CRISTINA RUBIO-ESCUDERO, Department of Computer Languages and Systems, University of Sevilla, 41012, Sevilla, Spain. ISABEL NEPOMUCENO-CHAMORRO, Department of Computer Languages and Systems, University of Sevilla, 41012, Sevilla, Spain. Abstract The generation of synthetic data is becoming a fundamental task in the daily life of any organization due to the new protection data laws that are emerging. Because of the rise in the use of Artificial Intelligence, one of the most recent proposals to address this problem is the use of Generative Adversarial Networks (GANs). These types of networks have demonstrated a great capacity to create synthetic data with very good performance. The goal of synthetic data generation is to create data that will perform similarly to the original dataset for many analysis tasks, such as classification. The problem of GANs is that in a classification problem, GANs do not take class labels into account when generating new data, it is treated as any other attribute. This research work has focused on the creation of new synthetic data from datasets with different characteristics with a Conditional Generative Adversarial Network (CGAN). CGANs are an extension of GANs where the class label is taken into account when the new data is generated. The performance of our results has been measured in two different ways: firstly, by comparing the results obtained with classification algorithms, both in the original datasets and in the data generated; secondly, by checking that the correlation between the original data and those generated is minimal. Keywords: Synthetic Data, Conditional Generative Adversarial Networks, Deep Learning 1 Introduction The introduction of the new data protection law [10] has supposed that the process of sharing personal data has become increasingly tough and difficult, especially in the medical field, where data is very sensitive. Because of this, scientists and doctors have to establish agreements between themselves before sharing any personal data. These requirements slow down or even prevent the exchange of data among researchers [4]. To face this problem, several solutions have been contemplated, seeking for finding or simulating data that are similar to the real one without involving sensitive information. Among these solutions, the use of Deep Learning techniques to generate synthetic data similar to the real one stands out [6,29]. The purpose of this synthetic data is to be used to train machine learning models that can then be used on real data, so that the training is done without having to make public the real data. ∗E-mail: b[email protected] Vol. 00, No. 0, © The Author(s) 2020. Published by Oxford University Press. All rights reserved. For permissions, please e-mail: journals.per[email protected]. doi:10.1093/jigpal/jzaa059 Downloaded from https://academic.oup.com/jigpal/advance-article/doi/10.1093/jigpal/jzaa059/5943718 by guest on 26 May 2021 2Synthetic Data with CGAN The precision of this technique is measured by comparing the results obtained with real data and synthetic data, so that they are as similar as possible. Generative adversarial networks (GANs) [11] and its variants have attracted many researchers in their research work due to its elegant theoretical basis and its great performance in the generation of synthetic data from real data [28], such as generating clinical data on blood pressure [4]oreven generating new magnetic resonance images for segmentation tasks [22]. GANs are a type of Deep Learning model in which two networks are trained against each other in a zero-sum game framework. Commonly one network is known as Generator an the other as Discriminator [26]. The purpose of this article is to evaluate the utility of the samples generated by an adversarial neural network from five different datasets. To work with these sets of data, we have used a Conditional Generative Adversarial Neural network [16], that takes into account the class to which the instances belong. We have based the evaluation of our proposal on two different methods: the first method is to measure the correlation between the real data and the synthetic. As mentioned above, the objective of the use of these techniques is the privacy of the data, so it is advisable that the transformation process is unidirectional so that real data can not be obtained from false data. Pearson’s correlation index, covariance and Spearman’s index will measure this phenomenon, so that a low value will be optimal, meaning that the two sets of data are not correlated and cannot be inferred from each other. The second method is to compare the accuracy obtained with a classification algorithm, specifically the XGBoost for the two sets of data, i.e. real and generated data. If this accuracy is similar, it means that the model trained with the false set serves to reach conclusions about the real set without having to use it for training. The article is organised as follows: Section 2 provides a detailed description about the methodology used in all the process. Section 3 shows the results obtained with the previous techniques previously described, finally, section 4 shows the conclusions that have been obtained after the research. 2 Methodology Our aim in this study is to provide a Deep Learning approach to simulate new data from different datasets of different topics. We used a type of GAN known as Conditional Generative Adversarial Network (CGAN) which is the key technique in our approach. This is because this type of networks shows very good results in datasets that have a target class, since they take into account this data configuration to train the neural network. The new data fits as closely as possible to the data according to which class each of the instances belongs to [20]. 2.1 Generative Adversarial Networks GANs are a Deep Learning model which comprises two different neural networks, a generator and a discriminator which are simultaneously trained competitively, in a zero-sum game framework. Figure 1 shows the basic architecture of a GAN network. The generative network (G) is in charge of learning how to assign elements of a latent space n(noise) to a certain data distribution, i.e. what it does is to generate new data G(n) that is as close as possible to the real data (x). On the other hand, the functionality of the discriminator (D) consists in differentiating between elements of the original distribution, xand those created by the generative network by calculating the probability of belonging to one set or another [11]. To summarise, the discriminator network is a standard convolutional network that can categorise the examples fed to it, a binomial classifier labelling instances as real or fake. The generator is an inverse convolutional network, in a sense, while a standard convolutional Downloaded from https://academic.oup.com/jigpal/advance-article/doi/10.1093/jigpal/jzaa059/5943718 by guest on 26 May 2021 Synthetic Data with CGAN 3 FIGURE 1. GAN network architecture. nrepresents random noise that serves as input to the generator network (G), while xexemplifies real data which is used, in conjunction with fake data G(n) to train the discriminator D. FIGURE 2. CGAN network architecture. nrepresents random noise that serves as input to the generator network (G), while xexemplifies real data which is used, in conjunction with fake data G(n) to train the discriminator D.cis the class taken into account in order to generate better instances in comparison with original GAN architecture. classifier takes an example and downsamples it to produce a probability, the generator takes a vector of random noise and upsamples it to an instance. The first throws away data through downsampling techniques like max pooling, and the second generates new data. 2.2 Conditional Generative Adversarial Networks A GAN does not take into account any type of condition with respect to the data. Usually the synthetic data to be generated has a type of property that distinguishes it, which must also be used to obtain synthetic data as close as possible to the real ones. After this approach, CGANs arised. CGANs are an extension of GAN where some condition is taken into account. This condition implies that both the discriminator and the generator have to take into account some additional information, let’s call it c, where cit can be any type of additional information, such as data from another nature or some class label [16]. Figure 2 illustrates the basic architecture of a CGAN. It can be observed that the structure is very similar to the typical adversarial neural networks; however, there is one more factor to take into account, and that is the class cto which the instance belongs. Downloaded from https://academic.oup.com/jigpal/advance-article/doi/10.1093/jigpal/jzaa059/5943718 by guest on 26 May 2021 4Synthetic Data with CGAN FIGURE 3. A summary of gradient boosting trees methodology. 2.3 Classification Task The main objective of our work is to generate fake data that is as similar as possible to the real data. In order to measure the performance of our proposal, we have decided to check whether the behaviour of classifying the generated data is similar to the behaviour of real data. For the classification task we chose a Tree Gradient Boosting algorithm, particularly the Extreme Gradient Boosting (XGBoost) implementation [5] which has demonstrated to have a really good performance in classification tasks [12]. Tree gradient boosting is an ensemble algorithm which converts several weak classifiers (in this case trees) into a strong classifier so as to improve global classification performance [21]. As can we seen in Figure 3, a gradient boosting is an additive model where a tree is added and combined with other trees in each iteration to reduce the loss. 2.4 Datasets In an attempt to ensure that our methodology fits well with any type of data, we have decided to create synthetic data from five different types of datasets which are described below. These type of data has been widely used in classification related studies [3,15,19] −Default of Credit Card Clients Dataset [14]: The dataset contains information about default payments, demographic factors, credit data, history of payment and bill statements of credit card clients in Taiwan from April 2005 to September 2005. The label class is called Default payment and it indicates if the next month the payment will be carried out or not (1 or 0). −Tokyo Earthquake Dataset [2,3]: This dataset collects demographic data from earthquakes in Tokyo between 2013 and 2015. Each one of the instances is made up of seismic indicators and a label, the label for every event is defined as a logical value (1 or 0): one, if the maximum magnitude for events is larger than a predefined threshold (in this case 5); or zero, otherwise. Downloaded from https://academic.oup.com/jigpal/advance-article/doi/10.1093/jigpal/jzaa059/5943718 by guest on 26 May 2021 Synthetic Data with CGAN 5 TABLE 1. Description of the datasets. Dataset # of instances # of attributes Percentage of each class Credit Card 30000 20 80/20 Earthquake 994 94 75/25 Electricity 10000 13 65/35 Diabetes 768 8 65/35 Wine 6497 11 95/5 −Simulated Electrical Grid Stability Dataset [1]: one of the milestones of electricity companies is to predict electricity demand from variables such as price for stable electricity grids. One of the newest techniques is the Decentral Smart Grid Control (DSGC) based on mathematical equations with input values. This dataset collects a simulation of these input values and indicates through a label whether the power grid will be stable or not. −Pima Indians Diabetes Dataset [25]: The objective of the dataset is to predict whether or not a patient has diabetes, based on medical diagnostic measurements. The dataset consists on several medical predictor variables and one target variable named Outcome. −Wine Quality Dataset [8]: Data related to red and white variants of the Portuguese ”Vinho Verde” wine. This dataset contains chemical data that defines the quality of the sample using a numerical scale. For this study, a threshold of five has been chosen to define whether the wine is of quality or not, thus having a binary label, i.e. one means high quality, zero means poor quality. Finally, Table 1 summarizes the number of features and instances together with the percentage of each class. We can observe that there is a wide variety of data distributions: balanced and unbalanced datasets, number of instances from 768 to 30000 and number of attributes from 8 to 94. 2.5 Software and experimental setting The CGAN network used in this study has been implemented with the Keras library [7]. Keras is a high-level neural networks API, written in Python and capable of running on top of Tensorflow. The classification of the data has been carried out with the scikit-learn library [18]. The executions were performed on an Intel machine, specifically Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, with 64 GB of RAM and 12 cores. 3 Results 3.1 Generating New Data with CGANs First, in order to obtain better results, a preprocessing step was performed. This preprocessing consists of an standard normalization due to the fact that Deep Learning models work better with normalized data and and without null values [17]. To apply CGAN architecture to the datasets the GAN-Sandbox [9] package was used. GAN- Sandbox has a number of popular GAN architectures implemented in Python using the Keras library and a Tensorflow backend. All the results obtained are available as Jupyter Notebooks in [27]. Downloaded from https://academic.oup.com/jigpal/advance-article/doi/10.1093/jigpal/jzaa059/5943718 by guest on 26 May 2021 6Synthetic Data with CGAN TABLE 2. XGBoost Classification results in terms of Accuracy, F1-Score and Area Under the curve. Accuracy F1-Score AUC Real 0.8204 0.4706 0.7789 Credit Card Fake 0.8519 0.6129 0.8801 Real+Fake 0.8304 0.5121 0.8258 Real 0.9637 0.9254 0.9810 Earthquake Fake 0.9889 0.9780 0.9996 Real+Fake 0.9616 0.9231 0.9887 Real 0.9998 0.9998 0.9999 Electricity Fake 0.9748 0.9803 0.9961 Real+Fake 0.9866 0.9895 0.9985 Real 0.7656 0.6459 0.8294 Diabetes Fake 0.8268 0.7368 0.8950 Real+Fake 0.7857 0.6738 0.8441 Real 0.9602 0.9797 0.7866 Wine Fake 0.9627 0.9809 0.8598 Real+Fake 0.96167 0.9804 0.8339 As mentioned above, the neural network is composed of two networks, the discriminator and the generator which have the following structure: 1. Generator Network: −1 Input layer: the input layer receives noise data and the class of the real data. −4 Dense layers with the parameters specified below: •First Dense layer: 30 neurons and rectified linear activation function. •Second Dense layer: 60 neurons and rectified linear activation function. •Third Dense layer: 120 neurons and rectified linear activation function. •Fourth Dense layer: as many neurons as number of columns in the real dataset. 2. Discriminator Network: −1 Input layer: the input layer receives the fake data generated by the generator network and the real data. −4 Dense layers with the parameters specified below: −First Dense layer with 120 neurons and rectified linear activation function. −Second Dense layer with 60 neurons and rectified linear activation function. −Third Dense layer with 30 neurons and rectified linear activation function. −1 Dense layer with 1 neuron and sigmoid activation function in charge of deciding whether the entry instance is real or not. After CGAN training with 50000 epochs, new instances were generated in order to have the same number of real and fake examples. Figure 4 shows scatter plots of the first two variables of each dataset. Each row of the image represents one dataset, to the right of the image we can see the graphs corresponding to the new data generated, differentiating by colours the classes to which they belong. In the left column we see the graphs corresponding to the real data. It can be seen that a Downloaded from https://academic.oup.com/jigpal/advance-article/doi/10.1093/jigpal/jzaa059/5943718 by guest on 26 May 2021 Synthetic Data with CGAN 7 FIGURE 4. Scatterplot of some attributes of the datasets for real and generated data. priori, the data obtained is not similar to real data, which was our goal, since one of the objectives of this work is to obtain false data that behave in the same way as real data in classification tasks, but without being able to establish a relationship between these two sets of data. Downloaded from https://academic.oup.com/jigpal/advance-article/doi/10.1093/jigpal/jzaa059/5943718 by guest on 26 May 2021 88 Capítulo 6. Congresos internacionales tiempo de retención, tiempo de deriva e intensidad. Esta representación en 3D puede ser plasmada en 2 dimensiones como si fuese una imagen, esto hizo que tomásemos la decisión de utilizar redes neuronales convolucionales, dado el gran éxito que obtienen en el tratamiento de imágenes, gracias a sus capas de convolución. Los resultados demostraron que esta técnica era una buena opción para la determinación del aceite de oliva tal como se puede observar en las tablas de resultados del propio artículo. Con respecto a los trabajos anteriores, la principal ventaja de esta metodología era que no necesitaba de un preprocesamiento de datos previo. Los datos utilizados fueron los mismos que los empleados en [32], 701 muestras de aceite de oliva separadas en dos cosechas, siendo las categorías de las muestras Lampante, Virgen Extra o Virgen. Destacar que estos dos estudios se realizaron en paralelo. Convolutional Neural Networks for Olive Oil Classification Bel´en Vega-M´arquez1(B), Andrea Carminati1, Natividad Jurado-Campos2, Andr´es Mart´ın-G´omez2, Lourdes Arce-Jim´enez2, Cristina Rubio-Escudero1, and Isabel A. Nepomuceno-Chamorro1 1Department of Computer Languages and Systems, University of Sevilla, Sevilla, Spain [email protected] 2Institute of Fine Chemistry and Nanochemistry, Department of Analytical Chemistry, University of C´ordoba, C´ordoba, Spain [email protected] Abstract. The analysis of the quality of olive oil is a task that is having a lot of impact nowadays due to the large frauds that have been observed in the olive oil market. To solve this problem we have trained a Convolutional Neural Network (CNN) to classify 701 images obtained using GC-IMS methodology (gas chromatography coupled to ion mobility spectrometry). The aim of this study is to show that Deep Learning techniques can be a great alternative to traditional oil classification methods based on the subjectivity of the standardized sensory analysis according to the panel test method, and also to novel techniques provided by the chemical field, such as chemometric markers. This technique is quite expensive since the markers are manually extracted by an expert. The analyzed data includes instances belonging to two different crops, the first covers the years 2014–2015 and the second 2015–2016. Both harvests have instances classified in the three categories of existing oil, extra virgin olive oil (EVOO), virgin olive oil (VOO) and lampante olive oil (LOO). The aim of this study is to demonstrate that Deep Learning techniques in combination with chemical techniques are a good alternative to the panel test method, implying even better accuracy than results obtained in previous work. Keywords: Convolutional Neural Network ·Olive oil classification · GC-IMS method 1 Introduction Olive oil is a fatty vegetable oil obtained from the fruit of the Olive tree, a traditional tree crop of the Mediterranean area. For more than six centuries [6], olive oil has been used in different daily areas: making cosmetics, medicines and c Springer Nature Switzerland AG 2019 J. M. Ferr´andez Vicente et al. (Eds.): IWINAC 2019, LNCS 11487, pp. 137–145, 2019. https://doi.org/10.1007/978-3-030-19651-6_14 [email protected] 6.2. Creation of Synthetic Data with Conditional Generative Adversarial Networks 99 6.2. Creation of Synthetic Data with Conditional Generative Adversarial Networks Autores Belén Vega-Márquez, Cristina Rubio-Escudero, José C. Riquelme, Isabel A. Nepomuceno-Chamorro Congreso International Workshop on Soft Computing Models in Industrial and Environmental Applications (SOCO 2019) Publicación Part of the Advances in Intelligent Systems and Computing book series Año 2019 Páginas 231-240 Volumen 950 DOI 10.1007/978-3-030-20055-8_22 ISBN 978-3-030-20054-1 Ranking Internacional Citas 8 (Scopus); 10 (Google Scholar); 2 (Web of Science) Tabla 6.2: Datos del artículo: Creation of Synthetic Data with Conditional Generative Adversarial Networks Resumen Este artículo presentado en un congreso internacional, fue el trabajo previo al publicado en la revista Logic Journal in the IGPL (5.3). En este artículo, se hacía una aproximación inicial de la generación de datos sintéticos mediante redes neuronales generativas adversariales condicionales. En este artículo se realizaba la prueba sobre el conjunto de datos de Credit Card Fraud Detection y se utilizaron las dos vías de validación previamente propuestas, por un lado los índices de correlación (Pearson, Covarianza, Spearman) y por otro lado los resultados de las métricas de los algoritmos de clasificación. En este caso solamente se utilizó el algoritmo XGBoost para probar el rendimiento. Se pudo hacer una primera demostración de que las CGANs eran un tipo de redes que podían ser usadas para la generación de datos sintéticos con un gran éxito. Creation of Synthetic Data with Conditional Generative Adversarial Networks Bel´en Vega-M´arquez(B), Cristina Rubio-Escudero, Jos´e C. Riquelme, and Isabel Nepomuceno-Chamorro Department of Computer Languages and Systems, University of Sevilla, Sevilla, Spain [email protected] Abstract. The generation of synthetic data is becoming a fundamental task in the daily life of any organization due to new protection data laws that are emerging. Generative Adversarial Networks (GANs) and its variants have attracted many researchers in their research work due to its elegant theoretical basis and its great performance in the generation of new data [19]. The goal of synthetic data generation is to create data that will perform similarly to the original dataset for many analysis tasks, such as classification. The problem of GANs is that in a classification problem, GANs do not take class labels into account when generating new data, they treat it as another attribute. This research work has focused on the creation of new synthetic data from the “Default of Credit Card Clients” dataset with a Conditional Generative Adversarial Network (CGAN). CGANs are an extension of GANs where the class label is taken into account when the new data is generated. The performance of our results has been measured by comparing the results obtained with classification algorithms, both in the original dataset and in the data generated. Keywords: Synthetic data · Conditional Generative Adversarial Networks ·Deep Learning · Credit Card Fraud Data 1 Introduction The introduction of the new data protection law [7] has supposed that the process of sharing personal data has become increasingly tough and difficult, especially in the medical field, where data is highly personal and can be used to harm patients themselves. Because of this scientists and doctors have to establish agreements between themselves before sharing any personal data. These requirements slow down or even prevent the exchange of data between researchers [1]. Facing with this problem, several solutions have been contemplated that seek to find or simulate data that are similar to the real one without involving individuals. Among these solutions, the use of Deep Learning techniques to generate c Springer Nature Switzerland AG 2020 F. Mart´ınez ´ Alvarez et al. (Eds.): SOCO 2019, AISC 950, pp. 231–240, 2020. https://doi.org/10.1007/978-3-030-20055-8_22 Capítulo 7 Otras publicaciones Los conocimientos adquiridos tras la realización de todos los trabajos relacionados con esta tesis doctoral hicieron posible la aplicación de los mismos a ámbitos no tan relacionados con el proyecto principal de investigación. La oportunidad de haber trabajado en diferentes proyectos de investigación ha dado como resultado trabajos relacionados con la inteligencia artificial y el aprendizaje automático. La primera vía de desarrollo estuvo enfocada a la predicción de series temporales mediante redes neuronales, más en concreto a la predicción de precio eléctrico en el mercado español. La segunda vía, dado mi background en ciencias de la salud consistió en el uso de conocimiento a priori para realizar inferencias de redes genéticas. 111 112 Capítulo 7. Otras publicaciones 7.1. Use of Deep Learning Architectures for Day- Ahead Electricity Price Forecasting Over Different Time Periods in the Spanish Electricity Market Autores Belén Vega-Márquez, Cristina Rubio-Escudero, Isabel A. Nepomuceno-Chamorro, Ángel Arcos-Vargas Revista Applied Sciences Año 2021 Páginas 6097 Volumen 11 (13) DOI 10.3390/app11136097 FI 2.679 (JCR 2020) Ranking Engineering, Multidisciplinary (38/90 - Q2); Chemistry, Multidisciplinary (101/178 - Q3); Physics, Applied (73/160 - Q2); Materials Science, Multidisciplinary (201/334 - Q3) Citas - Tabla 7.1: Datos del artículo: Use of deep learning architectures for dayahead electricity price forecasting over different time periods in the Spanish electricity market Resumen En este artículo se analiza el mercado eléctrico español aplicando técnicas de predicción de series temporales para la previsión del precio eléctrico. La desregulación del mercado eléctrico español por parte del Estado ha hecho que la determinación del precio de la electricidad sea una tarea ardua y difícil de realizar. Dentro de todas las variables que interfieren en el mercado eléctrico, el precio es quizás una de las más importante para los involucrados de este sector, por lo que la predicción del mismo se ha convertido en una tarea primordial a día de hoy. La principal dificultad de esta tarea es el gran número de factores que intervienen en el precio, debiendo considerar muchos aspectos importantes. Las técnicas de aprendizaje profundo han demostrado ser una buena herramienta capaz de capturar de forma precisa las relaciones existentes entre los distintos factores. En este trabajo hemos realizado un marco experimental y comparativo en la predicción de precios eléctricos teniendo en cuenta 3 períodos diferentes y distintos modelos de ciencia de datos usando solamente el precio del pasado para poder predecir el precio futuro, es decir, desde un punto de vista univarible. Los modelos concretos que se han usado han sido redes de gran memoria a corto plazo 7.1. Use of Deep Learning Architectures for Day-Ahead Electricity Price Forecasting. 113 (LSTM), redes neuronales convolucionales, redes neuronales convolucionales temporales (TCN), perceptrón multicapa (MLP), árboles de decisión y random forest El objetivo que hemos perseguido con este trabajo es ver si existen relaciones entre la eficacia de los modelos y los diferentes períodos temporales y por otro lado la parametrización exhaustiva de cada uno de los modelos realizados. Los 3 períodos temporales han sido elegidos buscando que fuesen lo más característicos posibles: el primero coincide con una época en la que se denunció un fraude por parte de una empresa eléctrica, el otro período coincide con el período de cuarentena, y el último es un período que nos sirve de base, sin ninguna particularidad. Los resultados obtenidos han demostrado que las técnicas de deep learning son una buena alternativa para la predicción de precios eléctricos en el mercado español y que el precio eléctrico está fuertemente influenciado por las características y circunstancias sociales que estén aconteciendo en el momento temporal concreto. applied sciences Article Use of Deep Learning Architectures for Day-Ahead Electricity Price Forecasting over Different Time Periods in the Spanish Electricity Market Belén Vega-Márquez 1,* , Cristina Rubio-Escudero 1, Isabel A. Nepomuceno-Chamorro 1and Ángel Arcos-Vargas 2   Citation: Vega-Márquez, B.; Rubio-Escudero, C.; Nepomuceno-Chamorro, I.A.; Arcos-Vargas, Á. Use of Deep Learning Architectures for Day-Ahead Electricity Price Forecasting over Different Time Periods in the Spanish Electricity Market. Appl. Sci. 2021,11, 6097. https://doi.org/10.3390/app11136097 Academic Editor: Hannu Laaksonen Received: 29 April 2021 Accepted: 28 June 2021 Published: 30 June 2021 Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). 1Department of Computer Languages and Systems, University of Seville, 41012 Seville, Spain; crubioescuder[email protected] (C.R.-E.); [email protected] (I.A.N.-C.) 2 Department of Industrial Engineering and Management Science, School of Engineering, University of Seville, 41092 Seville, Spain; [email protected] *Correspondence: [email protected] Abstract: The importance of electricity in people’s daily lives has made it an indispensable commodity in society. In electricity market, the price of electricity is the most important factor for each of those involved in it, therefore, the prediction of the electricity price has been an essential and very important task for all the agents involved in the purchase and sale of this good. The main problem within the electricity market is that prediction is an arduous and difficult task, due to the large number of factors involved, the non-linearity, non-seasonality and volatility of the price over time. Data Science methods have proven to be a great tool to capture these difficulties and to be able to give a reliable prediction using only price data, i.e., taking the problem from an univariate point of view in order to help market agents. In this work, we have made a comparison among known models in the literature, focusing on Deep Learning architectures by making an extensive tuning of parameters using data from the Spanish electricity market. Three different time periods have been used in order to carry out an extensive comparison among them. The results obtained have shown, on the one hand, that Deep Learning models are quite effective in predicting the price of electricity and, on the other hand, that the different time periods and their particular characteristics directly influence the final results of the models. Keywords: electricity price forecasting; deep learning; day-ahead market; time series forecasting MSC: [2010] 00-01; 99-00 1. Introduction Electricity is of great importance to society and essential for economic and industrial development. It is, in fact, one of the most important indicators for measuring the level of technological and industrial development of a country. It is a vital commodity and no day could be conceived without it. One of the biggest difficulties when talking about electricity management is that there is no way to store large quantities of it and, consequently, a constant balance between demand and supply is necessary, heading to highly volatile market prices [1]. In the 1990s, the electricity market underwent a process of deregulation [ 2 ], which meant that the market forces and the operator that manages the market were the ones that determined the balance between supply and demand and not the monopolistic system carried out by governments. In the last decades of the 20th century, most electrical systems were reorganised, shifting from a vertically integrated structure of companies (in many cases national companies) to a structure of separate businesses (unbundling) [ 3 ]. The theoretical basis for this Appl. Sci. 2021,11, 6097. https://doi.org/10.3390/app11136097 https://www.mdpi.com/journal/applsci Appl. Sci. 2021,11, 6097 8 of 19 As in [ 12 ], we have used the Multi-Input Multi-Output (MIMO) technique to train and evaluate the models. The MIMO strategy increases the predictions’ accuracy since it does not accumulate previous predictions’ errors. This strategy makes use of a sliding window scheme to create the input-output pairs that will be used by the models to be tested. All these output pairs have a fixed prediction window, in this case 24 h, while the training window has a size defined by a factor whose value also has to be optimized [ 32 ]. In both phases, we have tried to make the prediction of day D+ 1 as realistic as possible, i.e., at a set time on day D, the market price values are predicted for each hour of day D+1. To test our models, we have divided our originals datasets in two subsets: train dataset and validation dataset. The training dataset is used to train and search the best parameters of the models, and once they are found, we evaluate their accuracy with the validation dataset. The training datasets include all data except the last month of each dataset, which will be found in the validation datasets. To evaluate each model, we obtain the model’s prediction feeding it with the validation dataset windows. For any window of the validation dataset, the model will predict the next window. Therefore, we can connect all the predicted windows and compare the forecasted time series’s accuracy with the original one. The input-output pairs for this phase are tr_v and v, respectively. However, before we can obtain the predictions of each model, they must have been trained before. To train the models, they must be fed previously with the training subsets (tr and tr_v, respectively), which follow the same moving window philosophy as validation subsets (ts and v). 3.2.1. Hyperparameter Optimization Step We have based the experimental setup of this paper on [ 12 ], in which they have adjusted the parameters for each model used in the article. The parameters to be optimized are those that better identify each model, taking into account the characteristics of each one. These parameters have been collected in a grid, where they try to use different values for any model and parameters. The grid parameters are determined based on the typical values in the literature of each model. In all cases, many hyperparameters must be configured. To establish a fair comparison among architectures, they have explored from a single layer to a deeper network on every occasion. Since there are many parameters that have been considered, we have decided to refer only to those that are most important for each model. However, all the parameters used as well as their possible values can be found in the repository located at [33]. As mentioned above, our datasets have been divided into two parts, one for training and one for validation. The search for hyperparameters has been carried out with this first subdivision (training part), for which the sliding window strategy has been followed, so that each of the models has an input layer with as many neurons as the “past history” parameter indicates (tr) and an output layer with as many neurons as the prediction horizon has been established , in this case 24 h (ts). First, we explore the LSTM architecture, which has three main parameters and can generate 18 different models. These parameters are the number of recursive layers stacked, the number of recursive units in each layer, and whether the last layer returns the order of states or just the final state. If the return sequence is False, the last layer returns a value for each cycle unit. If the return sequence is True, the last layer returns a value for each time step, forming a shape matrix ( input timesteps ×number o f units ). Similarly, as we are in a multi-step-ahead forecasting problem, the LSTM model’s output is connected to a dense layer with one neuron for prediction. The number of layers in the LSTM model ranges from 1 to 4, and the number of units varies from 32 to 168. An overview of the parameters used can be seen in Table 2. Appl. Sci. 2021,11, 6097 9 of 19 Table 2. Grid of parameters used for LSTM models. Model Parameters Values LSTM Layers Units Return sequence 1, 2, 4 32, 64, 128 True, False As far as CNN models are concerned, they are composed of stacked convolution blocks, constituting a one-dimensional convolution along layer followed by a max-pooling layer. The convolutional blocks use decreasing kernel size, as it is usual in the literature. Single-layer models have a kernel of size 3, two-layers models have kernel sizes of 5 and 3, and four-layers models have kernels of size 7-5-3, and 3.We have only considered a pooling factor of 2, as the input sequence in the datasets is not excessively long. All the parameters used are summed up in Table 3. Table 3. Grid of parameters used for CNN models. Model Parameters Values CNN Layers Filters Pool size 1, 2, 4 16, 32, 64 0, 2 TCN models are defined by five principally parameters: number of layers, number of convolutional filters, dilation factors, convolutional kernel size, and whether to return the final output or the full sequence. The number of dilations and the kernel sizes have been selected according to the receptive field of the TCN, which follows the formula ( number o f layers ×kernel size ×last dilation ). All the parameters used can be seen in the table below (Table 4). Table 4. Grid of parameters used for TCN models. Model Parameters Values TCN Layers Filters Dilations Kernel size Return sequence 1, 3 32, 64 [1, 2, 4, 8], [1, 2, 4, 8, 16] 3, 6 True, False Multilayer Perceptrons are characterized as simple neural networks for which the main difficulty is to find the best combination between the number of layers and neurons, as well as the filters. The following table shows the configuration of hidden layers used in this work. All the parameters used can be seen in Table 5. Table 5. Grid of parameters used for MLP models. Model Parameters Values MLP Hidden Layers [8], [8, 16], [16, 8], [8, 16, 32], [32, 16, 8], [8, 16, 32, 16, 8], [32], [32, 64], [64, 32], [32, 64, 128], [128, 64, 32], [32, 64, 128, 64, 32] Decision trees are defined by four main parameters, these are, splitter, max depth, min samples split and min samples leaf. Splitter defines the strategy used to choose the split at Appl. Sci. 2021,11, 6097 10 of 19 each node, the two supported strategies are “best” to choose the best split and “random” to choose the best random split. Max depth parameter establish the maximum depth of the tree. Min samples split is the minimum number of samples requires to split an internal node. Lastly, min samples leaf is the minimum number of samples required to be at a leaf node. All the parameters used can be seen in Table 6. Table 6. Grid of parameters used for regression tree models. Model Parameters Values Tree Splitter Max depth Min samples split Min samples leaf best, random 2, 4, 6, 8, 10 2, 4, 6, 8 1, 3, 5, 7 Lastly, a Random Forest can be defined by four main parameters as well, number of estimators, max depth, min samples split and min samples leaf. The last three parameters mean the same as in decision trees, the only different parameter is the number of estimators which represents the number of the trees in the forest. All the parameters used can be seen in Table 7. Table 7. Grid of parameters used for random forest models. Model Parameters Values Random Forest Number of estimators Max depth Min samples split Min samples leaf 100, 300, 600 2, 4, 6, 8, 10 2, 4, 6, 8 1, 3, 5, 7 3.2.2. Validation Step Once the best hyperparameters have been obtained in the first step, the validation of the model is carried out. This consists of predicting each day of the validation subset, also using the sliding window strategy. To make this possible, the model is fed with information from the previous days (tr_v), defined by the “past history” factor, and the next day is predicted. Once the predictions have been obtained (o), they are compared to the actual price (v) to obtain the real effectiveness of each of the models. Concerning to metrics, the predictions are evaluated with MAE, WAPE, MASE and MAPE functions. All of these functions provide different information about the accuracy of the forecasted time series. Mean absolute error (MAE) is similar to MSE (Mean squared error), but instead of applying a square, it applies an absolute value. MAE uses the same scale as the data being measured and it is a standard measure to forecast error in time series analysis [34]. MAE(v,o) = mean(|v−o|) = 1 n n ∑ i=1 |vi−oi|(1) Weighted average percentage error (WAPE) [ 35 ] is the most different metric of the chosen ones. WAPE is a variation of MAPE, where the error is rescaled dividing it by the mean. Due to this, the error is comparable across time series of varying scales. Thanks to these properties, scale-independency and interpretability, WAPE is widely used to measure the forecast accuracy, and it is very recommended for forecasting with intermittent and low-volume data. WAPE(v,o) = MAE(v,o) mean(v)=mean(|v−o|) mean(y)(2) Mean absolute scaled error (MASE) is similar to MAE, but it has some properties that makes it a recommended metric to compare forecast accuracy. An example is the property Appl. Sci. 2021,11, 6097 11 of 19 of symmetry, which penalises positive and negative errors equally, as it does with the long and shorts forecast errors. MASE(v,o) = 1 n∑n i=1|vi−oi| 1 n−1∑n i=2|vi−vi−1|(3) Mean absolute percentage error (MAPE) is calculated by averaging the absolute error divided by the observed values. This approach is useful when the size of a prediction variable is significant in evaluating the accuracy of a prediction [36]. MAPE(v,o) = 1 n n ∑ i=1     vi−oi vi     ×100 (4) Finally, and given that time is also a variable to be taken into account when talking about Deep Learning and Machine Learning models, it has been decided to add the average time (T) taken to train each of the models as a validation metric. 4. Results In this section, we describe the experiments that were carried out to analyze the performance of the proposal. Firstly, selected datasets are described. Secondly, a comparison of the results obtained by the models explained in the previous section is included. Lastly, the software and hardware infrastructure used is specified. 4.1. Datasets Description In Spain, the electricity system underwent a process of liberalisation between 1997 and 1998 [37] , in which the tasks of transmission and distribution (regulated) and generation and retail (competitive) were separated as it has been presented above. The Spanish electricity market is in charge of the OMIE, and it is organised in sequential sessions: the day-ahead, intraday, and balancing sessions. The price of electricity is different for each of the hours of each day; this price is established by means of an auction. First, power generation companies and wholesale retailers buyers are convened in the day-ahead spot market to send their quotes that include energy price pairs within 24 h of the next day. The market operator (MO) is responsible for clearing the market and providing a temporary energy plan for each bidder within 24 h of the next day. By matching all the purchase and sale bids, the electricity market establishes the price that the energy will have for each of the hours of the following day, i.e., the hourly marginal price is obtained at the intersection of the supply and demand curves [38]. Every day, between 12:30 and 13:00 of the current day D , the price that the energy will have in each of the hours of the following day D+ 1, within the daily market, is made public [39]. As it can be intuited, it is very important for the system operators to have an idea of what the prices may be in the short term, as this will significantly influence the offers they make, so a system capable of making predictions for the next day is an indispensable tool for them [40]. The main objective of this study is to see how the choice of data can influence the effectiveness of the prediction results. For each of the chosen data sets, three months have been selected, with a total of 2208 time points (hours) for each dataset, with two of them being used to set the parameters of the models (train/test step), and the remaining month to validate them (validation step). For each of the three data sets, a normalization of the data has been carried out, the type of normalization is also a parameter to be chosen in the first stage, the possible values are a minmax or z-score normalization. Attempts have been made to choose time periods with some particularity. These are detailed below: •Fraud Period: In 2019, the National Commission for Markets and Competition (CNMC) fined Endesa and Naturgy for altering electricity prices between October Appl. Sci. 2021,11, 6097 12 of 19 2016 and January 2017 [ 41 ], the months chosen for this study. This period will help to test how sensitive the models are to periods that may present tax irregularities. To make the results more comparable, the following periods used will have the same length, i.e., three months. Hourly electricity price in this period can be observed in Figure 3where the yellow line represents the mean price of all the period. •Normal period: In order to have a reference period, a stage has been chosen in which no anomalies were detected, unlike the two next data sets. This stage covers the months from September to December 2019. Figure 4represents the hourly electricity price in this period, with the mean represented by a horizontal yellow line. •Quarantine Period: The year 2020 marked a change in many daily habits due to the global pandemic. One of the most notable changes was the increase in electricity consumption, as people spent more time at their homes due to quarantine requirements. The three months from 15 March 2020 to 15 June 2020 have been chosen to represent this stage. Hourly electricity price in this period can be observed in Figure 5where the yellow line represents the mean price of all the period. Figure 3. Time series of fraud period. On the x-axis we can see that this period covers electricity prices between October 2016 and January 2017, when the Fraud was reported. The horizontal line observed indicates the average for that period which is 63.46, a much higher value than the other periods in the study. Figure 4. Time series of normal period. On the x-axis we can see that this period covers electricity prices from September to December 2019. The horizontal line which can be seen indicates the average for that period with a value of 48.91. Appl. Sci. 2021,11, 6097 13 of 19 Figure 5. Time series of quarantine period. On the x-axis we can see that this period covers electricity prices between 15 March 2020 and 15 June 2020. The horizontal line shown indicates the average for this period, which is 28.09. 4.2. Experimental Results This part of the study focuses on presenting the results obtained in terms of forecasting accuracy over the validation dataset for each dataset and each model used. The metrics used for this purpose were MAE, WAPE, MASE, MAPE and Time (T). Table 8shows a summary of the best parameter configuration obtained for each dataset and metric, specifying the best model for each one of them. Table 8. Best parameter settings obtained for each dataset and each metric. For each of the possible scenarios, there is a description of which specific algorithm has obtained the best results as well as the best hyperparameters. Dataset Best Model Mae Best Model Mape Best Model Time Quarantine STM STM REE Layers: 1 Layers: 1 Splitter: Random Units: 128 Units: 128 Max depth: 2 Return sequence: False Return sequence: False Min samples split: 2 Min Samples Leaf: 7 Normal NN REE REE Layers: 3 Splitter: Random Splitter: Random Filters: 16 Max depth: 8 Max depth: 2 Pool Size: 0 Min samples split: 4 Min samples split: 6 Min Samples Leaf: 5 Min Samples Leaf: 7 Fraud REE REE REE Splitter: Random Splitter: Random Splitter: Random Max depth: 6 Max depth: 6 Max depth: 2 Min samples split: 8 Min samples split: 8 Min samples split: 6 Min Samples Leaf: 3 Min Samples Leaf: 3 Min Samples Leaf: 7 Table 9presents a comparison between the results obtained with the different architectures for each dataset in terms of MAE. For each of the datasets, the best result is specified in bold. In quarantine period, the best result was obtained with the LSTM model, with an average error of 3.535 € . Regarding normal period, the best model was CNN, with an average error of around 4.125 € . Finally, the lowest error for the fraud period was obtained with a regression tree, resulting in an average error of 4.034 € . The last row and column of the table represent the mean MAE for each model and dataset, respectively. The best average model is LSTM and the dataset with the lowest average MAE error is the quarantine period. Appl. Sci. 2021,11, 6097 14 of 19 Table 9. MAE results for each period and each model. The results have been obtained after choosing the best set of parameters for each case by carrying out a parameter optimization process. The last row and column correspond to the means for each model and dataset, respectively. The best model for each dataset is specified in bold. Dataset LSTM CNN TCN MLP TREE RF Average Quarantine 3.535 4.199 3.997 3.648 4.063 4.414 3.976 Normal 4.212 4.125 4.991 4.192 4.160 4.152 4.305 Fraud 4.444 4.847 5.606 4.834 4.034 4.881 4.774 Average 4.064 4.391 4.865 4.224 4.085 4.482 Concerning WAPE, the results obtained can be seen in Table 10. In case of the quarantine period, the best model was LSTM with an average WAPE of 0.112. Secondly, for the normal period the best model was CNN with a percentage of 0.084. Finally, for fraud the best model was the regression tree with an average WAPE of 0.059. The last row and column of the table represent the average WAPE per model and dataset, respectively. As with MAPE, the best mean model is LSTM, while for the datasets, the best results are obtained with the fraud period. Table 10. WAPE results for each period and each model. The results have been obtained after choosing the best set of parameters for each case by carrying out a parameter optimization process. The last row and column correspond to the means for each model and dataset, respectively. The best model for each dataset is specified in bold. Dataset LSTM CNN TCN MLP TREE RF Average Quarantine 0.112 0.134 0.127 0.116 0.129 0.142 0.127 Normal 0.086 0.084 0.102 0.085 0.085 0.085 0.088 Fraud 0.065 0.072 0.083 0.071 0.059 0.072 0.071 Average 0.088 0.097 0.104 0.091 0.091 0.009 The results in terms of MASE (Table 11) follow the same pattern as for the previous two metrics. The best result for the quarantine period is obtained with the LSTM model with a value of 2.600, for the normal period the best result is 1.764 using a CNN model. Finally, for the fraud period a best result of 1.458 is obtained with decision trees. In average terms, the lowest error is obtained with the fraud period, and in terms of model with an LSTM network, as was the case for WAPE and MAE. Table 11. MASE results for each period and each model. The results have been obtained after choosing the best set of parameters for each case by carrying out a parameter optimization process. The last row and column correspond to the means for each model and dataset, respectively. The best model for each dataset is specified in bold. Dataset LSTM CNN TCN MLP TREE RF Average Quarantine 2.600 3.080 2.988 2.693 2.979 3.237 2.929 Normal 1.801 1.764 2.130 1.793 1.769 1.775 1.839 Fraud 1.610 1.737 2.009 1.732 1.458 1.749 1.716 Average 2.004 2.194 2.376 2.073 2.069 2.254 MAPE is the only metric that differs from the others as can be seen in Table 12. In this case, for the normal period, the best score is obtained with trees, with a value of 0.095, Appl. Sci. 2021,11, 6097 15 of 19 while for quarantine and fraud the best result is still LSTM and Trees, with 0.126 and 0.059, respectively, as was the case for the three previous metrics. Table 12. MAPE results for each period and each model. The results have been obtained after choosing the best set of parameters for each case by carrying out a parameter optimization process. The last row and column correspond to the means for each model and dataset, respectively. The best model for each dataset is specified in bold. Dataset LSTM CNN TCN MLP TREE RF Average Quarantine 12.6 14.8 13.5 12.7 13.3 14.8 13.6 Normal 9.8 10.2 12.0 10.1 9.5 9.6 10.2 Fraud 6.8 7.2 8.5 7.2 5.9 7.1 7.1 Average 9.7 10.7 11.3 10.0 9.6 10.5 Considering the four tables together, it can be observed that the same patterns are repeated in most cases, i.e., for each period the best model coincides for MAE, WAPE, MASE and MAPE. If the average of these measures with respect to the datasets is taken into account, the influence of each of the datasets on the final result is clearly evidenced. The most characteristic case is the fraud period, which had on average a higher price than the other two periods, resulting in a higher average MAE. However, for the case of WAPE, the fraud period is the one that achieves a lower average WAPE, which is due to the fact that this metric is divided by the mean of the values, so the higher the mean, the lower the WAPE. Something similar happens with MASE and MAPE, as it can be observed that for the fraud period the values are lower with respect to the other two periods, this is due to the fact that they are metrics inversely proportional to the values of the time series. Table 13 shows the average training time for each of the cases presented in this work. If we take into account the data sets, there is not much difference between the training, since the length of the time series is the same. In contrast, the average time taken for each model is significantly different from one model to the other, with the tree model being the fastest and the LSTM networks the slowest. The total time spent on training and validation can also be seen in Table 14 Table 13. Time results in seconds for each period and each model. The results in time were obtained by calculating the average of each of the times used to train each of the models in the parameter optimisation process. Dataset LSTM CNN TCN MLP TREE RF Average Quarantine 2.698 0.390 2.124 0.368 0.002 0.593 1.029 Normal 2.712 0.391 2.139 0.367 0.002 0.601 1.035 Fraud 2.716 0.392 2.147 0.367 0.002 0.600 1.037 Average 2.708 0.391 2.137 0.367 0.002 0.598 Table 14. Total time in seconds for training and validation of each of the models. Dataset LSTM CNN TCN MLP TREE RF Average Quarantine 2651.669 485.885 3822.345 299.721 1.716 854.445 1352.63 Normal 2685.848 488.707 3878.524 302.678 1.976 865.803 1370.589 Fraud 2663.553 494.518 3839.173 303.331 1.972 863.618 1361.027 Average 2667.023 489.703 3846.68 301.91 1.888 861.288 Appl. Sci. 2021,11, 6097 16 of 19 Finally, Figures 6–8provide a graphical comparison between the predictions on the validation set and the actual price values. The graphs show the results obtained with the best model in terms of MAE for each of the three datasets. Figure 6. Comparison between actual and predicted price by model LSTM using validation data for the quarantine data period. Figure 7. Comparison between actual and predicted price by model CNN using validation data for the normal data period. Figure 8. Comparison between actual and predicted price by model Tree using validation data for the fraud data period. Looking at all the metrics presented, both those that take into account the efficiency of predictions and time, we can observe a drawback that occurs in many works, which is that the models that take the longest time are the ones that obtain the best results. In this case, it can be clearly seen that on average the model that takes the longest is LSTM, while on average it is the one that obtains the best predictions. This is why it is sometimes necessary to choose between these two fundamental aspects, computation time and prediction efficiency. Regarding which technique should be used, we have to say that since it is a very complex problem, the choice of which technique is more appropriate will depend on the input data, however, it can be corroborated that regardless of the particular Deep Learning technique we use the results are promising. 4.3. Software and Experimental Setting All the data used in this study have been collected thanks to the API provided by the System Operator Information System (e-sios). This API allows for data retrieval in JSON format through a code in Python so that it can be then processed. Appl. Sci. 2021,11, 6097 17 of 19 Due to the large amount of experiments carried out, the executions were performed on an Intel machine, specifically Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, with 64 GB of RAM and 12 cores. The source code with the different tests performed in this study can be found in [33]. 5. Conclusions In this paper, we have made a comparison among different well-known models in architecture and their use in the prediction of electricity market prices. This comparison has been made by optimising the parameters for each of the algorithms and each of the datasets used. Given the temporal nature of the electricity price, we decided to study different time periods, specifically three, one of which served as a baseline (normal period), another period in which fraud was detected in the electricity market (fraud period) and finally the period that included the quarantine months in 2020, due to the pandemic (quarantine period). The results obtained have shown that the electricity market is clearly influenced by the social and economic circumstances that are occurring on a particular day, which is why it is important to take into account the ideosyncrasy of the data and use models in accordance with this fact. Of all the models used, three stand out: LSTM, CNN and Regression Trees. These three models are the ones that on average have given the best results for each of the validation metrics used, specifically, LSTM has obtained better results in the case of the quarantine period, CNN for the normal period and finally Trees for the fraud period. In terms of execution time, Deep Learning models have taken the longest, while Machine Learning models have taken the shortest. This is probably the biggest limitation in our study, the big difference between the time of the hyperparameter search and the training and validation time itself. However, once the hyperparameter search has been performed, the time taken to train the model and validate it is negligible. Given this limitation we also propose to look at this problem from the Data Streaming point of view, so that the model only has to be trained once. One of the disadvantages of neural networks is precisely that, the execution time they consume, however, they have proven to be a very good choice if a high percentage of prediction success is the objective of the task. In view of the results obtained, it can be said that for the specific case study we have dealt with, the Spanish electricity market, the results obtained with the different techniques are quite promising. As for which technique in particular is the best, we have to say that being such a complex problem there is no single technique that works well for all data sets, but what we can assure in view of the results is that Deep Learning techniques work well for this type of problem. This paper has focused on a time series study from a univariate point of view, i.e., the price is predicted taking into account only the past of this variable. As future work we propose the use of exogenous variables that have a direct correlation with price, such as demand [ 42 ], renewable energy capacity, balancing market price or trade value as used in [ 43 ], thus moving from a univariate to a multivariate problem. Testing with all possible variables is an almost impossible task, but the future objective we propose is to collect information on a wide range of variables and use only those that have a high degree of correlation with the variable to be predicted. Furthermore, given the large use of these models in the literature, we believe that it would be interesting to make use of ensemble technologies to make the final prediction, as this paradigm allows the advantages of individual models to be combined to make a final model. Author Contributions: Conceptualization, B.V.-M. and Á.A.-V.; methodology, B.V.-M.; software, B.V.-M.; validation, C.R.-E., I.A.N.-C. and Á.A.-V.; formal analysis, B.V.-M.; investigation, B.V.- M.; resources, C.R.-E. and I.A.N.-C.; data curation, Á.A.V. and B.V.-M.; writing—original draft preparation, B.V.-M.; writing—review and editing, C.R.-E., I.A.N.-C. and Á.A.-V.; visualization, B.V.-M.; supervision, C.R.-E., I.A.N.-C. and Á.A.-V.; project administration, C.R.-E. and I.A.N.-C.; 7.3. An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting Autores Belén Vega-Márquez, Javier Solís-García, Isabel A. Nepomuceno-Chamorro, Cristina Rubio-Escudero Congreso International Workshop on Soft Computing Models in Industrial and Environmental Applications (SOCO 2021) Publicación Part of the Advances in Intelligent Systems and Computing book series Año 2021 Páginas 675-684 Volumen 1401 DOI 10.1007/978-3-030-87869-6_64 ISBN 978-3-030-87868-9 Ranking Internacional Citas - Tabla 7.3: Datos del artículo: An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting Resumen Este artículo de congreso supuso una ampliación al trabajo ya realizado previamente en 7.1 en el que se realizaba una predicción del precio de la electricidad desde un punto de vista univariable, es decir, usando solamente el precio. El objetivo de esta nueva experimentación fue, en vista a los resultados que ya obtuvimos en la predicción de la serie temporal de precios del mercado eléctrico español desde un punto de vista univariable, aumentar el número de variables de entrada, es decir, convertirlo a un problema multivariable, para así poder hacer una comparativa entre los 2 puntos de vista. El conjunto de modelos de deep learning a utilizar fue el mismo. Los datos se obtuvieron a través de la API que proporciona REE (Red Eléctrica Española), donde pueden obtenerse un gran número de variables relacionadas con el mercado eléctrico. La gran cantidad de variables posibles a utilizar hizo que se tuviese que realizar un trabajo previo de selección de características que consistió en elegir entre 5 nuevas variables ya utilizadas en la literatura para el mismo problema. Estas variables fueron la estimación de la demanda, el 148 Capítulo 7. Otras publicaciones porcentaje de CO2 libre, la generación de energía eólica, hidráulica y solar. Una vez que se obtuvieron dichos datos, se realizó una selección de solamente 2 características para cada uno de los 3 conjuntos de datos utilizados a través del coeficiente de correlación de Pearson. Los resultados obtenidos demuestran que el uso de variables exógenas aumenta la efectividad de nuestros modelos, pudiendo llegar a apoyar las decisiones económicas y políticas tomadas dentro del mercado eléctrico español. An Extensive Comparative Between Univariate and Multivariate Deep Learning Models in Day-Ahead Electricity Price Forecasting Bel´en Vega-M´arquez(B), Javier Sol´ıs-Garc´ıa, Isabel A. Nepomuceno-Chamorro, and Cristina Rubio-Escudero Department of Computer Languages and Systems, University of Sevilla, Sevilla, Spain {bvega,inepomuceno,crubioescudero}@us.es Abstract. Electricity is a product that has greatly changed the way we think about the world. In fact, it is one of the indicators that show the progress of a civilisation. With the deregulation of the electricity market in the 1990s, electricity price forecasting became a fundamental task in all countries. This task was a challenge for both producers and consumers, as many factors had to be taken into account. In this paper, given the large number of factors that influence the electricity market price we have conducted several experiments to see how the use of multiple variables can improve the effectiveness of price prediction. Tests have been performed with different time periods using data from Spain and different Deep Learning models, showing that the use of these variables represents an improvement with respect to the univariate model, in which only the price was considered. Keywords: Univariate vs multivariate ·Deep Learning ·Day-ahead electricity price forecasting ·Time series forecasting 1 Introduction The deregulation that took place in the electricity market worldwide in the late 1990s brought a major change in the way the market was viewed. This significant change meant the beginning of a very competitive market in which prices were constantly changing [10]. This change challenged all players in the market, both consumers and producers, as both seek to maximise their profits and optimise their services respectively. These interests have made price one of the target variables within this market, with special interest being paid to the prediction of the day-ahead electricity price. This day-ahead prediction is an advantage for all market participants, since on the one hand, having a reliable prediction of it, electricity producers can make their bids before the closing time in a strategic way and, and on the c The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 H. Sanjurjo Gonz´alez et al. (Eds.): SOCO 2021, AISC 1401, pp. 675–684, 2022. https://doi.org/10.1007/978-3-030-87869-6_64 Parte III Conclusiones y Trabajos Futuros Capítulo 8 Conclusiones y trabajos futuros Ahora hace falta recoger los trozos de prudencia, aunque siempre nos falte alguno; recoger la vida vacía y caminar esperando que lentamente se llene, si es posible otra vez, como antes, de sueños desconocidos y deseos invisibles Luis Cernuda Telarañas cuelgan de la razón 8.1. Conclusiones Esta tesis doctoral por compendio de artículos ha abordado el estudio de técnicas encuadradas dentro de la inteligencia artificial para la predicción de la calidad del aceite de oliva. Este estudio ha consistido principalmente en el desarrollo de un nuevo algoritmo capaz de realizar predicciones teniendo en cuenta la naturaleza ordinal de los datos. Además, también se ha hecho un estudio de técnicas de aprendizaje profundo para el problema de la predicción de las etiquetas de calidad del aceite debido a la diversidad de datos con las que contábamos, buscando siempre obtener los mejores resultados posibles. A continuación se detallan las conclusiones de cada uno de los pasos que se siguieron para realizar esta tesis doctoral: Se ha desarrollado un algoritmo de clasificación ordinal basado en metodologías ensemble. Esta metodología hacía uso de algoritmos base cuyas predicciones debían ser combinadas para dar una predicción final. De entre todas las formas de combinación posibles se eligió la de asignación de pesos a cada uno de estos algoritmos base. Esta combinación de pesos venía dada por un proceso de optimización que se realizaba mediante un algoritmo genético con el objetivo de obtener aquella combinación que minimizase el coste de predicción. La metodología propuesta fue publicada en el artículo 3.1, donde se hizo una 163 164 Capítulo 8. Conclusiones y trabajos futuros experimentación bastante extensa en la que se comparaba nuestro algoritmo con otros ya existentes en la literatura para clasificación ordinal. En cuanto a los datos del aceite de oliva, nuestro objetivo era hacer una comparación con el trabajo previo que había realizado el grupo que nos había proporcionado los datos, cuyos resultados están publicados en [10]. El problema principal que nos encontramos fue que ellos habían utilizado una validación tradicional, dividiendo el conjunto de datos en dos, para entrenamiento y validación. La nueva metodología que propusimos hacía una división de los conjuntos en tres, una parte para entrenar los clasificadores base, otra para optimizar los pesos y otra para validar, por lo que, aunque obtuvimos muy buenos resultados (mostrados en la sección 3.1), las cifras no eran totalmente comparables. Para poder hacer una experimentación totalmente comparable, decidimos utilizar técnicas de inteligencia artificial que utilizan divisiones del conjunto de datos en entrenamiento y validación. Estas nuevas técnicas fueron redes neuronales multicapa y redes neuronales convolucionales. Los resultados obtenidos se pueden observar en los trabajos [31, 32]. Los resultados mostraron que la inteligencia artificial fue una muy buena alternativa para el estudio de este problema, ya que los resultados mejoraron, en ambos casos, a los ya existentes. La privacidad de los datos y la dificultad que supone en el ámbito de la investigación nos hizo pensar en desarrollar una metodología de generación de datos sintéticos capaces de comportarse de igual forma antes los algoritmos de clasificación que los datos reales. Para ello se hizo uso de redes neuronales generativas adversariales cuyos resultados se pueden observar en [34, 36]. En estos trabajos se hizo una experimentación que consistía en la generación de datos sintéticos de datasets existentes en la literatura, demostrando que aunque la correlación entre los datos originales y los generados era prácticamente nula, se comportaban de igual forma ante algoritmos de clasificación. Esto podía ser muy provechoso para ámbitos en los que la privacidad de datos es muy importante, o incluso para generar datos cuando se dispone de un conjunto de datos muy limitado. Por último, aunque no son trabajos incluidos como parte de esta tesis doctoral, dado el aprendizaje que adquirí con el desarrollo de la misma, decidí realizar trabajos relacionados con una temática que acontece actualmente en España, y es la de la importancia de la predicción de los precios eléctricos. El objetivo que nos propusimos fue la predicción del precio a partir de series temporales, usando técnicas de aprendizaje profundo. Los resultados fueron publicados en [37, 35]. Con estos re- 8.2. Trabajo futuro 165 sultados se pudo comprobar que las técnicas de inteligencia artificial son una muy buena opción para tratar con este tipo de datos, ya que son capaces de detectar el carácter temporal de los mismos, e incluso encontrar patrones según la época en la que se enmarquen dichos datos. Además de la temática eléctrica, decidí aprovechar mi conocimiento en el ámbito de la salud que me proporcionó el grado que estudié para aunarlo con los conocimiento en ciencias de datos adquiridos en estos años para colaborar en un artículo de investigación cuyo objetivo fue el de descubrimiento de redes de asociación de genes basándonos en un algoritmo de árboles de regresión. 8.2. Trabajo futuro Como principal trabajo futuro se propone hacer una adaptación del algoritmo de clasificación ordinal para datos en streaming. El uso de sensores hace indispensable que, a día de hoy, cualquier metodología que se desarrolle tenga en cuenta este factor, permitiendo realizar algoritmos adaptables a las circunstancias en tiempo real. Una segunda línea de trabajo, dado que los profesionales en otros ámbitos que no son la informática puede que no estén muy familiarizados con las herramientas software, es la de crear una interfaz gráfica capaz de preprocesar los datos de entrada y que permita, de forma amigable para el usuario, entrenar los algoritmos de clasificación planteados en esta tesis. Otra línea de investigación que se propone es la de basándonos en la nueva metodología desarrollada para OCEAn, realizar un nuevo modelo de selección de características. Este nuevo modelo le daría una importancia u otra a cada una de los atributos de los conjuntos de datos según el peso que esté asignado. Este peso asignado se calcularía con la misma metodología que planteamos con OCEAn para el ensemble de clasificación ordinal. Por último, y quizás es la propuesta más ambiciosa, es la de generar un meta-modelo que aúne los algoritmos de aprendizaje supervisado más utilizados en la literatura. Este nuevo meta-modelo podría utilizar técnicas de stacking que combinase las salidas de los distintos algoritmos para que sirviesen de entrada a los demás, esto quizás podría incluso mejorar los resultados ya obtenidos.