Predictive safety Algorithms in Aerospace to develop sustainable digital Engineering

1 Tesis Doctoral Ingeniería Electrónica PREDICTIVE SAFETY ALGORITHMS IN AEROSPACE TO DEVELOP SUSTAINABLE DIGITAL ENGINEERING Autor: Jorge Bautista-Hernández Directores: Professor Dr. María Ángeles Martín-Prats Dr. Johannes Von-Thadden Departamento de Ingeniería Electrónica de la Escuela Técnica Superior de Ingeniería. Universidad de Sevilla Sevilla, 2024 2 Tesis Doctoral Ingeniería Electrónica PREDICTIVE SAFETY ALGORITHMS IN AEROSPACE TO DEVELOP SUSTAINABLE DIGITAL ENGINEERING Autor: Jorge Bautista-Hernández Directores: Professor Dr. María Ángeles Martín-Prats Dr. Johannes Von-Thadden Departamento de Ingeniería Electrónica Escuela Técnica Superior de Ingeniería Universidad de Sevilla Sevilla, 2024 3 4 Tesis Doctoral: PREDICTIVE SAFETY ALGORITHMS IN AEROSPACE TO DEVELOP SUSTAINABLE DIGITAL ENGINEERING Autor: Jorge Bautista-Hernández Directores: Professor Dr. María Ángeles Martín-Prats Dr. Johannes Von-Thadden El tribunal nombrado para juzgar la Tesis arriba indicada, compuesto por los siguientes doctores: Presidente: Vocales: Secretario: acuerdan otorgarle la calificación de: El Secretario del Tribunal Fecha: 5 6 To my family To my professors 7 8 Acknowledgements I would like to express my deepest gratitude to the Professor Dr. María Ángeles Martín-Prats for all her continuous patience, persistence and feedback provided during my PhD. Additionally, I could not have undertaken this journey without the support I have received from the Dr. Johannes Von-Thadden, who generously enabled me the possibility to perform my research in Airbus Poland. I am also grateful to my spouse Magdalena Kruza-Bautista for her inspiration and moral support. I would like also to extend my thanks to her parents Krzysztof and Boguslawa for their advices, motivation and continuous help, encouraging me to continuous in this direction. Special thanks to my lovely daughter Sofía who has sweetened me in the most difficult days. I am also thankful to my parents José and Concepción for their continuous dedication and trustfulness. Lastly, I would like to have the pleasure to recognize all Professors and Researchers collaboration within the Scientific Community, who with their knowledge and expertise have positively impacted and inspired me. I would also like to acknowledge the contributions of the electronic engineering department from Seville University and Airbus Poland for supporting my PhD with their research facilities and resources. 9 Declaration I declare that this thesis is a presentation of original work and I am the sole author. This work has not previously been presented for an award at this, or any other, University. All sources are acknowledged as References. This work has been published in the scientific journal Aerospace, for which I was the principal author. To cite: • J. Bautista-Hernández and M. Á. Martín-Prats, "Artificial Intelligence Approach in Aerospace for Error Mitigation," Aerospace, vol. 11, no. 4, p. 300, 2024. • J. Bautista-Hernández and M. Á. Martín-Prats, "The Impact of Data Injection on Predictive Algorithm Developed within Electrical Manufacturing Engineering in the Context of Aerospace Cybersecurity," Aerospace, vol. 10, no. 12, p. 984, 2023. • J. Bautista-Hernández and M. Á. Martín-Prats, "Monte Carlo Simulation Applicable for Predictive Algorithm Analysis in Aerospace," in Doctoral Conference on Computing, Electrical and Industrial Systems, 2023: Springer, pp. 243-256. 16 1 Introduction 1.1 The Electrical Wiring Interconnection System The electronic components in an aircraft are interconnected by a huge number of wire harnesses that can contain up to 530 km of cables and are routed in a very limited area. The electrical wiring interconnection system engineering process can be developed in four main stages as shown in Figure 1. Figure 1: The electrical wiring interconnection system engineering process. The first stage is the system functional architecture and its design. At this stage the electrical architecture, equipment interfaces, aircraft accommodation and electrical functional allocation are defined. During the functional design of the harnesses, specifications, connectivity, lay-out are carried out according to electrical requirements and general rules. The main output of this stage is mainly the creation of the 2D wiring diagrams for each of the harness. The second stage is the 3D design which is developed by the 3D routing. The design is performed according to the wiring diagrams and design specifications. The design consists in different phases which are performed in real space allocation where the electrical harness are modelled. This is a complex process 17 where many factors need to be considered. The digital model is presented as background and used during this design phase as a support for designers to perform their work [1]. It gives some references like for example routing path and allocation of other systems. 3D harness model are created from the electrical architecture which is defined in the wiring diagrams. References present in the 3D model are necessary in order not to contact other system present nearby and to be compliance with the electrical requirements. Harnesses are designed in a 3D CAD software and are specific for every aircraft. Designers are based on the 2D electrical wiring diagrams and electrical components are selected from the electrical library accordingly. The electrical components such as connectors, backshells and in general all necessary electrical equipment in order to model the electrical wiring are represented in a 3D model [2]. Electrical protection sleeve is also included in proper location. The designated interface points where the harnesses will be attached at aircraft site need also to be considered as designated points for harness attachment to the aircraft. The outcome of this stage is to generate from the 3D model, a future manufacturing drawing for harness built named ‘Formboard’. Overall, the harnesses design aims to create the 3D-models in a real scenario and complaint with the electrical requirements. The manufacturing documentation necessary to build the harnesses are based on the following group of items: • Formboard is the 1:1 scale manufacturing drawing on top of which the wires lay and are assembled with their electrical components. This is a 2D projection drawing which its digital generation impacts positively in the manufacturing process. Projection in digital is compliant with the latest 4.0 industry [3] and produces a lot of benefits such as excellent cost savings by, for example, reducing printing paper, what generates a positive impact on the environment. It is very useful since changes can be easily integrated, and data set is right at the first time. • Technical instructions which contain the electrical logic architecture with all wiring connections. It also contains all the scripts for cutting the wires and printing labels which will be installed within the harness. • Manufacturing Bill of Material contains list of necessary components and the definition of the manufacturing process in every stage of the harness built. • Electrical tests are the scripts necessary to perform the main harness checks, which are the pin to pin, short circuit and insulation test. These checks are performed to guarantee the connection integrity. 18 The third stage of the system engineering process is the manufacturing of the harness according to the design specifications and manufacturing requirements. This stage involves another complex process, which requires the creation of necessary documentation. The engineering documentation that is created is not fully automatized. Therefore, this situation generates errors, mainly created by humans. It also creates a negative impact in the harness lifecycle. It may be difficult to track the whole history of all harness modifications since changes occur very often. This is an undesired situation which can create loss of control of the continuous modifications implemented on the harness. The fourth stage of the process is the harness assembly and the integration on the aircraft. This stage involves the installation of electrical harness physically in the aircraft according to specifications for the installation. The electrical installation needs to avoid any unsafe conditions by proper physical and functional safety analysis where any failure conditions is extremely improbable. Overall, there are many requirements and specifications that not only need to comply during all this process but also they need to be compatible during the entire lifecycle of the harness since the defined process is split within different facilities. All of this makes that the electrical system engineering process is sensitive to generate errors and undesired outcomes. There are different types of errors which can contribute to generate accidents. The most important, which can contribute up to 80% of the total are generated by humans actions. Human errors are considered as a multi-event and mainly errors can be generated from design as errors within the 3D models, errors during manufacturing processes, installation issues with potential incorrect assembly and operational errors of the aircraft such as miscommunications. Therefore, some of these errors are likely to generate a hazard. Investigations related to wiring errors were carried out by aviation authorities. After fatal accidents generated by wiring malfunctions TWA 800 in 1996 and MD-11 in 1998 [8], the authorities such as Federal Aviation Administration (FAA), US National Transportation Safety Board (NTSB), US Department of Defense and European Aviation Safety Agency (EASA) has declared the electrical wiring system as critical for safety emphasizing its importance. The conclusion of safety investigations about electrical wiring harness established by authorities were caused by failures within the electrical installation from improper design, installation, maintenance or combination of them. These are sufficient evidences to understand that electrical harness manufacturing is sensitive to generate errors mainly contributed from manual operations generated by humans during different stages of the engineering processes. 19 The design is driven by continuous changes and modifications for different reasons such as changes from customer requirements, changes in the aircraft structure or latest updates. Additionally, the electrical harness are flexible parts prone to be modified rather than the structure. All these things can generate errors during the life cycle of each harness. Thus, safety in aerospace is threaten. Figure 2 represents the 3D electrical harness design in a specific area belonging to C295 military aircraft showing harness attachment and distance to the structure. Figure 2: 3D model representation of the electrical harness installation in a specific area in C295 military aircraft. 1.2 Traditional safety methods There are different types of data collection from accidents, non-conformances, deviations, and errors. Typically safety databases are made of reports with integrated tools to allow creation of understandable data for analysts in different formats [2]. The term safety risk in the context of the aerospace refers to the probability of undesired outcomes occurring from an existing unsafe event. Undesired 20 outcomes and hazards are present in all aerospace activities. Aerospace activities are the result of a chain of complex events involving multiple variables and different scenarios with rapid changes that requires fast decisions. Due to multiples dependent variables involved in the end-to-end process, the aerospace activities are susceptible of generate hazards. Thus, there are associated risks which need to be properly consider [3, 4]. Measures for risk mitigations are directly associated with safety risk. The methods to control safety risks are based on analysis of the likelihood at safety outcomes occurring during aerospace activities [3]. Aerospace is affected by latent conditions such as design and manufactures decisions, procedure writers, management, and maintenance activities. These conditions are generated mainly by people taking incorrect decisions and poor organizational process. As the result, the real time hazards predictions technology is a challenge that should be taken into consideration. [5]. Accidents are considered as concatenations between multiple events and latent failures. Models for accidents causations had been developed in the early 90’s with different conceptual approaches. The “Swiss Cheese Model” [6] has been adopted in aviation to analyze the causes that produce accidents. The model shows a trajectory of an accident going through different slices, the holes represent the weakness of the process. This model has evolved, and several different changes have been implemented. However, there are some limitations and has been highly discussed. Recent research shows that the model is not complete since is not considering the anticipation to the error to prevent accidents [5]. In the last years data has grown exponentially, therefore requirements are more difficult to meet and errors may appear more often. Equipment is more complex and more data accuracy is required for efficient performance [7]. The safety in aerospace refers to all activities aimed at preventing operational accidents and incidents together with the investigation of those that have already happened [2]. New technology improves flight safety. However, there is evidence that human errors are trigger for errors. Accidents models focus on the constraints which limit the correct response of the safety process and produce the loss. The following aspects such as anticipation of preventing potential failures, preparation for future challenges and development of improvements in the safety measures for passengers, for aircraft and operations are crucial in the understanding of the accidents [8]. Risk assessment processes are fundamental for risk mitigation and are necessary to consider critical event for safety operations [9]. Aviation has developed aeronautical decision-making to provide the best decisions in order to evaluate the likelihood and consequences of an event. After definition 21 of the risk, impact mitigation techniques are applied to avoid potential errors. However, the electrical systems are not only increasing in volume but also in complexity. The current databases in safety risk assessment are limited to continue providing efficient risk mitigation measures. Therefore, ensuring safety at the highest level is compromised [10]. 1.3 Artificial intelligence model development Integrating AI methodologies into the manufacturing processes can enhance both efficiency and safety in order to prevent failures in existing manufacturing processes. Frequently used AI techniques in supervised learning classification methods include support vector machine (SVM), random forest (RF), logistic regression (LR), K-nearest neighbour (KNN) and XGBoost (eXtreme Gradient Boosting). SVM uses associated learning algorithms for dataset classification and decide which class of a new data point will be in each group. RF uses decision trees for making decision of a new dataset. It maintains accuracy from the average effect return of each individual tree. LR is used to estimate the probability of the binary outcomes. KNN presents high consistency in the results for limited datasets. XGBoost achieves high accuracy in predictions. Each tree learns from previous and updates the residual errors [11]. Predictive algorithm functionalities, such as model evaluation, can be beneficial in assessing the performance of the model using appropriate metrics such as accuracy, precision, recall and F1-score to define the most suitable to be used for error prediction and risk mitigation in aerospace [12]. 1.4 Algorithms development in the cybersecurity context Cyber physical systems are based on the interconnection between computational devices and physical elements. They can be used to monitor processes and to provide the best decision. They are good systems to prevent errors before they occur. Cyber physical systems provide feedback in real time and present very good adaptive and predictive capacity. The latest research in the aerospace gives evidences that these systems create a positive impact on the aviation safety and enhance efficiency, integration and autonomy of the next generation of aerospace systems. The advanced technologies enable the creation of a cyber space model to respond to this necessity [13]. The automation and predictive capabilities contribute to the connected cyber physical spaces and to align with the technological advancements in the dynamic manufacturing industry. 22 Development of algorithms using artificial intelligence techniques are useful to connect cyber physical spaces, enable automatic manufacturing processes and predictive tasks. In this way, the human intervention is reduced and the error rate is minimized, while real-time risk assessment ensures accurate decisionmaking by the automatic system. In the new digital environment, where the cyber physical systems perform their activities, a holistic approach to integrate more automation and system interconnection is required [14]. Predictive algorithms are interconnecting complex systems within the cyberspace for their correct performance. In this context, data privacy present vulnerabilities and security protocols do not apply. Confidentiality, integrity and availability of the data, which are the three pillars that cybersecurity needs to guarantee, is a challenge to guarantee in the new cyber physical space [15]. Therefore, security techniques and countermeasures need to be considered and developed to protect the algorithm to avoid malicious propagation to the outcomes [16]. Development of AI-predictive algorithm is needed to prevent creation of errors during manufacturing processes in order to maintaining aerospace safety. It is created not only to keep the highest level of safety but also to consider a strategy to detect and prevent the algorithm data from a cyberattack. Cybersecurity measures in the cybersecurity context are necessary to protect the algorithm in order to avoid malicious propagation to the outcomes. 1.5 Research objectives The aim of this thesis is to develop an artificial intelligence predictive algorithm for error mitigation and to enhance cybersecurity measures in order to ensure aerospace safety. The specific objectives of this research are: • To apply AI technology in aerospace manufacturing, particularly for designing predictive algorithms which are used for error risks evaluation and mitigation. • To create a structured evaluation and visualization of threats aiming error prediction and risk mitigation through the risk matrix function. • To create a detector pathway to predict any potential cyberattack and apply cybersecurity safety techniques to protect the performance of the algorithm. • To decrease production time by implementation of an automatic script tools which will enable faster delivery of harness documentation. 23 The structure of the thesis is as follows: Chapter 1: This chapter provides a literature review with focus on previous electrical wiring interconnection safety systems in aerospace. It describes the evolution of the safety systems and their scope up to the current study. Chapter 2: This chapter presents the methodology used in this research, including the processes and the parameters used in the artificial intelligence model developed together with description of experimental design of the studies. Also, it includes a description of the research in the cybersecurity context. Chapter 3: This chapter provides the global results achieved through the AI- model in a aerospace manufacturing plant using a dataset related to wiring of a C295 military aircraft. It also describes the validation method of the model for implementation. Chapter 4: The last chapter presents main conclusions and future directions of this research. Appendix contains three scientific publications related to this research. 24 2 Methodology The best way to mitigate errors, in absence of advanced risk assessments, is through an AI model based on AI-techniques used for prediction. AI models are necessary to provide insights of error prediction, to enhance cybersecurity measures and to ensure aerospace safety systems. This chapter describes the model developed in this study together with the input parameters and outcomes obtained in the subsequent chapters. 2.1 Algorithm structure The algorithm structure which has been developed in this study is presented in Figure 3. Figure 3: The algorithm structure showing the input parameters with risk matrix function and outcomes. The input parameters used are X1, X2, X3, X4 which are based on the configuration of the electrical harness. Each parameter is represented by Formboard (X1), Technical Instructions (X2), Manufacturing Bill of Material (X3) and Electrical Test (X4). The criteria established to assign the scores and assess the harness complexity is based on number of zones (Z), number of wires (H) and number of electrical components (N). Figure 4 represents a typical electrical harness showing the parameters criteria used for harness assessment. 25 Figure 4: Typical electrical harness showing the X1, X2, X3, X4 parameters criteria used for harness assessment. The model developed in this study is based on the following elements: • The risk matrix mechanism represented by the function  (Z, H, N), is a modern approach used for safety based on evaluation risk, prioritization and impact. The risk matrix function in an explicit form is represented in the following equation:  (Z, H, N)=∑Xi= 4 i=1 X1(Z)+ X2(H)+ X3(N) + X4(H) (1) The rule established to define the risk matrix and likelihood on error creation is defined in Table 1. Table 1: Rule established to define risk matrix assessment in each harness. The function  represents the risk matrix mechanism. Category Rule Risk Matrix 1 0 ≤  ≤ 4 Very Low 2 4 <  ≤ 8 Low 3 9 ≤  ≤ 12 Medium 4 13 ≤  ≤ 17 High 5  > 17 Very High 32 Figure 10 represents the relation between risk matrix function  in each harness and the probability of error creation. Logistic regression calculates the probability from a given input using the logistic function. Figure 10: Logistic regression dataset fits to a sigmoid function which predicts the probability of error creation. 2.1.5 Confusion matrix The confusion matrix is used to classify the dataset in order to evaluate the performance of the LR. The most important metrics which are obtained from the confusion matrix are defined in the equations 2-5 presented below. Precision is defined as the ratio of correct positive predictions to total predicted positives. The recall or sensitivity ratio defined as the correct positive predictions to the total positive examples. The accuracy is defined as the ratio of correctly predicted examples to the total of examples. The F1 score is a harmonic mean of precision and recall. 33 Additionally, the machine learning classification report provides some additional values as outcomes. The macro average is the average of the precision, recall, and F1-score. The weighted average is the weighted average of precision, recall and F1-score presented in Table 3. Table 3: Classification reports of machine learning metrics such as precision, recall, and F1-score, with macro- and weighted average. Category 0 represents the negative class that means ‘no error creation’, and category 1 represents the positive class, meaning ‘error creation’. The algorithm splits the data set into two subsets, the training set and the test set. The training set is used to fit the model and the test set evaluates the performance of the model through the ‘confusion matrix’ shown in Figure 11. From the ‘confusion matrix’, the following details are the results calculated by the algorithm. • True positive (TP): The algorithm prediction is a value that matches reality. The algorithm has been correctly classified. The element at (1,1) represents the six data samples as true positives classified as ‘error creation’. • True negatives (TN): The prediction value also matches reality. The element at (0,0) represents the 26 data samples as true negatives classified as ‘no error creation’. The diagonal elements represent Precision = TP TP+FP (2) Recall or sensitivity = TP P = TP TP+FN (3) Accuracy = TP+TN P+N =TP+TN TP+TN+FP+FN (4) F1 = 2 1 Precision + 1 Recall = 2 TP 2 TP + FP+ FN (5) Category Precision Recall F1-Score 0 ‘no error’ 1.00 0.95 0.98 1 ‘error’ 0.91 1.00 0.95 Accuracy - - 0.97 Macro-average 0.95 0.98 0.96 Weighted average 0.97 0.97 0.97 34 the instances that are correctly classified. • False positives (FP): The algorithm has inaccurately classified the output. The classifier is represented at (1,0), FP = 0. • False negative (FN): The predicted output is a false negative, which means that it was incorrectly classified. The classifier is represented at (0,1), FN = 0. Figure 11: Confusion matrix outcome classification in true positives, true negatives, false positives, and false negatives. The highest value predicted provided by the classifier is presented in TN = 26 (26 instances present a low risk of error creation) and the lowest is for TP = 6 (6 instances present a high risk of error creation). False positives (FP) and false negatives (FN) are off-diagonal values representing ‘0’ instances incorrectly classified. 2.2 Monte Carlo simulation for model validation Monte Carlo simulation is an effective approach to use for the validation of the developed model, which is designed to improve safety and reliability in the aerospace industry. The Monte Carlo simulation has been carried out to assess the reliability of the engineering system, generating random numbers to estimate uncertain variables. The simulation is defined by generation of a stochastic trial to each of the input parameters on each of 157 electrical harness on a C295 military aircraft following a probabilistic function normally distributed. After simulation using random combination of the input variables 35 the output is calculated for each parameter and represented with the probabilistic function distribution. The input parameters p1, p2, p3 are randomly assigned 1000 times varying within the range of the maximum and minimum values for the whole electrical dataset on the C295 [19]. The variation values in parameters used in the simulation are: • Wiring length p1 min is 0.1m and max is 7.7m. • Electrical components p2 min is 1.4 and max is 7.7. • Protective sheath p3 min is 0.1m and max is 3.1m. 2.3 Algorithm protection in cybersecurity context Data privacy presents vulnerabilities in the new digital age. Cyberattacks are affecting organizations worldwide. Cybersecurity is needed to ensure confidentiality, availability and integrity of data from untrusted sources access [20]. The Figure 12 represents the strategy pathway to detect if a cyberattack has happened. The strategy to protect the algorithm of the risk matrix is based on anomaly detection and monitoring. Figure 12: Detection strategy pathway to protect the algorithm in cybersecurity context. Start Detection Strategy Is = (Xi)? (Z,H,N) = (Xi)=X1(Z)+X2(H)+X3(N)+X4(H) No Yes End Scores assigned to each harness Xi: Formboard, Technical, MBOM, Electrical Test Data corrupted Continue processing 36 The implement mechanism aims to detect unusual behaviour of the risk matrix function which is defined in equation 1 in order to identify security breaches. This function depends on four parameters namely X1, X2, X3, X4 shown in the explicit formula. Indeed, the risk matrix is equal to the sum of those parameters associated to each harness. The scalar function keeps this equality when the real data is used. Thus, this condition is met allowing the code continues its execution and therefore the algorithm outcomes are presented. However, if this condition is not met, the code execution will be stopped. Thus, the algorithm will not calculate any outcomes. 37 3 Results and discussion This chapter provides a summary of the main results achieved in this research. The detailed results are described in the published articles which are presented in the Appendix of this thesis. The global results related to the development of the predictive algorithm are presented as follows: • The manufacturing time has dropped from 223.6h to 15.7h generating a time reduction of 93% for the entire electrical dataset in the example of military aircraft (C295) used in this study. • The error rate is reduced by 90% during creation of the engineering manufacturing processes in the military aircraft showing a very low likelihood of 10% of error creation in the most complex harness. • The findings obtained by Monte Carlo simulation have confirmed that the model works accurately and therefore can be implemented in aerospace industry as a methodology to improve safety. • Comparison with several AI-techniques such as SVM, RF, LR, KNN and XGBoost indicate the superior performance of the LR. The LR has shown the best performance metrics of precision and F1-score compared to other proposed existing techniques. Also, LR shows relatively good performance for accuracy and recall. • The false behaviour generated by the data injection has affected focus on analysis of the model variability to focus on analysis of the model variability the calculation of manufacturing time by 23.73%. The analysis shows that the error probability has increased by approximately 24% generating a negative impact on the algorithm effectiveness and its outcomes. The outcomes of this research demonstrate that the use of AI innovative predictive algorithm during manufacturing processes enhance safety in aerospace industry. The findings in this study have confirmed that the model developed is accurate and reliable and therefore can be implemented in aerospace industry as an innovative methodology to enhance safety. However, the results should be analysed considering the limitations of the dataset. Additionally, the potential consequences and the implications of a cyber-attack in the performance of the algorithm are important to be considered. 38 The use of AutoML can play a key role to find automatically an appropriate model which provides reliable forecast for data prediction. The accuracy and effectiveness of the predictive models used can be improved without the need of additional human intervention. It can be also very valuable tool to perform the entire process automatically. To assess the effectiveness and suitability of the selected AI methods, we have compared the traditional machine learning approach with AutoML. Performance metrics are obtained using AutoML and traditional ML for each of the AI methods proposed. The results show that leveraging AutoML can find the best classifier while reducing the time invested in manual ML. The use of AutoML in systems, which related data availability is limited and uncertainties are presented in values can be very beneficial to use. However, continuous research and innovation will be necessary to apply for AutoML to reach the highest accuracy. The use of AutoML compared to traditional machine learning revealed that proposed methods show a difference up to 3% between both approaches. Future work directions should focus on analysis of the model variability by using more extended data from other type of aircraft. There is also special interest to analyze the potential applicability of this proposed methodology to other disciplines such as energy, healthcare, transportation or infrastructure. Healthcare can get the benefit of using the risk matrix which offers real-time information to the surgeons about the level of risk they will face during the surgical procedure. This approach can provide detection and prevention of failures. Additionally, it can provide also prioritization on which patient to act in terms of mortality, in order to save their lives. In the energy sector, risk matrix can be used to analyze the hazards associated to equipment failures to enhance safety strategies or mitigate losses by implementing more efficient processes. Furthermore, risk matrix can have a key role in product safety in automotive industry. The assessment of the risks can be useful to reduce accidents rate and decrease production time. With the increasing digitalization of automotive, risk matrix can be also useful to assess the cybersecurity threats. 39 4 Conclusions • Integrating AI techniques provide a rigorous analysis and enhance efficiency, reliability and safety in aerospace. • This innovative methodology brings to the state of the art a risk matrix which enables a structured evaluation and visualization of threats aiming error prediction and risk mitigation. • In this study, AI techniques such as SVM, RF, LR, KNN and XGBoost are applied to design predictive algorithms capable to predict and mitigate errors in aerospace. • Overall performance suggests that LR is the best choice in terms of F1- score and precision presenting the value of 98% and 100% respectively. LR shows relatively good performance for accuracy (97%) and recall (95%). • Results obtained using the dataset of a military aircraft C295 estimate a reduction of 93% in manufacturing time and 90% reduction in error rate of engineering processes. • Monte Carlo simulation, which was used for model validation, gives evidence that the model can be implemented as error predictor in aerospace industry to enhance safety. • Injected data to the existing dataset creates variation in the risk matrix function. The probability of error creation during the manufacturing processes has increased by 24.22% compared to the real dataset scenario. Thus, security techniques are considered and developed to protect the algorithm and avoid malicious propagation to the outcomes. • The results obtained using AutoML show that the best classifier can be found while reducing the time invested in manual ML. • Future work should use more extended data from other type of aircraft to analyse model variability. Furthermore, the flexibility presented by the risk matrix can accommodate other types of risk. The proposed approach can be generalized to other disciplines such as healthcare, energy, and automotive. 40 5 References [1] P. B. Ladkin and W. Schepper, "EMI, TWA 800 and Swissair 111," The resks digist, vol. 21, 2000. [2] L. Weber, International Civil Aviation Organization (ICAO). Kluwer Law International BV, 2021. [3] R. S. G. R. Casado, M. H. Alencar, and A. T. de Almeida, "Combining a multidimensional risk evaluation with an implicit enumeration algorithm to tackle the portfolio selection problem of a natural gas pipeline," Reliability Engineering & System Safety, vol. 221, p. 108332, 2022. [4] J. Yang and J. Kim, "Accident diagnosis algorithm with untrained accident identification during power-increasing operation," Reliability Engineering & System Safety, vol. 202, p. 107032, 2020. [5] R. M. A. Velásquez, "Root cause analysis for inverters in solar photovoltaic plants," Engineering Failure Analysis, vol. 118, p. 104856, 2020. [6] J. Reason, E. Hollnagel, and J. Paries, "Revisiting the Swiss cheese model of accidents," Journal of Clinical Engineering, vol. 27, no. 4, pp. 110-115, 2006. [7] A. Atak and S. Kingma, "Safety culture in an aircraft maintenance organisation: A view from the inside," Safety science, vol. 49, no. 2, pp. 268-278, 2011. [8] M. Rutkowski, "Safety as an element of creating competitive advantage among airlines given the example of the Airbus A350 XWB and the Boeing 787 Dreamliner aircraft," Zeszyty Naukowe. Transport/Politechnika Śląska, 2020. [9] N. Leveson, "A new accident model for engineering safer systems," Safety science, vol. 42, no. 4, pp. 237-270, 2004. [10] M. Stamatelatos et al., "Probabilistic risk assessment procedures guide for NASA managers and practitioners," 2011. [11] A. Utku and S. K. Kaya, "Multi-layer perceptron based transfer passenger flow prediction in Istanbul transportation system," Decision Making: Applications in Management and Engineering, vol. 5, no. 1, pp. 208-224, 2022. [12] H. Feng and Y. Zhang, "Prediction of Hourly Airport Operational Throughput with a Multi-Branch Convolutional Neural Network," Aerospace, vol. 11, no. 1, p. 78, 2024. [13] W. Lin, Y. Low, Y. Chong, and C. Teo, "Integrated cyber physical simulation modelling environment for manufacturing 4.0," in 2018 IEEE 41 International Conference on Industrial Engineering and Engineering Management (IEEM), 2018: IEEE, pp. 1861-1865. [14] C. A. Authority, "CAP 716: Aviation Maintenance Human Factors (EASA/JAR145 Approved Organisations): Guidance Material on the UK CAA Interpretation of Part 145 Human Factors and Error Management Requirements," Hum. Factors, 2006. [15] T. A. Mattei, "Privacy, confidentiality, and security of health care information: Lessons from the recent Wannacry cyberattack," World neurosurgery, vol. 104, pp. 972-974, 2017. [16] Y. Creado and V. Ramteke, "Active cyber defence strategies and techniques for banks and financial institutions," Journal of Financial Crime, vol. 27, no. 3, pp. 771-780, 2020. [17] G. La Rocca, "Knowledge based engineering: Between AI and CAD. Review of a language based technology to support engineering design," Advanced engineering informatics, vol. 26, no. 2, pp. 159-179, 2012. [18] B. Khoshoo, J. Blank, T. Q. Pham, K. Deb, and S. N. Foster, "Optimal design of electric machine with efficient handling of constraints and surrogate assistance," Engineering Optimization, pp. 1-19, 2023. [19] J. Bautista-Hernández and M. Á. Martín-Prats, "Monte Carlo Simulation Applicable for Predictive Algorithm Analysis in Aerospace," in Doctoral Conference on Computing, Electrical and Industrial Systems, 2023: Springer, pp. 243-256. [20] D. Catteddu and G. Hogben, "ABOUT ENISA," Cloud Computing: Benefits, risks and recommendations for information security, 2009. Aerospace 2024,11, 300 6 of 21 The risk matrix outcomes split the dataset into three main groups. The dataset is grouped into high, medium, and low risk of creating an error during the manufacturing process. Histograms in Figure 3show how the scores are assigned to the parameters X1 , X2 , X3 , and X4 in the case study of C295 military aircraft. From these histograms, only a very small percent of 3.8% of the dataset had scores of 4 and 5, which could potentially generate errors. The ability to focus attention on specific data or components rather than treating the entire system uniformly improves effectiveness in the processes. Clustering data will help to identify the features and similarities between them for failure anticipation. The clusters suggest that certain combinations of parameters can occur together, indicating relationships between these variables [35]. Aerospace 2024, 11, x FOR PEER REVIEW 7 of 23 Figure 3. Histograms showing the quantity of harness part numbers on ‘y’ axis and the scores assigned on ‘x’ axis for each input parameter. Histogram (a) represents parameter X1; (b) parameter X2; (c) parameter X3; (d) parameter X4. Figure 4 shows the risk matrix function evaluated for each harness within the dataset and the probability associated with the error creation during the manufacturing engineering processes in a color code. Figure 4. Risk matrix associated with electrical harness in military aircraft with level of risk of error creation in each harness part number of the dataset. Red color depicts a very high/high risk; yellow depicts a medium risk, and green depicts a low/very low risk of error creation. 2.3. Clustering Hierarchy The main types of algorithms differ in the type of input/output data and scope of the problem that should be solved. Unsupervised algorithm techniques based on clustering are useful for application within the harness dataset. These techniques can be beneficial for identifying similarities and differences among harnesses, and therefore, for a more detailed analysis [27,35]. Figure 3. Histograms showing the quantity of harness part numbers on ‘y’ axis and the scores assigned on ‘x’ axis for each input parameter. Histogram (a) represents parameter X1 ; (b) parameter X2; (c) parameter X3; (d) parameter X4. Figure 4shows the risk matrix function evaluated for each harness within the dataset and the probability associated with the error creation during the manufacturing engineering processes in a color code. Aerospace 2024, 11, x FOR PEER REVIEW 7 of 23 Figure 3. Histograms showing the quantity of harness part numbers on ‘y’ axis and the scores assigned on ‘x’ axis for each input parameter. Histogram (a) represents parameter X1; (b) parameter X2; (c) parameter X3; (d) parameter X4. Figure 4 shows the risk matrix function evaluated for each harness within the dataset and the probability associated with the error creation during the manufacturing engineering processes in a color code. Figure 4. Risk matrix associated with electrical harness in military aircraft with level of risk of error creation in each harness part number of the dataset. Red color depicts a very high/high risk; yellow depicts a medium risk, and green depicts a low/very low risk of error creation. 2.3. Clustering Hierarchy The main types of algorithms differ in the type of input/output data and scope of the problem that should be solved. Unsupervised algorithm techniques based on clustering are useful for application within the harness dataset. These techniques can be beneficial for identifying similarities and differences among harnesses, and therefore, for a more detailed analysis [27,35]. Figure 4. Risk matrix associated with electrical harness in military aircraft with level of risk of error creation in each harness part number of the dataset. Red color depicts a very high/high risk; yellow depicts a medium risk, and green depicts a low/very low risk of error creation. Aerospace 2024,11, 300 7 of 21 2.3. Clustering Hierarchy The main types of algorithms differ in the type of input/output data and scope of the problem that should be solved. Unsupervised algorithm techniques based on clustering are useful for application within the harness dataset. These techniques can be beneficial for identifying similarities and differences among harnesses, and therefore, for a more detailed analysis [27,35]. The algorithm used in this study applies clustering to analyze and to classify datasets. Figure 5presents the clustering of data points defined by the risk matrix. Thus, ‘Formboard’ (FB) is represented by cluster 0 (in blue), ‘Technical Instructions’ (IT) by cluster 1 (in orange), ‘Manufacturing Bill of Material’ (MBOM) by cluster 2 (in green), and ‘Electrical Test’ (ET) by cluster 3 (in red). Aerospace 2024, 11, x FOR PEER REVIEW 8 of 23 The algorithm used in this study applies clustering to analyze and to classify datasets. Figure 5 presents the clustering of data points defined by the risk matrix. Thus, ‘Formboard’ (FB) is represented by cluster 0 (in blue), ‘Technical Instructions’ (IT) by cluster 1 (in orange), 'Manufacturing Bill of Material' (MBOM) by cluster 2 (in green), and 'Electrical Test' (ET) by cluster 3 (in red). Figure 5. Cluster’s structure computed using Python grouping the harness dataset in four clusters. The agglomerative clustering hierarchy uses input data defined as 𝐴𝐴1∪… ∪ 𝐴𝐴𝑗𝑗 for the set for each observation in each cluster, with the following constraints: 𝐴𝐴1∪… ∪ 𝐴𝐴𝑗𝑗 represents the number of clusters. 𝐴𝐴 𝑗𝑗 ∩ 𝐴𝐴 𝑗𝑗′ = ∅ no overlapping j ≠ j’. The initialization of the clusters and merging process is as follows: • Calculation of the proximity matrix, which represents the distances between each pair of data points based on the minimum distance: dmin =�∑(qi−pi)2 n i=1 , where pi and qi are coordinates of the data points. • Identify two clusters with minimum distance. • Merge the identified clusters into a new cluster combining the elements 𝐴𝐴1∪… ∪ 𝐴𝐴𝑗𝑗. • After merging two clusters, create a new cluster (𝐴𝐴𝑛𝑛𝑛𝑛𝑛𝑛) represented as 𝐴𝐴𝑛𝑛𝑛𝑛𝑛𝑛 = 𝐴𝐴1+ 𝐴𝐴𝑗𝑗, which contains elements of both clusters 𝐴𝐴1 and 𝐴𝐴𝑗𝑗. • Recalculate the distances between the new cluster and the remainder. • Update the proximity matrix with the new distances. • Continue this process iteratively. • Repeat until the stop criterion is met with all elements merged in a cluster. Figure 6 shows the merging process depicted as a dendrogram, which provides information about the critical clusters to be considered. Therefore, the dataset is grouped in families that give information about potential anomalies to be considered in order to anticipate error creation. Figure 5. Cluster’s structure computed using Python grouping the harness dataset in four clusters. The agglomerative clustering hierarchy uses input data defined as A1∪. . . ∪Aj for the set for each observation in each cluster, with the following constraints: A1∪. . . ∪Ajrepresents the number of clusters.Aj∩Aj′=no overlapping j =j′. The initialization of the clusters and merging process is as follows: • Calculation of the proximity matrix, which represents the distances between each pair of data points based on the minimum distance: dmin =qΣn i=1(qi−pi)2 , where p i and qiare coordinates of the data points. •Identify two clusters with minimum distance. • Merge the identified clusters into a new cluster combining the elements A1∪. . . ∪Aj . • After merging two clusters, create a new cluster ( Anew ) represented as Anew =A1+Aj , which contains elements of both clusters A1and Aj. •Recalculate the distances between the new cluster and the remainder. •Update the proximity matrix with the new distances. •Continue this process iteratively. •Repeat until the stop criterion is met with all elements merged in a cluster. Figure 6shows the merging process depicted as a dendrogram, which provides information about the critical clusters to be considered. Therefore, the dataset is grouped in families that give information about potential anomalies to be considered in order to anticipate error creation. Aerospace 2024,11, 300 8 of 21 Aerospace 2024, 11, x FOR PEER REVIEW 9 of 23 Figure 6. Harness dendrogram in the hierarchical clustering illustrates a group of similar data within each node. Hierarchical clustering establishes relationships between similar datasets grouped into clusters and labeled by letters from A to P. 2.4. Logistic Regression The best fit for the outcome categorical variable is defined by a logistic regression. Figure 7 represents the relationship between the probability of error creation and the risk matrix. The risk matrix function presents different values for each harness within the dataset. The probability function associated with the logistic regression is used for new data predictions. The logistic regression passes the input through the logistic/sigmoid function and then treats the outcome result as a probability [29]. Figure 7. Logistic regression dataset fits to a sigmoid function which predicts the probability of error creation. Figure 6. Harness dendrogram in the hierarchical clustering illustrates a group of similar data within each node. Hierarchical clustering establishes relationships between similar datasets grouped into clusters and labeled by letters from A to P. 2.4. Logistic Regression The best fit for the outcome categorical variable is defined by a logistic regression. Figure 7represents the relationship between the probability of error creation and the risk matrix. The risk matrix function presents different values for each harness within the dataset. The probability function associated with the logistic regression is used for new data predictions. The logistic regression passes the input through the logistic/sigmoid function and then treats the outcome result as a probability [29]. Aerospace 2024, 11, x FOR PEER REVIEW 9 of 23 Figure 6. Harness dendrogram in the hierarchical clustering illustrates a group of similar data within each node. Hierarchical clustering establishes relationships between similar datasets grouped into clusters and labeled by letters from A to P. 2.4. Logistic Regression The best fit for the outcome categorical variable is defined by a logistic regression. Figure 7 represents the relationship between the probability of error creation and the risk matrix. The risk matrix function presents different values for each harness within the dataset. The probability function associated with the logistic regression is used for new data predictions. The logistic regression passes the input through the logistic/sigmoid function and then treats the outcome result as a probability [29]. Figure 7. Logistic regression dataset fits to a sigmoid function which predicts the probability of error creation. Figure 7. Logistic regression dataset fits to a sigmoid function which predicts the probability of error creation. 2.5. Confusion Matrix The confusion matrix considers the scores for each harness based on the number of zones (Z), number of wires (H), and number of components (N). The most important metrics in (2–5) are obtained from the confusion matrix as follows: • Precision is defined as the ratio of correct positive predictions to the total number of positive predictions. This is the ratio between the real positive predictions and the total number of positives as the sum of true and false positives. Aerospace 2024,11, 300 9 of 21 • The recall or sensitivity ratio is defined as the true positive prediction of the total number of positives. It is the ratio between real positives and real cases indicated as true positives or false negatives. • Accuracy is defined as the ratio of the true prediction to the total value. It is the ratio of the total number of true positives and negatives in relation to the total number of real positives and real negatives within the dataset. • The F 1 score is the harmonic mean of the precision and recall used to measure test accuracy. It varies from 1.0, showing perfect values of precision and recall, to 0, where either precision or recall is zero. Precision =TP TP +FP (2) Recall or sensitivity =TP P=TP TP +FN (3) Accuracy =TP +TN P+N=TP +TN TP +TN +FP +FN (4) F1=2 1 Precision +1 Recall =2 TP 2 TP +FP +FN (5) In addition, the machine learning classification report provides some additional values as outcomes. Category 0 represents the negative class that means ‘no error creation’, and category 1 represents the positive class, meaning ‘error creation’. The macro-average is the average of the precision, recall, and F 1 -score. The weighted average is the weighted average of the precision, recall, and F 1 scores, as shown in Table 3. The performance metrics are depicted in Figure 8. Table 3. Classification reports of machine learning metrics such as precision, recall, and F 1 -score, with macro- and weighted average. Category Precision Recall F1-Score 0 1.00 0.95 0.98 1 0.91 1.00 0.95 Accuracy - - 0.97 Macro-average 0.95 0.98 0.96 Weighted average 0.97 0.97 0.97 Aerospace 2024, 11, x FOR PEER REVIEW 11 of 23 Figure 8. Performance metrics representation for precision, recall, and F1 score, with macro- and weighted average. The algorithm splits the dataset into two subsets: training set and test set. The training set was used to fit the model, and the test set was used to evaluate the performance of the model through the confusion matrix as shown in Figure 9. Figure 9. Confusion matrix outcome classification in true positives, true negatives, false positives, and false negatives. From the confusion matrix, the predictions show the good performance of the algorithm. True positives (TP): Prediction is a value that matches reality. The element at (1,1) represents the six data samples as true positives classified as ‘error creation’. True negatives (TN): The prediction value also matches reality. The element at (0,0) represents the 26 data samples as true negatives classified as ‘no error creation’. The diagonal elements Figure 8. Performance metrics representation for precision, recall, and F 1 score, with macro- and weighted average. Aerospace 2024,11, 300 10 of 21 The algorithm splits the dataset into two subsets: training set and test set. The training set was used to fit the model, and the test set was used to evaluate the performance of the model through the confusion matrix as shown in Figure 9. Aerospace 2024, 11, x FOR PEER REVIEW 11 of 23 Figure 8. Performance metrics representation for precision, recall, and F1 score, with macro- and weighted average. The algorithm splits the dataset into two subsets: training set and test set. The training set was used to fit the model, and the test set was used to evaluate the performance of the model through the confusion matrix as shown in Figure 9. Figure 9. Confusion matrix outcome classification in true positives, true negatives, false positives, and false negatives. From the confusion matrix, the predictions show the good performance of the algorithm. True positives (TP): Prediction is a value that matches reality. The element at (1,1) represents the six data samples as true positives classified as ‘error creation’. True negatives (TN): The prediction value also matches reality. The element at (0,0) represents the 26 data samples as true negatives classified as ‘no error creation’. The diagonal elements Figure 9. Confusion matrix outcome classification in true positives, true negatives, false positives, and false negatives. From the confusion matrix, the predictions show the good performance of the algorithm. True positives (TP): Prediction is a value that matches reality. The element at (1,1) represents the six data samples as true positives classified as ‘error creation’. True negatives (TN): The prediction value also matches reality. The element at (0,0) represents the 26 data samples as true negatives classified as ‘no error creation’. The diagonal elements represent the instances that are correctly classified. False positives (FP) and false negatives (FN) are off-diagonal elements representing the number of instances incorrectly classified. 3. Results and Discussion 3.1. Statistical Analysis 3.1.1. Time Savings The time savings are achieved through the automation of electrical harness operations, such as rotations, rolls, and straightens, and by the assistance provided during drawing creation, such as label designation, symbols, and manufacturing annotations. The operational time allocated for data preparation is a constant defined as ε in (6). It is calculated for the entire dataset in this study as 55.7 h for both general and automatic procedures. However, the operational time σ (t), used during documentation creation, is defined as a function that depends on time. In the general procedure, the necessary time to perform these operations is 223.6 h versus the 15.7 h established in the automatic procedure. Thus, a reduction of 93% in time was estimated. The time calculation values for both procedures are outlined in Table 4. T(t)=ε+σ(t)(6) The graph depicted in Figure 10 shows a comparison between the operational time in the general procedure and the automatic procedure using a bar representation to visualize the time savings achieved by the automation process. The time drops from 223.6 h to 15.7 h, shown as a blue bar in Figure 10, which generates a reduction of 93% in the time for the entire electrical dataset in this military aircraft. ε = 55.9 h shown as an orange bar in Figure 10 is the sum of all the constant operational times necessary to be performed using both procedures for the entire dataset. Aerospace 2024,11, 300 11 of 21 Table 4. Time calculation comparison between general procedure and automatic script. τ(t) Activity ε σ(t) Flattening Model Flattening Parameters Data Extraction Harness Flattening EHF Operations Drawing 2D Branches Arrangement Designations Labels Branches Rotation 2D Annotations Roll Symbols Straighten Other General Procedure 13.9 13.8 14.3 13.9 223.6 Automatic Script 13.9 13.8 14.3 13.9 15.7 Aerospace 2024, 11, x FOR PEER REVIEW 12 of 23 represent the instances that are correctly classified. False positives (FP) and false negatives (FN) are off-diagonal elements representing the number of instances incorrectly classified. 3. Results and Discussion 3.1. Statistical Analysis 3.1.1. Time Savings The time savings are achieved through the automation of electrical harness operations, such as rotations, rolls, and straightens, and by the assistance provided during drawing creation, such as label designation, symbols, and manufacturing annotations. The operational time allocated for data preparation is a constant defined as ε in (6). It is calculated for the entire dataset in this study as 55.7 h for both general and automatic procedures. However, the operational time σ(t), used during documentation creation, is defined as a function that depends on time. In the general procedure, the necessary time to perform these operations is 223.6 h versus the 15.7 h established in the automatic procedure. Thus, a reduction of 93% in time was estimated. The time calculation values for both procedures are outlined in Table 4. Τ (t)= ε+σ(t) (6 ) The graph depicted in Figure 10 shows a comparison between the operational time in the general procedure and the automatic procedure using a bar representation to visualize the time savings achieved by the automation process. The time drops from 223.6 h to 15.7 h, shown as a blue bar in Figure 10, which generates a reduction of 93% in the time for the entire electrical dataset in this military aircraft. ε = 55.9 h shown as an orange bar in Figure 10 is the sum of all the constant operational times necessary to be performed using both procedures for the entire dataset. Figure 10. Time comparison with and without automatic scripts in a bar representation aims to show the saving in time. Figure 10. Time comparison with and without automatic scripts in a bar representation aims to show the saving in time. 3.1.2. Sensitivity Analysis Sensitivity analysis is performed to analyze the influence of variations in the input parameters on the time prediction of the AI model developed in this study. This approach is used to evaluate the robustness and reliability of the model [ 22 ]. The evaluation is based on the variation in input parameters across two different scenarios, and the impact generated on the model outputs. The key parameters identified that are used as inputs are the number of wires (H), terminations (Z), and electrical components (N). The variation is established systematically over a range of values in each of the electrical harness datasets varying from ± 20% to ± 40%. After assessing parameter variation, the next step is to analyze how the changes affect the outputs. Figure 11 depicts the savings in time achieved on a general and automatic procedure from baseline to a range of input parameter variations of +20%, +40%, − 20%, and − 40%. The assessment of the output changes, with regard to the baseline of the total time of 279 h defined in the creation of the manufacturing documentation, establishes a variation from the maximum value of 501 h at a +40% level of sensitivity to the minimum of 171.5 h at − 40% in a general procedure. The automation procedure shows a baseline of 41.92 h and a variation from 75.15 h at a 40% level of sensitivity to 25.72 h at − 40%. The parameter values either increased or decreased after the sensitivity test was performed, showing consistency in the output changes. This situation provides a good estimation of the likelihood of the parameter predictions, since an increment in the percentage of the values means a greater number of wires, more components, more harness complexity, and therefore a higher risk matrix and higher probability of error creation. Consequently, the time estimation for both procedures increased following an escalation in the input parameters. Likewise, the situation presents the opposite effect whether or not Aerospace 2024,11, 300 12 of 21 the values of the parameters decrease. The consistent behavior produces reliable results and gives evidence of the robustness of the model. Therefore, sensitivity analysis enhances confidence in the predictions of the model. The quantitative measures show consistency in model behavior and are evidence to validate model reliability and robustness. Aerospace 2024, 11, x FOR PEER REVIEW 14 of 23 Figure 11. Model sensitivity analysis varying the input data parameters of +20%, +40%, −20%, and −40%. 3.1.3. Correlation Parameters The correlation parameter analysis is performed to measure the strength and direction of the relation between each parameter and the risk matrix. The coefficients indicate that the closer to one the absolute values of the coefficients are, the stronger the correlation is. Consequently, an increase in a parameter can lead to an increase in the risk matrix. Figure 12 quantifies the relationship between input parameters showing that the variables are highly positively correlated. The correlation of the variables was quantified by calculating the Spearman correlation coefficient, which evaluates the relationship between the input parameters of the dataset. The parameters that present the highest influence on the risk matrix are X3 and X4 at 0.98 and 0.97, respectively. The higher the correlation presented within the parameters is, the greater the likelihood of error creation during the manufacturing processes is. Thus, variations in these parameters directly affect the outcomes of the predictive model. The Spearman coefficient provides valuable insights and can inform about the decision-making process which is related to error prediction. The existing correlation between parameters X3 and X4 demonstrates the impact on the risk matrix function, which is fundamental to analyzing the probability of error creation and defining strategies for risk mitigation. The calculation of each coefficient parameter ‘r’ in Equation (10) is represented in the correlation matrix, whose values are calculated using Equations (7)–(9) for each pair of variables. Cov (x, y) = ��(Xi−μx)�Yi−μy�� /n −1 (7 ) σx = ���(Xi−μx)2�/n −1 (8 ) σy = ����Yi−μy�2�/n −1 (9 ) r = cov (x, y) σx σy (10 ) Figure 11. Model sensitivity analysis varying the input data parameters of +20%, +40%, − 20%, and −40%. 3.1.3. Correlation Parameters The correlation parameter analysis is performed to measure the strength and direction of the relation between each parameter and the risk matrix. The coefficients indicate that the closer to one the absolute values of the coefficients are, the stronger the correlation is. Consequently, an increase in a parameter can lead to an increase in the risk matrix. Figure 12 quantifies the relationship between input parameters showing that the variables are highly positively correlated. The correlation of the variables was quantified by calculating the Spearman correlation coefficient, which evaluates the relationship between the input parameters of the dataset. The parameters that present the highest influence on the risk matrix are X3 and X4 at 0.98 and 0.97, respectively. The higher the correlation presented within the parameters is, the greater the likelihood of error creation during the manufacturing processes is. Thus, variations in these parameters directly affect the outcomes of the predictive model. The Spearman coefficient provides valuable insights and can inform about the decision-making process which is related to error prediction. The existing correlation between parameters X3 and X4 demonstrates the impact on the risk matrix function, which is fundamental to analyzing the probability of error creation and defining strategies for risk mitigation. The calculation of each coefficient parameter ‘r’ in Equation (10) is represented in the correlation matrix, whose values are calculated using Equations (7)–(9) for each pair of variables. Cov(x, y)=∑h(Xi−µx)Yi−µyi/n −1 (7) σx=rh∑(Xi−µx)2i/n −1 (8) σy=s∑Yi−µy2/n −1 (9) r=cov(x, y) σxσy(10) Aerospace 2024,11, 300 13 of 21 Aerospace 2024, 11, x FOR PEER REVIEW 15 of 23 Figure 12. The correlation matrix establishes the relation between the input parameters and the output risk matrix. Figure 13 represents the highest positive correlation between X3 and X4 and the risk matrix. It is observed that low scores assigned to these parameters generate a moderate risk matrix with scores between 8 and 14, which is mostly the situation expected in the dataset present for this light and medium aircraft. There are only a few scores that strongly impact the risk matrix, pointing at 18 and 20. It also shows a positive linear tendency between both parameters and the risk matrix. In summary, X3 and X4 are the parameters that most affect the outcomes of the model. Therefore, these parameters must be treated as critical parameters. The interpretation of the correlation parameter impacts the effectiveness of risk management strategies. Correlation results are valuable in explaining the influence of the parameters on error prediction and risk mitigation. Understanding the relationships between variables can influence the decision-making process. Identifying variables, which are significant, can provide effective risk mitigation strategies. Analyzing potential risks or vulnerabilities from certain variables can help in taking preventive measures and avoiding errors. Figure 13. The graph represents the scores assigned to X3 and X4 and the correlation existing between them and the risk matrix. Red dots represent the risk matrix function within the dataset associated to the input parameters X3 and X4. Figure 12. The correlation matrix establishes the relation between the input parameters and the output risk matrix. Figure 13 represents the highest positive correlation between X3 and X4 and the risk matrix. It is observed that low scores assigned to these parameters generate a moderate risk matrix with scores between 8 and 14, which is mostly the situation expected in the dataset present for this light and medium aircraft. There are only a few scores that strongly impact the risk matrix, pointing at 18 and 20. It also shows a positive linear tendency between both parameters and the risk matrix. In summary, X3 and X4 are the parameters that most affect the outcomes of the model. Therefore, these parameters must be treated as critical parameters. The interpretation of the correlation parameter impacts the effectiveness of risk management strategies. Correlation results are valuable in explaining the influence of the parameters on error prediction and risk mitigation. Understanding the relationships between variables can influence the decision-making process. Identifying variables, which are significant, can provide effective risk mitigation strategies. Analyzing potential risks or vulnerabilities from certain variables can help in taking preventive measures and avoiding errors. Aerospace 2024, 11, x FOR PEER REVIEW 15 of 23 Figure 12. The correlation matrix establishes the relation between the input parameters and the output risk matrix. Figure 13 represents the highest positive correlation between X3 and X4 and the risk matrix. It is observed that low scores assigned to these parameters generate a moderate risk matrix with scores between 8 and 14, which is mostly the situation expected in the dataset present for this light and medium aircraft. There are only a few scores that strongly impact the risk matrix, pointing at 18 and 20. It also shows a positive linear tendency between both parameters and the risk matrix. In summary, X3 and X4 are the parameters that most affect the outcomes of the model. Therefore, these parameters must be treated as critical parameters. The interpretation of the correlation parameter impacts the effectiveness of risk management strategies. Correlation results are valuable in explaining the influence of the parameters on error prediction and risk mitigation. Understanding the relationships between variables can influence the decision-making process. Identifying variables, which are significant, can provide effective risk mitigation strategies. Analyzing potential risks or vulnerabilities from certain variables can help in taking preventive measures and avoiding errors. Figure 13. The graph represents the scores assigned to X3 and X4 and the correlation existing between them and the risk matrix. Red dots represent the risk matrix function within the dataset associated to the input parameters X3 and X4. Figure 13. The graph represents the scores assigned to X3 and X4 and the correlation existing between them and the risk matrix. Red dots represent the risk matrix function within the dataset associated to the input parameters X3and X4. 3.1.4. Error Prediction Error prediction analysis is performed to understand factors that can contribute to error creation. Figure 14 compares the distribution of the dataset to a theoretical distribution, such as a normal distribution. The theoretical quantile is compared against the sample Aerospace 2024,11, 300 14 of 21 quantiles to assess whether the data closely follow a normal distribution. The normal distribution dataset is represented by the tendency line. The greater the fit to the points through this line, the closer the fit is to a normal distribution. The dataset was divided into three groups: small, medium, and large harnesses. The first cluster shows a low impact in the risk matrix, and the dataset points present a risk matrix below 6, in the case of small harnesses. The second cluster shows a medium impact on the risk matrix between 8 and 10, for medium harnesses. The third cluster shows a high impact in the risk matrix between 11 and 20, for large harnesses. Aerospace 2024, 11, x FOR PEER REVIEW 16 of 23 3.1.4. Error Prediction Error prediction analysis is performed to understand factors that can contribute to error creation. Figure 14 compares the distribution of the dataset to a theoretical distribution, such as a normal distribution. The theoretical quantile is compared against the sample quantiles to assess whether the data closely follow a normal distribution. The normal distribution dataset is represented by the tendency line. The greater the fit to the points through this line, the closer the fit is to a normal distribution. The dataset was divided into three groups: small, medium, and large harnesses. The first cluster shows a low impact in the risk matrix, and the dataset points present a risk matrix below 6, in the case of small harnesses. The second cluster shows a medium impact on the risk matrix between 8 and 10, for medium harnesses. The third cluster shows a high impact in the risk matrix between 11 and 20, for large harnesses. Figure 14. Quantile–quantile plot (Q-Q plot) which compares the theoretical quartiles against the sample quartiles in a large, medium, and small harnesses. The comparison of the dataset with a normal distribution is used to make decisions about the process. After data representation, the distribution of the dataset in clusters, and the fitting to the theoretical standard normal distribution, the maximum deviation occurs at cluster 3 with 3.85 units from the center of the data distribution above the tendency line, as shown in Figure 14. The vertical red line evaluates how the dataset deviates from the expected theoretical behavior represented by the tendency line. Figure 15 represented by the box-plots show how close the distribution of a dataset is to an ideal distribution using quartiles. Figure 14. Quantile–quantile plot (Q-Q plot) which compares the theoretical quartiles against the sample quartiles in a large, medium, and small harnesses. The comparison of the dataset with a normal distribution is used to make decisions about the process. After data representation, the distribution of the dataset in clusters, and the fitting to the theoretical standard normal distribution, the maximum deviation occurs at cluster 3 with 3.85 units from the center of the data distribution above the tendency line, as shown in Figure 14. The vertical red line evaluates how the dataset deviates from the expected theoretical behavior represented by the tendency line. Figure 15 represented by the box-plots show how close the distribution of a dataset is to an ideal distribution using quartiles. Aerospace 2024, 11, x FOR PEER REVIEW 17 of 23 Figure 15. The first box-plot shows a dataset within a small harness group (cluster 1) where the value of the median equals Q1 and Q3. The second box-plot shows a positive skew of the dataset within the medium harness group (cluster 2). The third box-plot shows a negative skew of the dataset within the large harness group (cluster 3). In the first cluster, the first quartile Q1 and third quartile Q3 match together, showing the same risk matrix outcome from the data points within this group. In the second cluster, the median is represented at a risk matrix value of 9.0, showing a positive skew slightly deviated from the normal distribution. The median of the second cluster is 0.25 above the median of the ideal normal standard distribution. In the third cluster, the median is represented at a risk matrix value of 14, which shows a negative skew. The median of the third cluster is 1.5 below the median of the ideal normal standard distribution. Box-plot results show the data distribution which is crucial for predicting error creation and establishing risk mitigation strategies. Skewed data distributions may require special attention to ensure robust modeling and data prediction. Cluster 3 presents more skew data than cluster 2. The accuracy of the statistical analysis and predictions can be affected depending on the skewness due to loss of symmetry of the data distribution. The highest deviation occurs in cluster 3, generating the highest impact in the risk matrix and therefore in the evaluation of error creation with 3.85 units above the tendency line of a normal distribution. Additionally, for reliability purposes, the proposed methodology developed in this study was validated using a Monte Carlo simulation as an effective approach to assess the reliability of the model [36]. 3.2. Proposal Method Comparison with AI Existing Techniques This study used the logistic regression (LR) for predictions since it indicates good performance between precision and recall, as shown in Table 5. The efficiency of this method is proved by comparison with the existing techniques such as SVM, RF, KNN, and XGBoost. Table 5. Performance metrics comparison between proposed LR and other AI methods. Model Accuracy Precision Recall F 1 -Score SVM 0.98 1.00 0.85 0.92 RF 0.97 1.00 0.82 0.90 LR 0.97 1.00 0.95 0.98 KNN 0.95 0.76 1.00 0.86 XGBoost 0.96 1.00 0.80 0.88 Figure 15. The first box-plot shows a dataset within a small harness group (cluster 1) where the value of the median equals Q 1 and Q 3 . The second box-plot shows a positive skew of the dataset within the medium harness group (cluster 2). The third box-plot shows a negative skew of the dataset within the large harness group (cluster 3). Aerospace 2024,11, 300 15 of 21 In the first cluster, the first quartile Q 1 and third quartile Q 3 match together, showing the same risk matrix outcome from the data points within this group. In the second cluster, the median is represented at a risk matrix value of 9.0, showing a positive skew slightly deviated from the normal distribution. The median of the second cluster is 0.25 above the median of the ideal normal standard distribution. In the third cluster, the median is represented at a risk matrix value of 14, which shows a negative skew. The median of the third cluster is 1.5 below the median of the ideal normal standard distribution. Box-plot results show the data distribution which is crucial for predicting error creation and establishing risk mitigation strategies. Skewed data distributions may require special attention to ensure robust modeling and data prediction. Cluster 3 presents more skew data than cluster 2. The accuracy of the statistical analysis and predictions can be affected depending on the skewness due to loss of symmetry of the data distribution. The highest deviation occurs in cluster 3, generating the highest impact in the risk matrix and therefore in the evaluation of error creation with 3.85 units above the tendency line of a normal distribution. Additionally, for reliability purposes, the proposed methodology developed in this study was validated using a Monte Carlo simulation as an effective approach to assess the reliability of the model [36]. 3.2. Proposal Method Comparison with AI Existing Techniques This study used the logistic regression (LR) for predictions since it indicates good performance between precision and recall, as shown in Table 5. The efficiency of this method is proved by comparison with the existing techniques such as SVM, RF, KNN, and XGBoost. Table 5. Performance metrics comparison between proposed LR and other AI methods. Model Accuracy Precision Recall F1-Score SVM 0.98 1.00 0.85 0.92 RF 0.97 1.00 0.82 0.90 LR 0.97 1.00 0.95 0.98 KNN 0.95 0.76 1.00 0.86 XGBoost 0.96 1.00 0.80 0.88 Figure 16 represents the performance metrics such as accuracy, precision, recall, and F 1 -score obtained using different techniques. The highest accuracy is obtained using SVM. The highest precision is achieved using SVM, RF, LR, and XGBoost. The highest recall is obtained using KNN and the highest F1-score is obtained using LR. Aerospace 2024, 11, x FOR PEER REVIEW 18 of 23 Figure 16 represents the performance metrics such as accuracy, precision, recall, and F1-score obtained using different techniques. The highest accuracy is obtained using SVM. The highest precision is achieved using SVM, RF, LR, and XGBoost. The highest recall is obtained using KNN and the highest F1-score is obtained using LR. Figure 16. Performance metrics such as accuracy, precision, recall, and F1-score obtained using different AI techniques. The LR has shown very good performance in comparison to other proposed, existing techniques in terms of F1-score. It may not have the highest accuracy but its overall performance suggests that it can be the best choice. SVM may be a good choice since it has achieved the highest accuracy and precision; however, the recall is lower than LR. 3.3. Experimental Verification In order to verify the accuracy of the proposed methodology, we have compared the performance of AI methods using model data of this study and experimental data presented by ElDali and Kumar 2021 [37]. The experimental datasets used the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) and Prognostics and Health Monitoring (PHM08) generated by NASA Ames Research Center (Mountain View, CA, USA). The datasets contain features related to a total of 435 aircraft engines from sensor measurements. The sensor values were selected according to the metrics presenting the highest suitability to define failure patterns. The measurements were used for predictions of the remaining useful lifetime of the aircraft engines and their fault estimation. To verify the performance of our model, we have applied AI methods including SVM, RF, LR, KNN, and XGBoost to the measurement datasets of the sensors. Each AI algorithm is used to determine its effectiveness using technical data (TD) from input parameters of this study and measured data from the sensors (SM). The performance metrics are represented in Table 6. Table 6. Performance metrics comparison obtained using different AI methods between technical dataset (TD) and sensor measurement (SM). Metrics AI Methods Accuracy Precision Recall F 1 -Score TD SM TD SM TD SM TD SM SVM 0.98 0.95 1.00 1.00 0.85 0.88 0.92 0.94 RF 0.97 1.00 1.00 1.00 0.82 1.00 0.90 1.00 Figure 16. Performance metrics such as accuracy, precision, recall, and F 1 -score obtained using different AI techniques. Citation: Bautista-Hernández, J.; Martín-Prats, M.Á. The Impact of Data Injection on Predictive Algorithm Developed within Electrical Manufacturing Engineering in the Context of Aerospace Cybersecurity. Aerospace 2023,10, 984. https://doi.org/10.3390/ aerospace10120984 Academic Editor: Spiros Pantelakis Received: 18 October 2023 Revised: 6 November 2023 Accepted: 21 November 2023 Published: 23 November 2023 Copyright: © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/). aerospace Article The Impact of Data Injection on Predictive Algorithm Developed within Electrical Manufacturing Engineering in the Context of Aerospace Cybersecurity Jorge Bautista-Hernández 1,2,* and MaríaÁngeles Martín-Prats 1 1Department of Electronics Engineering, University of Seville, 41004 Seville, Spain; [email protected] 2Department of Electrical Engineering, Airbus Poland, 02-256 Warsaw, Poland *Correspondence: [email protected] Abstract: Cybersecurity plays a relevant role in the new digital age within the aerospace industry. Predictive algorithms are necessary to interconnect complex systems within the cyberspace. In this context, where security protocols do not apply, challenges to maintain data privacy and security arise for the organizations. Thus, the need for cybersecurity is required. The four main categories to classify threats are interruption, fabrication, modification, and interception. They all share a common thing, which is to soften the three pillars that cybersecurity needs to guarantee. These pillars are confidentiality, availability, and integrity of data (CIA). Data injection can contribute to this event by the creation of false indicators, which can lead to error creation during the manufacturing engineering processes. In this paper, the impact of data injection on the existing dataset used in manufacturing processes is described. The design model synchronizes the following mechanisms developed within machine learning techniques, which are the risk matrix indicator to assess the probability of producing an error, the dendrogram to cluster the dataset in groups with similarities, the logistic regression to predict the potential outcomes, and the confusion matrix to analyze the performance of the algorithm. The results presented in this study, which were carried out using a real dataset related to the electrical harnesses installed in a C295 military aircraft, estimate that injection of false data indicators increases the probability of creating an error by 24.22% based on the predicted outcomes required for the generation of the manufacturing processes. Overall, implementing cybersecurity measures and advanced methodologies to detect and prevent cyberattacks is necessary. Keywords: predictive algorithms; cybersecurity; machine learning; advanced persistent threats 1. Introduction The latest reports in 2022 from the European Union Agency for Cybersecurity (ENISA) show 586 reported cybersecurity incidents compared to 77 in 2012. Cyberattacks are increasing not only in frequency, but also in complexity, and are affecting organizations worldwide. Safety is an important pillar to protect the overall assets. Risk assessment procedures are likely to define the level of impact, the vulnerabilities, and the affected assets in order to minimize the risk and reach the highest level of safety [ 1 ]. This level of safety cannot be achieved in the new digital age since the security protocols are more vulnerable and data privacy can be easily exposed. On the other hand, predictive algorithms, which are developed to perform processes automatically and to reduce costs in organizations, are sensitive to threats such as, for example, data modifications [ 2 ]. Thus, data protection is essential in this context. Cybersecurity is needed to ensure the confidentiality, availability, and integrity of data resulting from access to untrusted sources. The aim of this study is to analyze the impact of data modification in order to observe the time increase in electrical harnesses manufacturing and error rate as hazard outcomes of the aerospace predictive algorithm after the dataset has been compromised. The predictive algorithm performance Aerospace 2023,10, 984. https://doi.org/10.3390/aerospace10120984 https://www.mdpi.com/journal/aerospace Aerospace 2023,10, 984 2 of 13 is also shown before and after the event has occurred. Thus, countermeasures to protect the dataset in the cybersecurity context are considered and applied for this purpose. Algorithms development aims to perform tasks faster, which are designed to enhance safety. Most of them are modeled to replace manual tasks by automation. In aerospace, there are different types of errors which can lead to accidents. Human errors can contribute up to 80% of the total errors [ 3 ]. Safety investigations about electrical wiring harness, conducted by authorities, conclude that some aircraft accidents were caused by failures within the electrical installation, resulting from improper design, maintenance, or a combination of them. These errors were mainly created from manual operations generated by humans during different stages of the engineering processes. Traditional methods, related to risk assessment, focus on the identification, analysis, and management of risks. However, they remain obsolete in the new digital world [ 4 , 5 ]. Thus, the use of predictive algorithms applied within the aerospace industry, which aims to prevent failures and decrease errors, is fundamental to maintain aerospace safety at the highest level [ 6 – 8 ]. Technologies, such as machine learning developed within the artificial intelligence, are key to enhance such algorithms [ 9 , 10 ]. Indeed, the new digital age requires more digital sources in order to interconnect processes to overcome the system complexity in the cyber–physical space environment. On the other hand, multiple users not only have access to the digital application, where algorithms are executed and outcomes are displayed, but also they are frequent users on a daily basis. Consequently, data can be compromised. Therefore, it is necessary to protect data using techniques within the framework of the cybersecurity. The four main categories to classify threat types are interruption, fabrication, modification, and interception in the cybersecurity context. They all aim to develop malicious content in a system. Complex techniques developed by attackers and malware are out of the scope of this paper [11,12]. The new digital environment requires a holistic approach that integrates more automation and system interconnection towards a better analysis of the error creation [ 13 ]. The development of predictive algorithms using machine learning techniques aims to connect the cyberspace environment, enable system automation, and prevent error creation during the manufacturing processes in order to maintain aerospace safety at the highest level [ 14 ]. Moreover, this environment presents vulnerabilities that affect data privacy. Thus, the need for cybersecurity is essential in this digital domain. Moreover, in the defense sector, the simulation for data injection could represent a significant risk to the digital assets. The rise of advanced persistent threats (APTs) which are a very highly sophisticated malware is evolving and aims to avoid security measures. Consequently, the attackers often send phishing emails until the first target system gets compromised. Once the malware has been deployed, other intrusive tools can enable the propagation to the internal network. Therefore, data extraction can be conducted, allowing the attackers to steal sensitive information. Thus, all the significant risks are essential for the organization to adopt cybersecurity measures [ 15 ]. Additionally, it is important to recognize potential breaches and analyze the consequences in order to strengthen the protection of the digital assets. Safety and cybersecurity in aerospace have become crucial priorities to be maintained in modern aircraft systems. The increasing systems interconnection and high dependency on software have enhanced the potential threats, leading cybersecurity to be a high priority in ensuring safety [ 16 ]. Vulnerabilities in software require focusing on its resilience in order to guarantee system security [ 17 ]. The implementation of advanced technology in modern aviation such as autopilot, engines control, air data communication, and electrical power is controlled by software. The flight control of the aircraft has evolved. Software is in charge of the code execution which generates the outputs to control the aircraft. The high level of security, which is required to integrate this system in aerospace, has established that up to 70% of the software development is dedicated to the robustness of the code [ 18 , 19 ]. This process includes the generation of rigorous tasks such as failure detection, isolation, synchronization, reconfiguration in a detected failure, and system control. In this context, Aerospace 2023,10, 984 3 of 13 any failure that occurs within data or code execution should not be propagated to the aircraft systems [18]. The design of any aviation system is implemented according to the high safety standards, which include guidelines on development, verification, validation, and configuration. Design assurance levels are also established to safeguard the aerospace assets [ 19 ]. All these approaches are aligned with the National Institute of Standards and Technology (NIST) within the cybersecurity framework, aiming to enhance security systems. Consequently, safety analysis is of paramount importance to ensure aircraft safety [20]. In this paper, the analysis was conducted after injecting malicious data, which has generated a negative impact on the performance of the predictive algorithms. The motivation to carry out this research was to analyze the impact of cyberattacks, to highlight its importance, and to understand the consequences generated on the engineering processes in the aerospace manufacturing plant. Thus, the performance of the algorithm, after the dataset was compromised, has been the main focus of this study. An assessment of the impact on time and error rate was carried out. Additionally, a detection strategy to detect a potential cyberattack has been set up to protect the algorithm. The hypotheses, which were considered, assumed that the attacker has information on the dataset of the victim and has succeeded to access the system. The following research questions are formulated as follows: •Can data injection stop the manufacturing of electrical harness? •Can this event be avoided by the application of proper cyber defense techniques? •Does the quantity of compromised data affect the efficiency of the algorithm? The remainder of this paper is as follows: Section 2presents the structure of the model algorithm developed and the necessity of the contribution of the cybersecurity to the cyber–physical space. Section 3takes an approach of the inconsistencies generated after data have been injected into the algorithm and shows an analysis of the consequences observed on the outcomes. Also, this Section includes the detection strategy to stop the algorithm operation after data have been injected. Section 4contains the conclusions and future work. 2. Materials and Methods The increasing interconnectivity and complexity of the systems requires the use of advanced technologies in cyberspace. The cyberspace environment highly relies on digital applications, which are based on innovative techniques developed within the context of artificial intelligence such as predictive algorithms [ 21 ]. Machine learning techniques are used to develop predictive algorithms, which play a relevant role to optimize manufacturing processes, mitigate errors, and enhance safety. At the same time, cyberspace is more vulnerable and exposed to cyberattacks in this new digital environment. Thus, the importance of cybersecurity has been raised as a main priority to maintain the integrity and security of the digital cyberspace [12]. 2.1. Predictive Algorithm The predictive algorithm applies input parameters within the real dataset presented in the ‘bill of material’ of each electrical harness in a military aircraft. This dataset is used for the creation of the processes necessary to manufacture the electrical harnesses. The experimental data in this case study are the electrical harnesses manufactured and installed in an aircraft. The initialization of the kernel is based on the data from the selected parameters, where the risk matrix function classified each harness and established a different level of probability of error creation during the manufacturing processes. After processing the data, the algorithm outputs show an evaluation of the likelihood of error creation and threats visualization represented by the following elements: • The automation script: The generation of the automatic processes will avoid human errors which are the main trigger of error creation [22]. Aerospace 2023,10, 984 4 of 13 • The dendrogram: The hierarchical representation of the dataset can help to establish relationships between different levels of risk by clustering the dataset in groups with similarities [23]. • The logistic regression: The statistical method can be used to model the probability of occurrence using the variables from the risk matrix and to predict the risk of error creation in new harnesses assessment [24]. • The confusion matrix: It is used to evaluate the performance of the classification model used for predictions through the logistic regression method. The true positives, true negatives, false positives, and false negatives provide insights into the performance of the logistic regression model in predicting risk events. Thus, it helps to not only define the performance of the algorithm, but also to detect inconsistences within the dataset. A large amount of true or false negatives can indicate the low performance of the algorithm [25]. Data generation was analyzed and processed through the following algorithm structure, as represented in Figure 1. Aerospace 2023, 10, x FOR PEER REVIEW 4 of 13 processing the data, the algorithm outputs show an evaluation of the likelihood of error creation and threats visualization represented by the following elements: • The automation script: The generation of the automatic processes will avoid human errors which are the main trigger of error creation [22]. • The dendrogram: The hierarchical representation of the dataset can help to establish relationships between different levels of risk by clustering the dataset in groups with similarities [23]. • The logistic regression: The statistical method can be used to model the probability of occurrence using the variables from the risk matrix and to predict the risk of error creation in new harnesses assessment [24]. • The confusion matrix: It is used to evaluate the performance of the classification model used for predictions through the logistic regression method. The true positives, true negatives, false positives, and false negatives provide insights into the performance of the logistic regression model in predicting risk events. Thus, it helps to not only define the performance of the algorithm, but also to detect inconsistences within the dataset. A large amount of true or false negatives can indicate the low performance of the algorithm [25]. Data generation was analyzed and processed through the following algorithm structure, as represented in Figure 1. Figure 1. Diagram representation of the structure of the predictive algorithm. 2.2. Risk Matrix The risk matrix is the mechanism based on the assessment of the electrical harnesses which depends on the following parameters. These input parameters are the number of zones (Z), number of wires (H), and number of electrical components (N) in each harness. The risk matrix function is represented in the following equation:  󰇛Z, H, N)=∑X=   X󰇛Z)+X 󰇛H)+X 󰇛N) + X󰇛H) (1) This function is defined with four parameters X, which are evaluated on a scale of 1 to 5, with 1 being the simplest geometry and 5 the most complex. These four parameters (X,X,X, X) depend on three other parameters, namely, the number of zones (Z), number of wires (H), and number of electrical components (N) present on each electrical harness. X represents the scores assigned due to the complexity of the 3D geometry. X, X, X are related to the scores assigned based on the electrical architecture, number Figure 1. Diagram representation of the structure of the predictive algorithm. 2.2. Risk Matrix The risk matrix is the mechanism based on the assessment of the electrical harnesses which depends on the following parameters. These input parameters are the number of zones (Z), number of wires (H), and number of electrical components (N) in each harness. The risk matrix function is represented in the following equation: Φ(Z,H,N)=∑4 i=1Xi=X1(Z)+X2(H)+X3(N)+X4(H)(1) This function is defined with four parameters Xi , which are evaluated on a scale of 1 to 5, with 1 being the simplest geometry and 5 the most complex. These four parameters ( X1 , X2 , X3 , X4) depend on three other parameters, namely, the number of zones (Z), number of wires (H), and number of electrical components (N) present on each electrical harness. X1 represents the scores assigned due to the complexity of the 3D geometry. X2 , X3 , X4 are related to the scores assigned based on the electrical architecture, number of electrical connections, and number of electrical components. They are used as input parameters to define the manufacturing processes of electrical harnesses [14]. 2.3. Cybersecurity Context Cyber defense techniques are crucial to protect systems, data, and networks from threats. The application of cyber defense techniques to the predictive algorithm ensures confidentiality, integrity, and availability of the sensitive data. The analysis of vulner- Aerospace 2023,10, 984 5 of 13 abilities, anomalies, and countermeasures is an essential strategy to defeat cyberattack scenarios [26]. In the cybersecurity context, the way to compromise data can be presented as follows: • Data modification: Malicious data can search for specific data within the dataset and modify them to achieve their goals. • Changing random values: To change data values randomly to cause confusion and make the data less reliable. • Data deletion: To delete certain information in order to cause significant problems, especially if the deletion of the data is critical to the business or customers. • Data reformatting: To change the data format in order to make it more difficult to use. •Insertion of false data: To falsify data into the dataset to deceive users who query it. In this study, the existing dataset which is used as input data to analyze the algorithm performance, has been compromised. The main consequences related to the error rate within the results after data injection have been analyzed. Data injection was carried out on the predictive algorithm, which is in charge of dataset analysis, patterns identification, and the outcomes forecast for decision making. It is hosted in the internal network within the infrastructure. The provided access is via security protocols which are updated regularly. No external access to the information system is allowed. The predictive algorithm interacts with data sources and means, which comply with security measures. However, the exposure of the system to the attack surface can allow unauthorized users to potentially exploit vulnerabilities and cause the damage [ 27 ]. Vulnerabilities within the code and unpatched software can expand the attack surface, amplifying the risk of data breaches. Indeed, a smaller attack surface mitigates the risk and makes the exploitation more difficult and the system more secure [ 28 ]. Despite security measures, social engineering attack surface such as phishing can contribute to the enlargement of the surface attack, highlighting the need for more security measures. Thus, the development of detection strategy can significantly reduce an organization from a potential cyberattack [29]. These types of data, which were injected in this study, were linked to the risk matrix function. The experiment was performed at the electrical harness department in the aerospace industry using a dataset related to 157 harnesses installed in a C295 military aircraft. The data were modified by selection of random values within the minimum and maximum of the scalar function risk matrix defined in Equation (1) for each electrical harness. The data affected before and after injection, represented by different levels of the risk matrix function, are presented in Table 1. One hundred and twenty-three harnesses have presented a low-risk matrix, 29 presented a moderate-risk matrix, and 5 of them have presented a high-risk matrix. After data injection, 147 showed a low-risk matrix, 10 showed a moderate-risk matrix, and none of them have presented a high-risk matrix. Overall, 38 (24.22%) harnesses have shown a different risk matrix function after data have been injected. Table 1. Data affected before and after data injection represented by different levels of the risk matrix function. Risk Matrix Before Data Injection After Data Injection Low 123 147 Moderate 29 10 High 5 - 3. Results The real dataset of 157 electrical harnesses, which were manufactured for this type of aircraft, has shown that only 3.18% of the harnesses have presented a high-risk matrix, 18.47% a medium-risk matrix, and most of the harnesses, 78.34%, have presented a lowrisk matrix of error creation during the manufacturing processes. However, after the dataset has been modified, 93.63% of the harnesses have shown a low-risk matrix, 6.36% a medium-risk Aerospace 2023,10, 984 6 of 13 matrix, and none of the harnesses have presented a high-risk matrix. The probability of error creation during the manufacturing processes has increased by 24.22% in comparison with the real dataset scenario. This situation threatens the safety in aerospace. Thus, countermeasures are necessary to be applied and to protect the dataset. Table 2shows the likelihood of error creation before and after data injection. Table 2. Likelihood of error creation associated with the risk matrix before and after data injection. Risk Matrix Real Data Injected Data High 3.18 0 Medium 18.47 6.36 Low 78.34 93.63 Figure 2depicts the impact of data injection on modifying the real dataset. This event has created a false behavior on the algorithm performance. The risk matrix has established different scores on each electrical harness, changing the probability of error creation. The blue line represents the real dataset associated with the real risk matrix for the experimental dataset and the orange line represents the variation on the risk matrix after data have been injected. This false behavior generated by the data injected is affecting the calculation of the manufacturing time. Aerospace 2023, 10, x FOR PEER REVIEW 6 of 13 Low 123 147 Moderate 29 10 High 5 - 3. Results The real dataset of 157 electrical harnesses, which were manufactured for this type of aircraft, has shown that only 3.18% of the harnesses have presented a high-risk matrix, 18.47% a medium-risk matrix, and most of the harnesses, 78.34%, have presented a lowrisk matrix of error creation during the manufacturing processes. However, after the dataset has been modified, 93.63% of the harnesses have shown a low-risk matrix, 6.36% a medium-risk matrix, and none of the harnesses have presented a high-risk matrix. The probability of error creation during the manufacturing processes has increased by 24.22% in comparison with the real dataset scenario. This situation threatens the safety in aerospace. Thus, countermeasures are necessary to be applied and to protect the dataset. Table 2 shows the likelihood of error creation before and after data injection. Table 2. Likelihood of error creation associated with the risk matrix before and after data injection. Risk Matrix Real Data Injected Data High 3.18 0 Medium 18.47 6.36 Low 78.34 93.63 Figure 2 depicts the impact of data injection on modifying the real dataset. This event has created a false behavior on the algorithm performance. The risk matrix has established different scores on each electrical harness, changing the probability of error creation. The blue line represents the real dataset associated with the real risk matrix for the experimental dataset and the orange line represents the variation on the risk matrix after data have been injected. This false behavior generated by the data injected is affecting the calculation of the manufacturing time. Figure 2. Risk matrix representation of dataset before (real data presented in blue color) and after injection (false data presented in orange color). Figure 3 shows that the total manufacturing time established for the entire real dataset has decreased from 239.3 h in the green bar to 182.5 h, as represented in the gray bar. The time has decreased by 23.73% after data injection, what has generated a false time calculation. This anomaly can be an indicator of a cyberattack. Figure 2. Risk matrix representation of dataset before (real data presented in blue color) and after injection (false data presented in orange color). Figure 3shows that the total manufacturing time established for the entire real dataset has decreased from 239.3 h in the green bar to 182.5 h, as represented in the gray bar. The time has decreased by 23.73% after data injection, what has generated a false time calculation. This anomaly can be an indicator of a cyberattack. The dendrogram is another indicator to analyze the impact of data injection. Figure 4 represents the clusters showing similarities between the dataset and grouping them into families. This situation allows for identifying patterns and improving the decision making. The number of clusters has been increased after data injection from one cluster to three. In the case of real dataset, it is possible to distinguish one family which needs to receive special attention. The groups which were created with similarities are clusters I, J, K and L, M, N, O, P. The rest of the dataset is compact within the main cluster. However, after data injection, the number of clusters has increased to three and the data have been dispersed. After dispersion of the dataset, the following groups are defined from A0to R0. Aerospace 2023,10, 984 7 of 13 Aerospace 2023, 10, x FOR PEER REVIEW 7 of 13 Figure 3. Manufacturing time representation of dataset before (real data presented in green color) and after injection (false data presented in gray color). The dendrogram is another indicator to analyze the impact of data injection. Figure 4 represents the clusters showing similarities between the dataset and grouping them into families. This situation allows for identifying patterns and improving the decision making. The number of clusters has been increased after data injection from one cluster to three. In the case of real dataset, it is possible to distinguish one family which needs to receive special attention. The groups which were created with similarities are clusters I, J, K and L, M, N, O, P. The rest of the dataset is compact within the main cluster. However, after data injection, the number of clusters has increased to three and the data have been dispersed. After dispersion of the dataset, the following groups are defined from A′ to R′. Figure 4. Dendrograms presenting groups of similar data in each node before injection (real data shown on the left) and after injection (false data shown on the right). The logistic regression shown in Figure 5 represents the best curve fitted to the dataset. The curve shows the probability of error creation (binary dependent variable) versus the risk matrix (independent variable). The blue curve is referred to the real data and the orange curve to the injected data. The probability function associated with the logistic regression is used for new predictions. The decision boundary with real and false data is different after the data have been injected. The risk matrix function presents different values in both situations. Therefore, the model will wrongly predict the new instances. Figure 3. Manufacturing time representation of dataset before (real data presented in green color) and after injection (false data presented in gray color). Aerospace 2023, 10, x FOR PEER REVIEW 7 of 13 Figure 3. Manufacturing time representation of dataset before (real data presented in green color) and after injection (false data presented in gray color). The dendrogram is another indicator to analyze the impact of data injection. Figure 4 represents the clusters showing similarities between the dataset and grouping them into families. This situation allows for identifying patterns and improving the decision making. The number of clusters has been increased after data injection from one cluster to three. In the case of real dataset, it is possible to distinguish one family which needs to receive special attention. The groups which were created with similarities are clusters I, J, K and L, M, N, O, P. The rest of the dataset is compact within the main cluster. However, after data injection, the number of clusters has increased to three and the data have been dispersed. After dispersion of the dataset, the following groups are defined from A′ to R′. Figure 4. Dendrograms presenting groups of similar data in each node before injection (real data shown on the left) and after injection (false data shown on the right). The logistic regression shown in Figure 5 represents the best curve fitted to the dataset. The curve shows the probability of error creation (binary dependent variable) versus the risk matrix (independent variable). The blue curve is referred to the real data and the orange curve to the injected data. The probability function associated with the logistic regression is used for new predictions. The decision boundary with real and false data is different after the data have been injected. The risk matrix function presents different values in both situations. Therefore, the model will wrongly predict the new instances. Figure 4. Dendrograms presenting groups of similar data in each node before injection (real data shown on the left) and after injection (false data shown on the right). The logistic regression shown in Figure 5represents the best curve fitted to the dataset. The curve shows the probability of error creation (binary dependent variable) versus the risk matrix (independent variable). The blue curve is referred to the real data and the orange curve to the injected data. The probability function associated with the logistic regression is used for new predictions. The decision boundary with real and false data is different after the data have been injected. The risk matrix function presents different values in both situations. Therefore, the model will wrongly predict the new instances. This situation negatively affects the performance of the algorithm. Thus, the algorithm has lost its reliability after data have been injected. Figure 6represents the confusion matrix showing the performance of the algorithm. The predictions are labeled as: true negatives (TN), true positives (TP), false negatives (FN), and false positives (FP). After running the simulation with the real dataset, the results are as follows: TN = 26, TP = 6, FN = FP = 0. However, after data have been injected, the results are: TN 0 = 1, TP 0 = 3, FN 0 = 0, FP 0 = 27; the increase in the number of false positives can be observed. This situation indicates a decrease in the performance of the predictive algorithm. Therefore, the classified data can be used as a good indicator for cyberattack detection. The confusion matrix collects all the threat indicators. This early detection mechanism enables a fast reaction and reduces the malicious risk [30]. Aerospace 2023,10, 984 8 of 13 Aerospace 2023, 10, x FOR PEER REVIEW 8 of 13 This situation negatively affects the performance of the algorithm. Thus, the algorithm has lost its reliability after data have been injected. Figure 5. Logistic regression curve before (real data shown on the left) and after injection (false data shown on the right) representing different fittings of the dataset in each situation. Figure 6 represents the confusion matrix showing the performance of the algorithm. The predictions are labeled as: true negatives (TN), true positives (TP), false negatives (FN), and false positives (FP). After running the simulation with the real dataset, the results are as follows: TN = 26, TP = 6, FN = FP = 0. However, after data have been injected, the results are: TN′ = 1, TP′ = 3, FN′ = 0, FP′ = 27; the increase in the number of false positives can be observed. This situation indicates a decrease in the performance of the predictive algorithm. Therefore, the classified data can be used as a good indicator for cyberattack detection. The confusion matrix collects all the threat indicators. This early detection mechanism enables a fast reaction and reduces the malicious risk [30]. The classifier has changed its performance from 32 correct predictions split between 26 correctly predicted as true negatives and 6 correctly predicted as true positives, while the algorithm was using correct data. However, after data have been injected, there are only four correct predictions split between one correct prediction as true negative and three correct predictions as true positives. There are 27 incorrect predictions as false positives, which involve a decrease in the performance metrics. Figure 6. Confusion matrix outcomes model before (real data shown on the left) and after injection (false data shown on the right). Table 3 shows the main outcomes of metrics defined for the algorithm. It shows the impact generated on the performance after data have been injected into the algorithm. Figure 5. Logistic regression curve before (real data shown on the left) and after injection (false data shown on the right) representing different fittings of the dataset in each situation. Aerospace 2023, 10, x FOR PEER REVIEW 8 of 13 This situation negatively affects the performance of the algorithm. Thus, the algorithm has lost its reliability after data have been injected. Figure 5. Logistic regression curve before (real data shown on the left) and after injection (false data shown on the right) representing different fittings of the dataset in each situation. Figure 6 represents the confusion matrix showing the performance of the algorithm. The predictions are labeled as: true negatives (TN), true positives (TP), false negatives (FN), and false positives (FP). After running the simulation with the real dataset, the results are as follows: TN = 26, TP = 6, FN = FP = 0. However, after data have been injected, the results are: TN′ = 1, TP′ = 3, FN′ = 0, FP′ = 27; the increase in the number of false positives can be observed. This situation indicates a decrease in the performance of the predictive algorithm. Therefore, the classified data can be used as a good indicator for cyberattack detection. The confusion matrix collects all the threat indicators. This early detection mechanism enables a fast reaction and reduces the malicious risk [30]. The classifier has changed its performance from 32 correct predictions split between 26 correctly predicted as true negatives and 6 correctly predicted as true positives, while the algorithm was using correct data. However, after data have been injected, there are only four correct predictions split between one correct prediction as true negative and three correct predictions as true positives. There are 27 incorrect predictions as false positives, which involve a decrease in the performance metrics. Figure 6. Confusion matrix outcomes model before (real data shown on the left) and after injection (false data shown on the right). Table 3 shows the main outcomes of metrics defined for the algorithm. It shows the impact generated on the performance after data have been injected into the algorithm. Figure 6. Confusion matrix outcomes model before (real data shown on the left) and after injection (false data shown on the right). The classifier has changed its performance from 32 correct predictions split between 26 correctly predicted as true negatives and 6 correctly predicted as true positives, while the algorithm was using correct data. However, after data have been injected, there are only four correct predictions split between one correct prediction as true negative and three correct predictions as true positives. There are 27 incorrect predictions as false positives, which involve a decrease in the performance metrics. Table 3shows the main outcomes of metrics defined for the algorithm. It shows the impact generated on the performance after data have been injected into the algorithm. Table 3. Metrics calculation to evaluate the algorithm performance before and after data injection. Metrics Comparison Data Real Data Injection Precision =TP TP+FP 1.0 0.1 Recall =TP TP+FN 1.0 1.0 Accuracy =TP+TN TP+TN+FP+FN 1.0 0.19 F1=2 TP 2 TP+FP+FN 1.0 0.13 The results obtained from the confusion matrix, which compare the metrics using real and false data, are depicted in Figure 7. Aerospace 2023,10, 984 9 of 13 Aerospace 2023, 10, x FOR PEER REVIEW 9 of 13 Table 3. Metrics calculation to evaluate the algorithm performance before and after data injection. Metrics Comparison Data Real Data Injection Precision = TP TP + FP 1.0 0.1 Recall = TP TP + F N 1.0 1.0 Accuracy = TP + TN TP + T N + FP + F N 1.0 0.19 F = 2 TP 2 TP + FP + F N 1.0 0.13 The results obtained from the confusion matrix, which compare the metrics using real and false data, are depicted in Figure 7. Figure 7. Comparison between accuracy, precision, recall, and F 1 score evaluation metrics between real data (presented in blue color) and false data (presented in orange color). Overall, data injection compromises the integrity principle based on confidentiality, availability, and integrity of the data [31]. As the attacker can modify the inputs, the detection mechanism is necessary to identify this scenario. This anomaly will affect not only the manufacturing time, but also the risk matrix function , which shows the probability of creating an error during the manufacturing processes. It is critical for the proper performance of the algorithm. Thus, security practises are essential to ensure safety and to protect the algorithm. The protective mechanism is based on the following detection strategy. The strategy is set up on anomaly detection and monitoring. The implemented mechanism aims to detect the unusual behavior of the risk matrix function defined in Equation (1) in order to identify security breaches. This function depends on four parameters shown in the explicit formula. Indeed, the risk matrix is equal to the sum of those parameters associated with each harness. The scalar function maintains this equality when the real data are used as input of the algorithm. Thus, this condition is fulfilled, allowing the code to continue its execution. However, if this condition is not fulfilled, the code execution will be stopped. Thus, the algorithm will not calculate any outcomes and will not show any false data. This detection strategy is represented in Figure 8. Figure 7. Comparison between accuracy, precision, recall, and F 1 score evaluation metrics between real data (presented in blue color) and false data (presented in orange color). Overall, data injection compromises the integrity principle based on confidentiality, availability, and integrity of the data [ 31 ]. As the attacker can modify the inputs, the detection mechanism is necessary to identify this scenario. This anomaly will affect not only the manufacturing time, but also the risk matrix function Φ , which shows the probability of creating an error during the manufacturing processes. It is critical for the proper performance of the algorithm. Thus, security practises are essential to ensure safety and to protect the algorithm. The protective mechanism is based on the following detection strategy. The strategy is set up on anomaly detection and monitoring. The implemented mechanism aims to detect the unusual behavior of the risk matrix function defined in Equation (1) in order to identify security breaches. This function depends on four parameters shown in the explicit formula. Indeed, the risk matrix is equal to the sum of those parameters associated with each harness. The scalar function maintains this equality when the real data are used as input of the algorithm. Thus, this condition is fulfilled, allowing the code to continue its execution. However, if this condition is not fulfilled, the code execution will be stopped. Thus, the algorithm will not calculate any outcomes and will not show any false data. This detection strategy is represented in Figure 8. The model was previously validated using Monte Carlo simulation, which is a method to assess the reliability of the engineering system. The simulation was carried out using the following input parameters: p1for wiring length, p2for number of electrical components, and p 3 for the protective sheath quantities present in the ‘bill of material’ on each harness. The Monte Carlo simulation determines the performance of the model for implementation purposes. The baseline run of Monte Carlo simulation was executed 1000 times across a range between maximum and minimum values of meters of wiring length (0.29–2374.79), units of number of components (4–2243), and meters of protective sheath (0.06–22.59) [ 32 ]. Aerospace 2023,10, 984 10 of 13 Aerospace 2023, 10, x FOR PEER REVIEW 10 of 13 Figure 8. Detection strategy to stop the algorithm computation after data have been successfully injected. The model was previously validated using Monte Carlo simulation, which is a method to assess the reliability of the engineering system. The simulation was carried out using the following input parameters: p1 for wiring length, p2 for number of electrical components, and p3 for the protective sheath quantities present in the ‘bill of material’ on each harness. The Monte Carlo simulation determines the performance of the model for implementation purposes. The baseline run of Monte Carlo simulation was executed 1000 times across a range between maximum and minimum values of meters of wiring length (0.29–2374.79), units of number of components (4–2243), and meters of protective sheath (0.06–22.59) [32]. The results of the Monte Carlo simulation were evaluated through the cumulative distribution function, which was defined for the three parameters selected. The results showed that the expected values were within a range of high probability of occurrence. After running the simulations, a new outcome was expected to have a wiring length of 2 m, 45 electrical components, and a protective sheath length between 1.49 and 4.48 m within a range of probabilities between 50 and 90%. These results showed that 123 out of Start Detection Strategy Is Φ = Σ (X i )? Φ(Z,H,N) =Σ(X i )=X 1 (Z)+X 2 (H)+X 3 (N)+X 4 (H) No Yes End Scores assigned to each harness X i : X 1 ,X 2 ,X 3 ,X 4 Data corrupted Continue processing Figure 8. Detection strategy to stop the algorithm computation after data have been successfully injected. The results of the Monte Carlo simulation were evaluated through the cumulative distribution function, which was defined for the three parameters selected. The results showed that the expected values were within a range of high probability of occurrence. After running the simulations, a new outcome was expected to have a wiring length of 2 m, 45 electrical components, and a protective sheath length between 1.49 and 4.48 m within a range of probabilities between 50 and 90%. These results showed that 123 out of 157 (78.34%) of the total electrical harnesses used in the simulation have presented the most common values, which are typical for this type of aircraft. Moreover, the 1000 runs provide enough evidence to consider the predictive algorithm as a reliable system. Conclusively, the predictive algorithm can be implemented as an auto-failure predictor [32–34]. 246 J. Bautista-Hernández and M. Á. Martín-Prats define the accuracy of the algorithm by table of contingency. Finally, the computation time will be the minimum necessary to define the optimal number of iterations using the method of gradient descent after regression logistic is applied to the categorical variable. Fig. 1. Diagram representation of the predictive algorithm and Monte Carlo simulation 3.2 The Input Parameters The first step in the creation of the predictive algorithm is to define the parameters. These are based on the electrical configuration of the harness to be manufactured andeventually installed on the aircraft. These are the number of zones (Z), number of wires (H) and number of electrical components (N) which define the risk matrix F. Such criteria defines each harness category based on these parameters. The risk matrix function defines the probability of creating an error during the manufacturing process of electrical harness as part of the outcomes from predictive algorithms. The risk matrix as a function can be expressed as follows: φ=φ(Z,H,N)(1) Scores assigned to each parameter will be evaluated on a scale from 1 to 5, being 1 the simplest geometry and 5 the most complex. A detailed description of the model has been recently proposed by Bautista Hernández and Martín Prats assessing its performance towards introduction in aerospace applications [6]. The baseline parameters for the Monte Carlo simulation will be based on the following relevant metrics, p1for wiring length, p2for number of electrical components and p3for the protective sheath quantities present in the ‘bill of material’ on each harness for a military aircraft. The Monte Carlo function determinates the performance of the model for implementation purposes. The generic function for Monte Carlo simulation λ, is expressed as follows: λ=λ(p1,p2,p3)(2) Monte Carlo Simulation Applicable for Predictive Algorithm 247 TheC295aircraft dataset with a total of 221 electrical harness to fully define the electrical system of this aircraft. In total in this case study, 157 harness considered are presenting the following parameters, 18523.95 m of the total wiring length, 21200.55 electrical components and 250.84 m of protective sheath length. The baseline run Monte Carlo simulation was executed 1000 times across a range between maximum and minimum values of meters of wiring length (0.29–2374.79), units of number of components (4– 2243) and meters of protective sheath (0.06–22.59). 3.3 Sensitivity Analysis The sensitivity analysis was carried out to understand the correlation between the input parameters and indicate the importance for the outcomes. Additionally, this analysis evaluates the model performance by considering the response of the input variables parameters after simulation [7]. The high correlation between the variables indicates that increasing of their values will enhance outcomes values. Thus, the probability of creating an error will be also higher [8]. The outcomes presented were analyzed to understand the impact of the maximum and minimum values within the dataset suggesting that the model performs well. In this medium-light size aircraft the predicted outcomes after 1000 runs were within the range of expected values. The box-plots performed for the chosen parameters represented in Fig. 2aim to analyze the data set and demonstrate the spread of numerical data through their quartiles follows one of the known distributions probabilistic. The Fig. 2shows the dataset distribution associated to each parameter defined by the generic function Monte Carlo in a logarithmic scale. In the first case related to the p1, the first quartile Q1=e0.9 =2.45 m marks one quarter (25%) of the ordered dataset and the value 1.5 IQR (-) below the first quartile is 1.5 IQR (-) =e0.1 =1.01 m. The maximum value in the dataset is 2374 m and the value for 1.5 IQR (+) =e 6.3 =544.21 m above the Q3=e3.2 =24.52 m, which marks three quarter (75%) of the ordered dataset. The maximum value is above the 1.5 IQR (+), so in this case the maximum is an outlier which may indicate the measurements are not in the center of the data. In the second case related to the p2, the first quartile Q1=e2.4 =11.02 components and the value 1.5 IQR (-) =e1.4 =4.05 components. The maximum value of the dataset is 2243 components. In this case, this maximum value is above the 1.5 IQR (+) =e6.9 =991 components showing outliers. In the third case related to the p3, the first quartile Q1=e0.1 =1.1 m and the 1.5 IQR (+) =e2.3 =9.97 m below the maximum value of this parameter in the dataset which is 22.59 m. The analytical results estimate the input data for the three parameters selected can be fitted to a normal distribution. 248 J. Bautista-Hernández and M. Á. Martín-Prats Fig. 2. Box-plots representation for input metrics p1,p 2,p 3on the Monte Carlo function λ 3.4 Monte Carlo Simulation The Monte Carlo simulation is a method to assess reliability of an engineering system. The simulation uses as input parameters the p1,p 2,p 3values. These values are randomly assigned 1000 times variating within the range of the maximum and minimum values defined from the entire dataset in a logarithmic scale shown in the Table 1. This simulation is used to obtain outcomes estimation from a set of stochastic trial from proper definition quantities [4]. Table 1. Minimum, maximum, mean and standard deviation values of the input parameters used in Monte Carlo simulation Description parameter min max mean std units Wiring length p10.1 7.77 2.46 1.90 m Number of components p21.38 7.71 3.66 1.36 – Protective sheath p30.1 3.11 0.44 0.74 m Such statistical distributions are used within the range of the lower and the higher end value for each parameter to estimate the outcomes across a Monte Carlo simulation with 1000 runs. Probabilistic distributions are considered for probability calculations within the dataset. The assignment for calculation of the probability in each individual run was carried out using the normal function from the random Python xlwings library within the given interval. As the input dataset variable X is log-normally distributed then the input values were transformed to a distribution Y =ln(X) for representation [9]. The simulation is defined by generation of a stochastic trial to each of the input parameters on each of 157 electrical harness on a C295 military aircraft following a probabilistic function normally distributed. After simulation using random combination of the input variables the output is calculated for each parameter. Finally, the outcomes are represented with the probabilistic function distribution and the cumulative function of density. 3.5 Method of Gradient Descent The regression logistic used to predict the categorical variable needs 144 iterations until the algorithm converges. This situation requires a high computation time. The method Monte Carlo Simulation Applicable for Predictive Algorithm 249 of gradient descent can be used to find the optimal solutions by proper adjustment of the parameters which minimize the function in order to reach the global minimum faster and the convergence of algorithm [10]. Being f :⊂Rn→R, this method can find θ⊂/f(θ)min.TheEqs.3,4and 5 define the best solution path in order to find the optimal solution. θ∈arg min f(θ)(3) θt+1=θt−δt∇f(θt)(4) ∇f(θt)=(∂f/∂x1(θ),...,∂f/∂xn(θ)) (5) δtis the learning rate which determinates the number of iterations until the algorithm converges. The convenient value of δguarantees minimize the number of iterations improving the computation time. The use of this method allows the algorithm to perform faster. Thus, the number of iterations decrease to 41 generating significantly benefit in the computation time. 4 Results The tendency of each parameter presents a linear relation between the parameters chosen for the Monte Carlo simulation (length, number of electrical components and protection sheaths). The correlation of these parameters is quantified by the correlation matrix shownintable2. A positive value of the coefficient means an increase of this parameter necessary involves an increase in the other parameter. The strongest correlation occurs when the absolute value of the coefficient is as close as possible to 1. From the correlation matrix we observed that p1and p3, as well as parameters p2with p3have lower positive correlation coefficients, but still indicate some degree of correlation between these variables. Table 2. Correlation matrix for the input parameters used for the Monte Carlo simulation Description Parameter Length Components Protection Wiring length p11.0000000 0.928046 0.470757 Number of components p20.928046 1.0000000 0.471076 Protective sheath p30.470757 0.471076 1.0000000 The Monte Carlo simulation has performed 1000 runs and the output parameters as a result of the new predicted values for the wiring length, numbers of electrical components and protective sheath are normally distributed. The cumulative distribution function shows the probability that each parameter value stays below specific threshold. 250 J. Bautista-Hernández and M. Á. Martín-Prats 4.1 First Output - Wiring Length The Eqs. (6–7) quantify through the probabilistic distribution function the prediction of the outputs in a new harness. As a first output of the new parameter for the wiring length, the p1will be between 2.01 and 3.00 m with a probability of occurrence between 50 and 90% (Fig. 3). P(0.5)→X1=e0.7=2.01 m (6) P(0.9)→X2=e1.1=3.00 m (7) Fig. 3. Results for p1outcomes on the wiring length prediction 4.2 Second Output – Number of Components The Eqs. (8–9) quantify through the probabilistic distribution function the prediction of the outputs in a new harness. As a second output of the new parameter for the number of components, the p2will stay between 45 and 245 units with a probability of occurrence between 50 and 90% (Fig. 4). P(0.5)→X1=e3.8=45units (8) P(0.9)→X2=e5.5=245units (9) 4.3 Third Output – Protective Textile Sheaths The Eqs. (8–9) quantify through the probabilistic distribution function the prediction of the outputs in a new harness. As a third output of the new parameter protective sheath Monte Carlo Simulation Applicable for Predictive Algorithm 251 Fig. 4. Results for p2outcomes on the number of electrical components prediction length, the p3in a new harness will be between 1.49 and 4.48 m with a probability of occurrence between 50 and 90% (Fig. 5). P(0.5)→X1=e0.4=1.49 m (10) P(0.9)→X2=e1.5=4.48 m (11) Fig. 5. Results for p3outcomes on the protective sheath prediction The reliability assessment at each parameter level is needed to calculate the probability that a new component does not exceed a limit defined through the cumulative distribution function [11]. The previous figures refer to the Monte Carlo simulation represent the random values of the parameters which are variating from maximum to the minimum within the three parameters. The distribution function shows the outcomes predicted are normally distributed. The y-axis shows the frequencies where a parameter appears and the x-axis the value of that parameter at that frequency. The cumulative distribution function estimates 252 J. Bautista-Hernández and M. Á. Martín-Prats the probability of the new output. The x-axis represents the value of the parameter and the y-axis represents the probability that the event occurs. The following assumptions considering those parameters p1,p 2,p 3are normally distributed/∼N(μ,σ)arerepresentedintheFig.6,showingthecalculationofexpected outcomes lead to a Gaussian function Eq. 12. (x)=1 σ√2π e−1 2x−μ σ2 (12) Table 3. Normal distributions fitted to the outcomes obtained after the Monte Carlo simulation has been carried out Description Parameter Normal distribution Wiring length p1N (2.46, 1.89) Number of components p2N (3.66, 1.36) Protective sheath p3N (0.61, 0.74) The purpose of the normal distributions is to obtain a probability function to help to understand the variability of the system simulated and to give an evidence of its reliability. In the future, they can also be useful to make predictions of the parameters. These considerations in terms of modelling validation are necessary to be taken into account. The auto failure-predictor prioritizes to determinate accurate predictions from a given dataset parameters for model validation [8]. The Fig. 6shows these parameters are normally distributed, the x-axis corresponds to thevalueofthisparameter and they-axisthedensity function ofprobabilityshowinghow likely is to observe that value. Each colour represents the different Gaussian distributions for each of those parameters selected for the Monte Carlo simulation. Fig. 6. Normal or Gaussians distribution for the dataset parameters p1,p 2,p 3 Monte Carlo Simulation Applicable for Predictive Algorithm 253 The Fig. 7depicts the dataset within the time and frequency domain. The series of points in left figure are referred to the parameters of the model and show the signal changes within time. The blue line represents the first parameter p1, which is the wiring length. The red line represents the second parameter p2, which is the number of electrical components and the third parameter p3, represents by the orange line is the protection sheaths. The ‘y axis’ refers to the amplitude of the signal what is the scale of each point of the dataset. The ‘x axis’ refers to each point of the dataset on a single harness out of 157. The information obtained from wiring length and number of components show that the harness with more wiring length will also present more number of electrical components, so these parameters are positively correlated. It is also observed they have similar amplitude between peaks. In harness with longer wiring length it is shown that the blue line is above the red line showing that in this group of harness the amplitude of this parameter is greater than the number of components. This type of harness are the most critical harness since the present high risk matrix. From the orange line, the protective sheath is the unpredictable parameter, in some points the dataset has ‘zero values’ what it means there are harness which are not in presence of this parameter. This parameter is also in phase with the other two parameters, what it means, if the parameter value is greater than 0, the increasing of the wiring length and number of electrical components will also reflect an increasing of the protective sheath parameter. The use of this representation helps to predict future values based on the previous history data. Fig. 7. Time/frequency domain representation for the dataset parameters p1,p 2,p 3 From the representation in the frequency domain, the right Fig. 7shows how much the amplitude of the signal is present in each frequency interval. The times series analysis and the observed values are represented in a data spectrum. The information from this representation includes in detail the phase difference between the signals. The wiring length and number of components are in phase. There is a minimal difference in phase 254 J. Bautista-Hernández and M. Á. Martín-Prats at low frequency (f =0.05 Hz). At this frequency the wiring length decreases, however the number of electrical components increases. In this specific situation it is expected that the number of electrical components should increase as well. This situation occurs in avionics harness where the number of components is in percentage very closed to the wiring length percentage. There are many electronic devices to be connected in a very small area. At f =0.35 Hz is another particular case where the signals are not in phase and there is a difference between the phases on both signals. This situation illustrates only one electrical harness presents more components than wiring length. The protective sheaths parameter is presented in a constant rate and the information obtained shows is an unpredicted parameter, concluding that this parameter should not be considered as a relevant parameter for validation of the model. 5 Discussion For the purpose to validate the system, the Monte Carlo simulation is a proper method to analyse how the performance is affected by the parameters. The input parameters are stochastically assigned for each dataset. The random behaviour of the components aim to define sensitivities indexes to quantify the outcomes and inform about the importance of variation in the features of the system. From the predictive results, it is shown that potentially a new harness within this aircraft is expecting to have greater than 2 m wiring length, 45 components and 1.49 m of protective sheath. This event occurs with a probability higher than 50%. The expected values in summary are shown in the following Table 3. Table 4. Expected values of the input parameters and probability of occurrence after Monte Carlo simulation Description Expected value Wiring length 2.01 m Number of components 45 units Protective sheath 1.49 m The impact of these components and their metrics contribute to assess the risk of the system to fail. Failures in aerospace are not only costly and time-consuming but they can involve catastrophic consequences [9]. The outcomes are in accordance with the expected values and within the range for this aircraft. The correct behaviour of the outcomes is relevant to consider the system as a reliable and a safe system. This represents the best model prediction (Table 4). From the risk matrix, this type of harness is defined as a medium-low risk what it means there is low probability to create an error during the creation of manufacturing process. This situation shows very reliable results since mostly harness in this medium light aircraft are medium size presenting not a high risk in terms of error creation. This Monte Carlo Simulation Applicable for Predictive Algorithm 255 information provided in advanced generate a lot of benefits, such as better performance on the end-to-end process shortening the potential error creation between stages. Additionally, the whole process system gets more reliable providing a full visibility of the entire process. The aging effect on these parameters is also important to take into consideration. The wiring length aging can influence to the wiring properties and make them shorter. The number of components can be also reduced due to material wear decreasing components efficiency. The protection sheaths aging can increase the risk of short circuit after damaging the wires. Additionally, the accuracy of the model is ensured by calculation of the minimum square error. The results show a value of 0.13 what implies that the model is predicting well the outcomes for the given input parameters. The expected values obtained from the cumulative function of density in Table 3are also good metrics to ensure accuracy of the modelling parameters. 6 Conclusions Safety is a main priority and necessary to be established between all main actors involved in the end-to-end process. The positive communication enhances not only safety but also commitment with high requirements in order to anticipate before the undesirable situations occur. Starting from this research developed within aerospace the aim is to definethe sensitivityrespect three main inputsparameters which are the mostrelevantfor electrical harness in order to define the model as reliable to be used as a reliable model. The importance of these parameters defines effectiveness for the risk matrix calculation and sensitivity analysis. In this paper we presented evidences that the model used for prediction of errors creation in a military aircraft case study is a reliable auto-failure detector. The model has used the following relevant parameters in the electrical harness definitionsuchaswiringlength,numberofelectricalcomponentsandlengthofprotective sheaths. For the sensitivity analysis the sample size used n =1000 for these factors were modelled in a Monte Carlo simulation. After analysis, results show these parameters are highly correlated, showing that the wiring length and number of components are proportional and reliable parameters to be used to define the risk matrix of the model. Additionally, we observed the avionics harness are critical presenting both parameters in very similar proportion. The protection sheath shows an unpredictable behaviour and to be used as an estimator of the risk matrix needs to be consider together with another parameter, either wiring length or number or components. The dataset has been validated using real data manufactured by one of the main aircrafts making this system as reliable model and safe. This research also establishes future trends applicability to keep safety at the highest level, not only in aerospace but also in other disciplines. This is the first milestone in order to be implemented in the industry. The results show that the model can be implemented as an auto-failure predictor and can be used to make realistic estimation of these parameters. Acknowledgments. The authors acknowledge the contributions to the automatic and electronic engineering department from the Seville University and to Airbus Poland for supporting this work with their research facilities and resources.