Information Fusion of Infrared Images and Acoustic Signals for Bearing Fault Diagnosis of Rotating Machinery

HAL Id: hal-04165899 https://hal.science/hal-04165899v1 Submitted on 19 Jul 2023 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés. Information Fusion of Infrared Images and Acoustic Signals for Bearing Fault Diagnosis of Rotating Machinery Mohammad Siami, Hamid Shiri, Tomasz Barszcz, Jacek Wodecki, Radoslaw Zimroz To cite this version: Mohammad Siami, Hamid Shiri, Tomasz Barszcz, Jacek Wodecki, Radoslaw Zimroz. Information Fusion of Infrared Images and Acoustic Signals for Bearing Fault Diagnosis of Rotating Machinery. Surveillance, Vibrations, Shock and Noise, Institut Supérieur de l’Aéronautique et de l’Espace [ISAESUPAERO], Jul 2023, Toulouse, France. �hal-04165899� 1 Fusion of Infrared Images and Acoustic Signals for Bearing Fault Diagnosis of Rotating Machinery Mohammad Siami1*, Hamid Shiri2, Tomasz Barszcz3, Jacek Wodecki2, Radoslaw Zimroz2 1AMC Vibro Sp. z o.o., Pilotow 2e, 31-462 Kraków, Poland 2Department of Mining, Faculty of Geoengineering, Mining and Geology, Wrocław University of Science and Technology, 50-370 Wroclaw, Poland 3Department of Robotics and Mechatronics, AGH University of Science and Technology, Al. Mickiewicza 30, 30-059 Kraków, Poland [email protected]* Abstract To minimize the operation and maintenance costs of rotating machinery, damaging conditions should be detected in the early stages. In recent years, infrared (IR) imaging and acoustic-based condition monitoring methods have gained attention for this purpose. To solve the problems when one single type of data, either acrostic signals or IR images, cannot be individually used to assess the severity of faults in rotating machinery, in this paper we proposed an information fusion method to improve the utilization of multisource sensor systems. Fusing the extracted information from both sources is performed with the aid of a fuzzy inference system. The effectiveness of the proposed techniques is demonstrated through the data that has been captured by an inspection mobile robot for monitoring conveyor belt idlers at a mining site. 1 Introduction The conventional condition monitoring (CM) methods used for rotating machinery mainly include vibration analysis, electrical signals, temperature, encoder signals, and so on. However, due to the nature of the harsh operating environments, the installation of CM sensors can be rather complicated or costly. Laser doppler vibrometers, high-speed cameras, or fiber-optic sensors are expensive non-contact instruments that cannot be utilized for the CM of large-scale rotating machines. On the other hand, IR images and acoustic signals can be captured with non-contact instruments that are relatively less expensive and can be implemented on mobile robots for performing inspection tasks in hard-to-reach environments for inspectors[1]–[6]. IR thermography is categorized as a non-contact and non-instructive CM technique that can be used to identify thermal anomalies by analyzing the emitted IR radiation from objects. As most common bearing faults in rotating machines lead to unwanted friction between internal components of the bearings, it can cause heat with a specific temperature distribution. Different researchers made contributions to the application of IR imaging methods for the CM of rotating machinery bearings. Another effective non-contact measurement for bearing fault detection is the acoustic signal. The captured acoustic signals in real-world scenarios are commonly mixed with high-energy noises. Therefore, the acquired acoustic signals, need to be analyzed to identify fault frequency signatures[7]– [14]. Although single-sensor measurements are complementary for the CM of rotating machines, to accomplish more complex tasks, multi-sensor configurations are becoming increasingly important. The processing of the gathered data in a multi-sensor configuration is considered an extremely complex task, as the availability of heterogeneous data leads to the necessity of developing integration and fusion CM methods that are compatible with the studied data[7], [15]–[17]. In this paper, the multi-sensor information fusion method is proposed to realize conveyor belt idler diagnosis using IR images and acoustic signals. The information fusion method for processing the multi-sensor CM systems can be categorized into three different groups, including the data-level, feature-level, and decision-level fusion methods. In datalevel fusion, or, in other words, sensor-level fusion, the obtained raw data from each sensors are fused to be represented as a single data unit without going through any feature extraction. In feature-level fusion, the extracted features from each sensor are individually processed and integrated into a feature vector 2 using a certain approach. In decision-level fusion, the corresponding data from each sensor is separately processed, and the obtained results that represent the monitored equipment are separately stored. Afterward, the final decision is shaped based on the fusion of results acquired from each sensor using certain methods such as Dempster-Shafer (D-S) evidence theory or fuzzy decision theory[18]. The feature-level fusion methods are more suitable to perform data fusion for sensors of the same type; however, when they are used to fuse acquired data from sensors with different dimensions, the fault information cannot be easily identified due to the different characteristics and distribution of the studied features. In our case study, the vibration signal can be considered one-dimensional waveform data, while the IR image is two-dimensional image data. Therefore, decision-level fusion is used to fuse the results to improve the overall understanding of the health state of the studied idlers. The specific goals of this article are threefold. (1) We proposed an image processing pipeline based on the attention residual U-Net (AResU-Net) for semantic segmentation of thermal faults in conveyor belt idlers. (2) A statistical method has been used to analyze the acoustic signal for fault detection of conveyer belts (3) A decision-level information fusion based on the fuzzy inference system is proposed to take advantage of infrared images and acoustic signals to improve the overall performance of the fault detection system. 2 Semantic segmentation of IR images based on AResU-Net Since 2012, applications of deep learning (DL) methods for performing different image processing tasks have received huge attention. AResU-Net is an inspired convolutional neural network that was specially developed for performing semantic segmentation tasks in image processing applications Fig 1. This architecture takes advantage of attention mechanisms and residual blocks in comparison to the base UNet architecture that was first proposed in 2016[19]. In this architecture, the attention blocks are organized as skip connections to enable the network to pay only attention to desirable regions. Moreover, to improve the depth of the network, residual blocks are employed to reduce the chance of gradient vanishing[20]–[26]. Figure 1: AResU-Net architecture The acquired IR images have been processed and undergo normalization process to improve the segmentation results[18] The employed AResU-Net have been trained based on the data set of IR images captured by inspection mobile robot in three different experiment. Through the first two experiments the inspection mobile robot moves forward (FW) and backward (BW) alongside the inspected conveyor system number one. In the third experiment the mobile robot has been moved forward through alongside the conveyor number two and captured IR images from same point of view. The number of extracted frames is discussed in table 1. 3 Conveyor 1 (FW) Conveyor 1 (BW) Conveyor 2 (FW) Total number of extracted frames 6135 6275 10897 Percentage of positive cases 5.21% 6.67% 5.74% Percentage of negative cases 94.78% 93.32% 94.26% Table 1. Comparison of positive and negative samples extracted from captured IR videos In order to defined the degree of severity in each sample the segmented areas in each sample is measured and considered as the main feature for identification of the overheated idlers. 3 Acoustic-based diagnostics of belt conveyor idlers Firstly the captured raw acoustic signals by the mobile robot are loaded. Cyclostationary is a beneficial method for fault identification in industrial rotating machinery. The cyclic modulations in captured acoustic signals can be determined via analysis of a bi-frequency map called Cyclic Spectral Coherence (CSC). Therefore, In order to analyze, the complex acoustic signal cyclostationary approach is applied for accurate detection of faults, and bi-frequency map are computed for each sample see Fig.2 . In this figure, the Cyclic Spectral Coherence (CSC) is plotted for three different cases. (a) (b) (c) Figure 2: Raw, spectrogram, and Cyclic Spectral Coherence (CSC) for three different cases (a) Healthy cases (b) Belt connection cases (c) Fault cases 4 Afterward, several stochastic features see Table 1, were calculated for each frequency band of extracted bi-frequency maps, and a summation of extracted features are calculated. Feature dimension reduction is an essential step in the processing of acoustic signals. The demission reduction techniques help avoid challenges due to computational cost and complexity of the following classification task. To this end, principal component analysis (PCA) was used to reduce the dimensionality of the extruded features. In the next step, the PC1 is selected as the main feature for following procedure based on the fact that PC1 has more information in comparison to PCA components. In this step, by using the PC1, the idlers can be categorized into two main groups namely: healthy cases and faulty cases include belt connections cases. It should be noted that the belt surface is never a single consistent loop. Therefore, It should be manufactured as a linear strip, which is then installed as a loop by making a connection between ends. In some cases, the source of non-cyclic impulses that are not related to faulty idlers are intense noises produced by a metal joint connecting two pieces of the belt. The moving metal joint is heating the idlers and producing much stronger noises than a typical interaction between the belt and idlers. Due to this fact, actual faulty signatures usually appear on coefficients of idler speed, while other harmonic patterns related to metal join occur on the low-frequency band; In this step the frequency features were used to separate faulty and joint belt cases. However, in a real mining environment, many noise sources affect acoustic signals and cause the inefficiency of frequency methods. To address this issue, the local mode decomposition (LMD) is used to denoising the signal before extracting frequency features. Later, the envelope analysis is applied to the rest of the cases that filter by LMD. Finally, the sum of the first three frequencies with high amplitude on envelope frequency is calculated to separate real faulty cases from other cases[8], [27]. Mean Max Root mean square Standard deviation Variance Skewness Kurtosis Shape factor Impulse factor Crest factor Moment 3 Moment 4 Moment 5 Moment 6 Energy Median Table 2: Studied features 4 Fuzzy-Based Decision Fusion In this study, a decision-level fusion strategy was applied to the acquired results from the processed IR images and acoustic signals using a fuzzy inference system. The purpose of the proposed model is to produce an improved estimation of machine health from a set of independent data sources. The fuzzy nature can be recognized in many machine faults; therefore, a fuzzy inference system can be considered a suitable method for data fusion and fault diagnosis proposes. Fuzzy analysis has been proven to be an effective method for handling different failure modes in machines. It precisely classifies the fault pattern, like the way a human supervisor processes vague information[28], [29]. In this work, to assess the risk of idler failure, Equation 1 is defined, where risk (R) is defined by the intersection of consequences of the extracted T (thermal features) and A (acoustic features). 𝑅=A×𝑇 (1) In fuzzy system models, the relation between the input and output parameters is defined by linguistic labels (fuzzy sets) using IF-THEN rules. Every fuzzy inference model consisted of four different levels: fuzzification, rules, inference engine, and defuzzification Figure 3: 5 Figure 3: Overview of Fuzzy Inference System In the first step, the fuzzifier module convert the crisp values into fuzzy values. Through the second step, the defined rules based on the expert knowledge are employed to construct databases. The following form is the most common way to represent expert knowledge: IF 𝑋1 is 𝐴1 and … and 𝑋𝑛 is 𝐴𝑛 THEN 𝑌 is 𝐵, (2) In Equation 3, 𝑋j and 𝑌 indicate the corresponding linguistic variables to input and output respectively while, input and output linguistic variables, respectively, 𝐴j and 𝐵 refer to the linguistic variables with corresponding linguistic terms. In the third step, the combined fuzzy rules are applied on input data to shape the fuzzy output. Finally in the last step through defuzzification process the output value is shaped by weighting and averaging all of the fuzzy rules. 4.1 Fuzzification and rules To contract the FIS, the inputs (sensor decisions) need to be partitioned based on their domain in several fuzzy sets. The details of fuzzy sets that were applied to the first step of FIS is presented in Table 3. Table 3. Definition of fuzzy and crisp ratings In this paper, Gaussian type of membership function was employed to design the FIS. The Gaussian membership function can be defined base on the two parameters by Equation 3 where center and width of the membership function can be defined by the c and 𝝈 parameters. Risk factors Linguistic term Interpretation Crisp rating Fuzzy rating IR images (T) Very high (VH) High (H) Medium (M) The chance of thermal defects is very high The chance of thermal defects is high The chance of thermal defects is medium 5 4 3 2360.8≤T<2897 1824.6≤T<2360.8 1288.4≤T<1824.6 Low (L) The chance of thermal defects is low 2 752.2≤T<1288.4 Very low (VL) The chance of thermal defects is very low 1 216≤T≤752 Acoustic signal (A) Very high (VH) High (H) Medium (M) Low (L) Very low (VL) There is a credible sign of bearing faults in proceed signal There is a relatively credible sign of bearing faults in proceed signal There is a likely sign of bearing faults in proceed signal There is a low evidences sign of bearing faults in proceed signal There is a no credible sign of bearing faults in proceed signal 5 4 3 2 1 75≤A<90 60≤A<75 45≤A<60 30≤A<45 15≤A≤30 Risk categoty (R) High (H) Medium (M) Low (L) Immediate action is required Not acceptable without review by supervisor No action is needed 3 2 1 10≤R<15 5≤R<10 0≤R≤5 6 Gaussian (𝑥;𝑐,𝜎)=exp (−1 2(𝑥−𝑐 𝜎)2) (3) Different approaches can be used to represent the knowledge to form database however in this paper, the Mamdani’ s method employed for this purpose. The IF-THEN rules were used to make a map between the input and output parameters (R, T and A). For this purpose, 125 IF-THEN rules have been designed to provide a database Table 4. IF Scenario occurrence probability is P1 and Class of attention is E, THEN Ri level is Low. IR images T1 (VL) L L L M H T2 (L) M M M H H T3 (M) H H H H H T4 (H) H H H H H T5 (VH) H H H H H A1 (VL) A2 (L) A3 (M) A4 (H) A5 (VH) Acoustic signals Table 4. Definition of rules 4.2 Defuzzification To transform the fuzzy outputs into a scalar output quantity, an union operator needs to be employed to converting the fuzzy values into crisp values. One of the most used defuzzification method is Center of Gravity (COG) which can be defined as follow: 𝑥∗=∫𝜇𝑖(𝑥)𝑥𝑑𝑥 ∫𝜇𝑖(𝑥)𝑑𝑥 (4) In Equation 4, 𝑥∗is the defuzzified output where 𝜇𝑖(𝑥) and x represent the the aggregated membership function and output variable respectively. 5 Experiments and Results In this section, we provide an example to demonstrate the potential applications of the proposed model in the information fusion of an acquired IR image and an acoustic signal in a real-world scenario. Through the examinations conducted, an inspection mobile robot has been used for condition monitoring of conveyor belt idlers. The IR images and acoustic signals have been acquired from 50 different idlers. To identify the fault severity in each idler, the acquired results from both sources are fused based on a fuzzy inference system. The consistency of the rules is shown in Fig. 4 by displaying the dependency of the output as a function of the inputs. 7 Figure 4: Normalized surface view of the proposed fuzzy inference system Fig 4 demonstrates that the constructed FIS can make a more realistic evaluation of the input parameters (T and A) by supplying gradual transitions in risk scores. The lowest, bright red part of the plot represents the resultant low level of risk R resulting from the low scenario from the fusion of T and A parameters. This situation can reflect a situation where the risk of sudden failure in the idler is low and does not require an immediate decision to change the inspected module. 6 Conclusion A fusion methodology for conveyor belt idler fault diagnosis using decision fusion of IR images and an acoustic signal has been presented. Experiment tests on a rotor test stand are performed to validate the effectiveness and accuracy of the method. It is demonstrated that in fuzzy fusion methods, the input and output information can be described in linguistic terms, which are more realistic and flexible in real-world situations like condition monitoring. It is worth mentioning that with the rapid development of condition monitoring methods, the idea of decision-level data fusion in this paper can be developed based on the different deep learning networks for fusing different kinds of data with different dimensions. Funding This work was supported by the European Commission via the Marie Sklodowska Curie program through the ETN MOIRA project (GA 955681) — Mohammad Siami and Hamid Shiri. This activity has received funding from the European Institute of Innovation and Technology (EIT), a body of the European Union, under the Horizon 2020, the EU Framework Program for Research and Innovation. This work is supported by EIT RawMaterials GmbH under Framework Partnership Agreement No. 19018 (AMICOS. Autonomous Monitoring and Control System for Mining Plants). Scientific work published within the framework of an international project co-financed from the funds of the program of the Minister of Science and Higher Education titled “ PMW ” 2020–2021; contract no. 5163/KAVA/2020/2021/2. Acknowledgment The authors (Mohammad Siami and Hamid Shiri) gratefully acknowledge the European Commission for its support of the Marie Sklodowska Curie program through the ETN MOIRA project (GA 955681) 8 References [1] R. Zimroz Radosław and Król, “Failure analysis of belt conveyor systems for condition monitoring purposes,” Mining Science, vol. 128, no. 36, pp. 255–270, 2009, [Online]. Available: http://www.miningscience.pwr.edu.pl/Failure-analysis-of-belt-conveyor-systemsfor-condition-monitoring-purposes,59825,0,2.html [2] A. Grzesiek, R. Zimroz, P. Śliwiński, N. Gomolla, and A. Wyłomańska, “Long term belt conveyor gearbox temperature data analysis – Statistical tests for anomaly detection,” Measurement (Lond), vol. 165, 2020, doi: 10.1016/j.measurement.2020.108124. [3] T. Kozłowski, J. Wodecki, R. Zimroz, R. Błazej, and M. Hardygóra, “A diagnostics of conveyor belt splices,” Applied Sciences (Switzerland), vol. 10, no. 18, 2020, doi: 10.3390/APP10186259. [4] F. Alharbi et al., “A Brief Review of Acoustic and Vibration Signal-Based Fault Detection for Belt Conveyor Idlers Using Machine Learning Models,” Sensors, vol. 23, no. 4, 2023, doi: 10.3390/s23041902. [5] Y. Liu, C. Miao, X. Li, J. Ji, and D. Meng, “Research on the fault analysis method of belt conveyor idlers based on sound and thermal infrared image features,” Measurement, vol. 186, p. 110177, 2021, doi: https://doi.org/10.1016/j.measurement.2021.110177. [6] C. Mai et al., “Laser Sensing and Vision Sensing Smart Blind Cane: A Review,” Sensors, vol. 23, no. 2, 2023, doi: 10.3390/s23020869. [7] M. Siami, P. Trybała, T. Barszcz, and R. Zimroz, “A Sensor Fusion System with Thermal Infrared Camera and Lidar for Automatic Detection and Localization of Overheated Idlers on Conveyor Systems,” May 2022. [8] M. Siami, H. Shiri, T. Barszcz, and R. Zimroz, “Unsupervised Learning Based Data-Driven Anomaly Detection for Acoustic Monitoring of Idlers in Conveyor Systems,” May 2022. [9] M. Siami, T. Barszcz, R. Zimroz, and J. Wodecki, “Robot-based Damage Assessment Method for Identification of Overheated Idlers in Conveyor Systems Using Histogram Analysis Techniques,” May 2022. [10] T. Barszcz, M. Siami, J. Wodecki, and R. Zimroz, “Automated IR Image Segmentation for Identification of Overheated Idlers in Belt Conveyor Systems Based on Outlier-Detection Method,” SSRN Electronic Journal, May 2022, doi: 10.2139/ssrn.4054247. [11] M. Siami, T. Barszcz, J. Wodecki, and R. Zimroz, “Automated Identification of Overheated Belt Conveyor Idlers in Thermal Images with Complex Backgrounds Using Binary Classification with CNN,” Sensors, vol. 22, no. 24, 2022, doi: 10.3390/s222410004. [12] J. Wodecki, H. Shiri, M. Siami, and R. Zimroz, “Acoustic-based diagnostics of belt conveyor idlers in real-life mining conditions by mobile inspection robot,” Conference on Noise and Vibration Engineering, ISMA 2022, 2022. [13] M. Siami, T. Barszcz, J. Wodecki, and R. Zimroz, “Design of an Infrared Image Processing Pipeline for Robotic Inspection of Conveyor Systems in Opencast Mining Sites,” Energies (Basel), vol. 15, no. 18, 2022, doi: 10.3390/en15186771. [14] B. Griffith, D. Türler, and H. Goudey, “IR Thermographic Systems: A Review of IR Imagers and Their Use,” Lawrence Berkeley National Laboratory: Berkeley, CA, USA, 2001. [15] H. B. Mitchell, Image fusion: theories, techniques and applications. Springer Science & Business Media, 2010. [16] K. Miao, J. Ma, Z. Li, Y. Zhao, and W. Zhu, “Research on multi feature fusion perception technology of mine fire based on inspection robot,” in Journal of Physics: Conference Series, 2021, p. 12064. doi: 10.1088/1742-6596/1955/1/012064. [17] J. J. Zhang, Z. Y. Ye, and K. F. Li, “Multi-sensor information fusion detection system for fire robot through back propagation neural network,” PLoS One, vol. 15, no. 7 July, pp. 1–13, 2020, doi: 10.1371/journal.pone.0236482. [18] Y. Chandola, J. Virmani, H. S. Bhadauria, and P. Kumar, “Chapter 4 - End-to-end pre-trained CNN-based computer-aided classification system design for chest radiographs,” in Deep