Condition level deteriorations modeling of RC beam bridges with U-Net convolutional neural networks

Structures Condition level deteriorations modeling of RC beam bridges with U-Net convolutional neural networks --Manuscript Draft-- Manuscript Number: STRUCTURES-D-22-00902R1 Article Type: Research Paper Keywords: condition assessment; regional bridges; inspection reports; deterioration modeling; deep learning; nondestructive evaluation Corresponding Author: Ye Xia, Ph.D. Tongji University Shanghai, CHINA First Author: Xiaoming LEI Order of Authors: Xiaoming LEI Ye XIA, Ph.D. Seyedmilad KOMARIZADEHASL Limin SUN Abstract: Reinforced concrete (RC) beam bridges have suffered structural deterioration due to loads, environmental conditions, etc. Regular visual inspections of bridges effectively monitor the structural condition level and provide a vast amount of condition-related data for years. This study proposes a deep learning-based condition level deterioration modeling method with a U-Net model to improve the prediction accuracy of future structural conditions. The proposed method is supported by the data gathered from the years of regional bridge inspection reports. Before training the model, the regional condition-related features regarding the influence of bridge ages and the superstructure types are investigated, and the correlations between selected features and structural conditions are also revealed. The acquired inspection database validated the high prediction accuracy and classification performance of each bridge's main part and system with the proposed deterioration modeling method. Its robustness is tested under a variety of data missing rate scenarios. The optimum model architecture and its effectiveness are also validated through comparative studies. This study provides a novel method to predict the future structural condition with inspection data and could serve as a reference for more reasonable utilization of the bridge condition deterioration model in structural condition assessment and management. Response to Reviewers: Please see the attached files. Powered by Editorial Manager® and ProduXion Manager® from Aries Systems Corporation - 1 - Condition level deteriorations modeling of RC beam 1 bridges with U-Net convolutional neural networks 2 Xiaoming Lei1, Ye Xia2, *, Seyedmilad Komarizadehasl3, Limin Sun4 3 1. Department of Bridge Engineering, Tongji University, Shanghai, China; (E-mail: 4 [email protected]) 5 2 Department of Bridge Engineering, Tongji University, Shanghai, China; Shanghai Qizhi Institute, 6 Shanghai, China; (E-mail: y[email protected]) 7 3 Department of Civil & Environmental Engineering, Polytechnic University of Catalonia, 8 BarcelonaTech, Spain; (E-mail: seyedmilad.komarizade[email protected]) 9 4 State Key Laboratory of Disaster Reduction in Civil Engineering, Tongji University, Shanghai, 10 China; Department of Bridge Engineering, Tongji University, Shanghai, China; Shanghai Qizhi 11 Institute, Shanghai, China; (E-mail: [email protected]du.cn) 12 * Corresponding Author 13 Abstract: Reinforced concrete (RC) beam bridges have suffered structural deterioration due to loads, 14 environmental conditions, etc. Regular visual inspections of bridges effectively monitor the structural 15 condition level and provide a vast amount of condition-related data for years. This study proposes a 16 deep learning-based condition level deterioration modeling method with a U-Net model to improve 17 the prediction accuracy of future structural conditions. The proposed method is supported by the data 18 gathered from the years of regional bridge inspection reports. Before training the model, the regional 19 condition-related features regarding the influence of bridge ages and the superstructure types are 20 investigated, and the correlations between selected features and structural conditions are also revealed. 21 The acquired inspection database validated the high prediction accuracy and classification 22 performance of each bridge's main part and system with the proposed deterioration modeling method. 23 Its robustness is tested under a variety of data missing rate scenarios. The optimum model architecture 24 and its effectiveness are also validated through comparative studies. This study provides a novel 25 method to predict the future structural condition with inspection data and could serve as a reference 26 for more reasonable utilization of the bridge condition deterioration model in structural condition 27 assessment and management. 28 Keywords: condition assessment; regional bridges; inspection reports; deterioration modeling; deep 29 learning; nondestructive evaluation 30 1. Introduction 31 In recent decades, the vast expansion of civil infrastructure has aided economic development and 32 social advancement [1, 2]. According to a survey report provided by China's Ministry of Transport in 33 2020, the country has 912,800 bridges totaling 66,285,534 meters. Quite a few bridges are classified 34 as having minor damage or worse, which become a potential safety risk to the serviceability of 35 transportation networks [3, 4]. Similar situations have also been reported in other nations [5, 6]. 36 Simultaneously, the vast amount of inspection and monitoring data accumulated over the years on 37 Manuscript Click here to view linked References 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 2 - regional bridges contains several information on structural performance and evolution patterns. 38 Consequently, this study aims to build a reliable data-driven method to reveal bridge condition 39 deteriorations. 40 Existing reinforced concrete (RC) beam bridges will undoubtedly develop cracks, honeycombs, 41 pockmarked surfaces, water erosion, and protective layer shedding due to loads, structural defects, 42 material qualities, environmental conditions, and other reasons [7, 8]. These defects or slight damages 43 may cause technical condition deteriorations or lead to huge safety issues [9, 10]. In addition, most of 44 the defects start from the structural surfaces and develop to the inside of the structure. 45 Bridge regular inspection is an effective nondestructive evaluation (NDE) way to master the 46 structural conditions of regional bridges [11, 12]. There is a standardized visual inspection and 47 assessment process for the bridge inspection to ensure the validity and accuracy of the results [13, 14]. 48 Before the bridge inspection, combining with the bridge history files and bridge damage inspection 49 and assessment form, the agencies will develop a reasonable and feasible plan to carry out the on-site 50 inspection. Then, based on the inspection results of each bridge component, the inspectors will rate 51 each bridge component, unit, superstructure, substructure, deck, and system [15, 16]. Finally, the 52 recorded paper files and electronic files are archived to provide a data basis for assessing and 53 predicting the condition of regional bridges. 54 Several governments and regions worldwide have created their own bridge datasets to hold 55 structural inspection data [17, 18]. The National Bridge Inventory (NBI) is a standard database for 56 assessing the condition level of American bridges. It contains statistics from state departments of 57 transportation gathered by the Federal Highway Administration since 1968. Some studies have 58 utilized the NBI database to build the bridge deterioration models and management systems that might 59 help stakeholders make better judgments. Bridge inspection data in China is recorded in the local 60 bridge management administration. Some of them lack effective management to reveal the regional 61 structural deterioration features. The traditional passive management strategy normally performs 62 measurement when substantial deterioration is observed. Active management techniques based on 63 past measured data are gaining popularity in recent years. 64 Traditional methods for modeling the deterioration of bridge conditions are usually classified as 65 analytical and probabilistic approaches [19-21]. The former approach primarily employs physical 66 deterioration mechanisms of reinforcement bar and concrete cement to determine the residual 67 structural capacities. The development of this approach is constrained due to the complexity of the 68 damage process and the substantial uncertainties associated with stochastic deterioration behavior 69 [22]. In probabilistic deterioration models, unobserved factors, measurable mistakes, and intrinsic 70 uncertainties might be represented in the deterioration process, which introduces the stochastic 71 process to mimic structural deterioration [23, 24]. The Markov decision process (MDP) is a discrete72 time stochastic process that is commonly used to model structural deteriorations. The distinctive 73 feature of a Markov chain is that the potential future states are fixed, regardless of how the process 74 got to its current state. It might not be a reasonable assumption to make when assessing the structural 75 condition. 76 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 3 - Artificial neural networks (ANNs) have been widely used in various structural safety 77 applications, including damage detection and condition prediction [25, 26]. They can handle large 78 amounts of data and allow for direct usage of input data without the need for hand-crafted feature 79 extraction. In the research, the research and operational indicators are used to qualitatively and/or 80 quantitatively evaluate the condition of bridges, Ivankovic et al. [27] reviewed and compared several 81 indicators in the bridge management. Huang [28] used ANNs to examine historical measurement data 82 and related maintenance operations on bridge decks, achieving a condition classification accuracy of 83 75%. Li and Burgueno [29] examined multiple ANNs for identifying bridge abutment conditions, 84 finding that the trained models had a prediction accuracy of above 70%. Bukhsh et al. [30] added the 85 feature concatenation in the multi-task ANNs model, and its performance outperforms other machine 86 learning algorithms. The ANNs could understand the degradation of structural conditions by 87 collecting important aspects from past measured data, as these findings indicate [31, 32]. Ariza et al. 88 [33] compared the performance of different models (ANNs, MDP, hidden Markov model (HMM), 89 and Semi-MDP) in forecasting the bridge deck performance. The HMM model better predicted the 90 deck ratings than MDP and Semi-MDP, but the ANNs model achieved the best prediction 91 performance. 92 Deep learning (DL) algorithms can extract deep relevance and intrinsic characteristics from large 93 volumes of data [34]. Convolutional neural networks (CNNs) are a type of deep learning architecture 94 that can extract multi-level features from a large number of inputs effectively and automatically [35]. 95 Image classification, object recognition, semantic segmentation, and instance segmentation are just a 96 few of the data-driven and image-related civil engineering domains where CNNs have been widely 97 used. In the field of structural assessment, Liu and Zhang [36] used the NBI database to choose over 98 twenty characteristics and train CNNs for three bridge main parts: superstructure, substructure, and 99 deck. Fiorillo and Nassif [37] utilized CNNs to find relationships between structural elements and 100 NBI ratings, then forecasted future element deterioration patterns. Zhu and Wang [38] combined 101 CNNs and recurrent neural networks (RNNs) to estimate the future condition levels of bridges in the 102 next three to four years using historical bridge data for the last three decades in Texas, USA. Liu et at. 103 [39] employed a CNN-based model with the maximum likelihood estimation of parameters in a 104 Markov chain. A 26-year forecast obtained a robust and low prediction error, with the greatest mean105 squared error near 0.5. Although the application of CNNs in bridge condition level prediction achieves 106 promising results, it may encounter issues of gradient vanishing and incomplete feature extractions. 107 Recently, the U-Net model has been widely applied in the field of civil engineering, providing 108 promising methods to interpret structural safety issues. A CrackU-Net was created by Ju et al. [40] to 109 extract accurate fracture information from pavement photos. For pixelwise crack identification, the 110 CrackU-net surpassed both CNNs and fully convolutional networks (FCNs). Yang et al. [41] 111 improved the computer vision-based crack width identification precision using the U-Net model, and 112 achieved a 0.01-mm precision under a 96% guarantee rate. Dong et al. [42] focused on the pixel-level 113 fatigue crack segmentation from large-scale images with U-Net architecture. Fewer training epochs 114 and a simpler model structure improve crack segmentation performance. Li et al. [43] incorporated 115 the CNNs with the classic U-Net model to construct an end-to-end crack image segmentation 116 framework. Most previous research employed the U-Net model to extract accurate damage 117 information (cracks, etc.) from visual inspection images. Few studies have established the relationship 118 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 4 - between structural condition levels and structural features with the help of the feature extraction of 119 U-Net. 120 This study proposes an innovative bridge condition level deterioration modeling method with 121 the U-Net model, enabling a more precise prediction of future structural conditions than the other 122 methods. The method is validated with the data acquired from the years of inspection reports of 123 regional bridges. Based on the acquired inspection data, its regional condition-related features 124 regarding the influence of bridge ages and the superstructure types are also investigated, and the 125 correlations between selected features and structural conditions are revealed. Corresponding results 126 could provide a reference for more reasonable utilization of the bridge condition deterioration model 127 in structural condition assessment and management than its conventional usage. 128 The remainder of this study is organized as follows: Section 2 introduces the overall U-Net 129 model-based structural condition level deterioration modeling framework. Section 3 depicts the 130 condition-related database of regional bridges in detail, as well as the extracted condition level-related 131 elements from years of inspection reports. The high prediction accuracy and classification 132 performance of each bridge main part and system with the proposed U-Net deterioration modeling 133 method are validated with in Section 4. The optimum U-Net architecture and its effectiveness are also 134 validated through comparative studies. Some conclusions and limitations of this study are illustrated 135 in Section 5. 136 2. Frameworks with the U-Net model 137 2.1 Frameworks 138 Since regional bridges are located in similar operating environments, their structural condition 139 degradation trends should be correlated. For example, the bridges on the same highway are designed 140 and constructed by similar construction processes and followed by similar design codes; during the 141 operation stage, these bridges serve in the same operating environment and are subject to the related 142 traffic loads; in the maintenance stage, the bridge inspection and maintenance actions are employed 143 by the similar agencies. Therefore, the bridge deterioration modes that address the correlations 144 between the structural parameters and condition levels are hidden in the years of regional bridge 145 inspection reports. 146 This paper intends to extract these correlations with deep learning techniques to effectively 147 reveal the regional structural deterioration features and predict the condition levels in the future. For 148 aging bridges, the structural deterioration model usually plays an important role in estimating the 149 structural condition level. The deterioration model should correctly interpret the structural 150 deterioration behavior and its trend. Since traditional statistical analysis and machine learning 151 techniques could only reveal some weak nonlinear relationships, this study uses the deep learning152 based framework of bridge condition level modeling to construct complex nonlinear relationships 153 among dependent and independent parameters and identify possible interactions within predictors. 154 The overview of the deep learning-based framework used in this study is shown in Figure 1. 155 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 5 - 156 Figure 1 Deep learning-based framework of bridge condition level modeling 157 DNNs are kinds of DL techniques that contain quite a few neurons and links to transfer the 158 information. Before training the DL model, the dataset should be extracted from the years of regional 159 bridge inspection reports. Before training the model, the extracted dataset is purified to remove data 160 error and redundancy. The duplicate items are deleted to avoid information redundancy. The missing 161 or abnormal items are inferred from other data [44, 45]. The reorganized samples that fit the DL 162 model's inputs are randomly split into training, testing, and validation datasets to train the model. The 163 DL model is trained with the training dataset to minimize the loss between predictions and ground 164 truths and adjust the hyper-parameters of the DL model. The validation dataset is applied to determine 165 the optimal hyper-parameters, and then the testing dataset is utilized to measure the model 166 performance. 167 2.2 U-Net convolutional neural networks 168 U-Net is a kind of DL model, and it has two significant aspects to improve the feature extraction 169 and class prediction performance. The encoder-decoder architecture enables good prediction 170 performance with very few labeled samples. It also utilizes feature concatenate techniques to achieve 171 multi-scale feature fusion. The U-Net architecture used in this research is shown in Figure 2. 172 173 Figure 2 The U-Net architecture used in this research 174 In the encoder-decoder architecture, the encoder section is a contracting path, while the decoder 175 section is a symmetric expanding path. Both encoder and decoder sections contain several successive 176 convolutional blocks. The convolutional blocks in the encoder section are utilized to capture the 177 Key features extraction Regional inspection data Number Abbreviation Parameters 1 RoadL Road level 2 Route Route number 3 Lane Number of lanes 4 Width Road width 5 Mile Mile 6 Side Bridge side 7 Age Bridge age 8 Year Bridge built year 9 Length Bridge total length 10 Type Superstructure type of bridge 11 Span Maximum span of bridge 12 ADT Average Daily Traffic 13 ADTT Annual Average Daily Truck Traffic 14 SuperM Maintenance actions for superstructure 15 SubM Maintenance actions for substructure 16 DeckM Maintenance actions for deck 17 SuperCL Condition levels for superstructure 18 SubCL Condition levels for substructure 19 DeckCL Condition levels for deck 20 BridgeCL Condition levels for bridge system Deterioration modeling Condition level prediction 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 6 - context information or global features of inputs. In contrast, the transposed convolutional blocks in 178 the decoder section are utilized to reconstruct the targets. In the convolutional block, the convolutional 179 layer is followed by a batch normalization layer. An activation layer is configured in each 180 convolutional block to extract hidden features for prediction from the input matrix. 181 The convolutional kernels in convolutional neural networks play the role of feature extractions 182 with different scales, thus, it can catch hidden features in different spatial dimensions. In the field of 183 computer vision, small convolutional kernels commonly produce local receptive features related to 184 textures and edges, while large convolutional kernels commonly generate broad receptive fields and 185 capture global features, such as, the relationship between the object and their contexts. Since the 186 classification task aims to assign the class label to each data point, it is required for the deep learning 187 model to learn both local and global features of the input. 188 The concatenate connections are employed between the encoder and decoder to ensure the 189 reconstruction of the input with better resolution. It attempts to achieve the goal by the skip connection 190 action, by which high-resolution features from the encoders are combined with up-sampled features 191 in the decoders. The combination of details features in the high-dimension level and background 192 information in the low-dimension level assists in producing a more precise output. 193 Since the raw U-net model is designed for computer vision tasks to accept images as the input, 194 its architecture should be adapted to process condition-related bridge data. Specifically, some 195 important parameters in the U-Net architecture are summarized in Table 1. Our model retains the main 196 features of the architecture proposed by Ronneberger [46] but simplifies the architecture to reduce the 197 computational burden and maintain the prediction performance. The input of the U-Net is the bridge 198 condition matrix with the size of 16×16. The inputs are processed with successive convolutional and 199 transposed convolutional layers in the encoder-decoder to extract multi-level features. Due to the 200 small size of the input matrix, the total layers are set as 8. The maximum kernel size is 5. The 201 concatenate connection is added between layer 2 and layer 8, layer 3 and layer 7, and layer 4 and layer 202 6 to achieve appropriate feature combinations. The other hyper-parameters, including convolutional 203 stride, the activation functions, etc., are also demonstrated. In addition, the optimum architecture 204 determination is illustrated in section 5.4. 205 Table 1 The configuration of the U-Net architecture 206 Layer type Input size Output size Kernel size Stride Activation function Conv 16×16×1 12×12×32 5×5 1×1 LeakyReLU Conv+BN 12×12×32 8×8×64 5×5 1×1 LeakyReLU Conv+BN 8×8×64 4×4×128 5×5 1×1 LeakyReLU Conv+BN 4×4×128 1×1×256 4×4 1×1 LeakyReLU UConv+BN 1×1×256 4×1×128 4×1 1×1 ReLU UConv+BN 4×1×256 8×2×64 5×2 1×1 ReLU UConv+BN 8×2×128 12×3×32 5×2 1×1 ReLU UConv+BN 12×3×64 16×4×1 5×2 1×1 ReLU 2.3 Focal loss for imbalanced dataset 207 Class imbalance is a commonly encountered classification issue in the deep learning project. The 208 Structural condition level distribution of regional bridges suffers from a great imbalance, such as the 209 proportion of bridge condition level with level 1~3 accounts for about 80% ~ 90%. This imbalance 210 will cause the deep learning model to overfit the high condition level samples and underfit the low 211 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 7 - condition level samples, and the samples of dominant classes will comprise most of the entire loss 212 and dominate the model training. Therefore, this study introduces focal loss (FL) to alleviate this 213 imbalance impact. 214 The traditional cross-entropy loss (CEL) for binary classification is expressed in Equation (1). 215 𝑦′ represents the output value of the activation function of 𝑦 =1, and 1−𝑦′ represents the output 216 value of the activation function of 𝑦 =0. It fails to consider the difference between the dominant 217 classes and rare classes. Dominant samples have more chances to be employed to train the deep 218 learning model and lead to lower loss. 219 𝐶𝐸𝐿 =−𝑦𝑙𝑜𝑔(𝑦′)−(1−𝑦)log(1−𝑦′)={ −log⁡(𝑦′) 𝑦=1 −log(1−𝑦′)𝑦=0 (1) Rather than the traditional CEL, this study introduces the FL function (shown in Equation (2)), 220 which adds moderating factor 𝛾 and balancing factor 𝛼 to the traditional CEL. The 𝛾 weighs the 221 losses caused by class imbalance that is prone to lead to misclassification, and⁡𝛼 balances the sample 222 imbalance. The phenomenon that samples of the rare classes are prone to be classified with poor 223 performance could be softened. When 𝛾 =0, FL equals to the traditional CEL. When 𝛾 increases, 224 it weights down the loss caused by the dominant classes in the model training, and Lin et al. [47] 225 suggested that 𝛾 =2 helps the model achieve the best performance. FL effects with different 226 moderation factors is illustrated in Figure 3. Since 𝛼 is less effective than⁡𝛾 in softening the class 227 imbalance, this study only uses the moderating factor in FL to improve the prediction performance. 228 𝐿𝑓𝑙 ={ −𝛼(1−𝑦′)𝛾log⁡(𝑦′) 𝑦=1 −(1−𝛼)𝑦′𝛾log(1−𝑦′)𝑦=0 (2) 229 Figure 3 Focal loss with different moderation factors 230 2.4 Evaluation metrics 231 Evaluation metrics used to assess the developed model are necessary to ensure the prediction 232 performance to be achieved. To quantitatively evaluate the classification performance of the trained 233 classifier, this study employs statistical indicators. Table 2 shows the confusion matrix of a binary 234 classification. The model performance evaluation indicators can be defined based on the confusion 235 matrix. TP, FP, TN, and FN refer to the result of a test and the correctness of the classification. In the 236 field of structural condition level classification, TP implies correctly classifying the data classes to 237 the whole condition levels, FP implies that a result that indicates a given class exists when it does not, 238 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 8 - and TN implies that correctly classifying the condition level as other classes, and FN implies that a 239 predicted result which wrongly indicates that a class does not hold. 240 Table 2 Confusion matrix for a binary classification 241 Confusion matrix Predicted results Positive Negative Labeled results Positive True positive (TP) False positive (FP) Negative False negative (FN) True negative (TN) There are several indicators that could be used to evaluate classification performance. The 242 indicators 𝑅𝑒𝑐𝑎𝑙𝑙=𝑇𝑃 (𝑇𝑃+𝐹𝑁)⁄ and 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛=𝑇𝑃 (𝑇𝑃+𝐹𝑃)⁄ are commonly used to 243 evaluate the model performance that is trained with the balanced dataset. Since the structural condition 244 dataset is imbalanced, 𝐹1−𝑠𝑐𝑜𝑟𝑒 performs better evaluate the imbalanced classification 245 performance. It is a weight that averages the 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 and 𝑅𝑒𝑐𝑎𝑙𝑙, and it is usually more useful 246 than 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦. Generally, the closer these additional indicators are to 1, the better the classification 247 performance is, and these indicators are mathematically defined in Equations (3) ~ (4). 248 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦= 𝑇𝑃+𝑇𝑁 𝑇𝑃+𝐹𝑃+𝑇𝑁+𝐹𝑁 (3) 𝐹1−𝑠𝑐𝑜𝑟𝑒 =2×𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛×𝑅𝑒𝑐𝑎𝑙𝑙 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑅𝑒𝑐𝑎𝑙𝑙 (4) 3. Condition-related data of regional bridges 249 3.1 Introduction of China inspection inventory 250 In China, the technical condition assessment of highway bridges mainly includes the assessment 251 of bridge components, bridge units, decks, superstructures, substructures, and bridge systems. 252 “Standards for Technical Condition Evaluation of Highway Bridges” (JTG/T H21-2011)[48] specifies 253 that the condition assessment of highway bridges should use the hierarchical analysis method to 254 evaluate the structural condition comprehensively. Based on the control limits and assessment 255 indicators, each bridge component should be evaluated first, then each bridge unit should be evaluated, 256 next the bridge deck, superstructure, and substructure should be evaluated, and finally, the overall 257 technical condition of the bridge system should be evaluated. The overall hierarchical analysis 258 flowchart is shown in Figure 4. 259 260 Figure 4 The hierarchical analysis used in the technical condition assessment of highway bridges 261 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 15 - 441 Figure 9 Prediction accuracy curves in the training process 442 4.3 Prediction performance 443 Furthermore, Figure 10 shows the variation of prediction accuracies of three bridge main parts 444 and bridge system in the validation dataset during the model training. The trend of these four 445 prediction accuracies is consistent with Figure 9, which shows that all of them are nearly stabled after 446 60 rounds of training epoch. The prediction accuracy of the bridge deck is the highest, at 95.86%; the 447 prediction accuracy of the superstructure and substructure is similar, around 94.80%; and the 448 prediction accuracy of the bridge system is the lowest, converging at 86.03%. The well-trained 449 classifier reaches a satisfactory prediction performance and good generalization performance. 450 451 Figure 10 Prediction accuracy curves for different bridge parts and system 452 With the above evaluation metrics, the classification performance on the testing dataset is 453 displayed in Table 5 and Table 6. The accuracy and F1-score bring about 95% for the three bridge 454 main parts. The prediction performance of the bridge system is slightly weaker. From three bridge 455 main parts and systems, the prediction accuracy and F1-score of each condition level are 456 approximately the same, indicating that focal loss avoids the effect of imbalanced data distribution. 457 Table 5 Classification accuracy of each structure 458 Indicators L1 L2 L3 L4 L5 Superstructures 95.71% 95.14% 99.00% 94.57% 98.57% Substructures 95.57% 95.00% 99.14% 95.71% 99.29% Decks 96.57% 96.14% 99.00% 93.14% 98.00% 050 100 150 200 86% 88% 90% 92% 94% Accuraccy Epoch Training Validation 050 100 150 200 70% 80% 90% 100% Accuracy Epoch SuperCL SubCL DeckCL BridgeCL 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 16 - Systems 92.14% 88.00% 93.86% 94.00% 98.86% Table 6 Classification F1-score of each structure 459 Structures L1 L2 L3 L4 L5 Superstructures 94.36% 94.01% 96.65% 94.44% 94.74% Substructures 95.39% 92.63% 96.74% 96.55% 100.00% Decks 96.20% 94.84% 94.31% 97.87% 100.00% Systems 89.40% 84.03% 84.01% 83.33% 85.71% 4.4 Impact of data missing rate 460 Due to a variety of reasons, measured data on the bridge inspection reports are sometimes 461 missing or incorrect in practice. Human mistakes and missing data influence prediction performance 462 and efficiency. Improper paper storage and damaged electronic documents may impact data 463 consistency and accuracy. Some parameters, such as RoadL, Route, Lane, Width, Mile, Side, Length, 464 Type, and Span, have little impact on the condition level prediction since they are constant and could 465 be updated with data from other inspection years. Others might mislead the classifier. In this section, 466 the prediction robustness of the U-Net model is validated under various data missing rates. The 467 missing rate is defined by the ratio of the number of database elements with missing data to the total 468 number of database elements. 469 The mean interpolation method fills in the missing data when training the model conventionally. 470 Figure 11 depicts the prediction accuracy curves for different data missing rates. As the missing rate 471 rises, the classification accuracy of each bridge part and system drops. The prediction performance of 472 the trained classifier is the worst when the missing data rate reaches 20%. The overall prediction 473 performance decreased by approximately 4.75%, with limited impact. Data missing impacts the 474 bridge system most, it might be concluded that the model with the worst prediction performance under 475 the complete data also has the worst robustness. The result implies that the increase in data missing 476 rate can reduce the accuracy and stability of the classifier. 477 478 Figure 11 Prediction accuracy curves for different data missing rate 479 0% 5% 10% 15% 20% 70% 80% 90% 100% Accuracy Data loss rate BridgeCL SubCL DeckCL SuperCL 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 17 - 4.5 Optimal architecture determination 480 In this section, some critical issues related to the design of the U-Net model are discussed. Three 481 candidate U-Net models with different kernel sizes, the number of layers, and the number of skip 482 connections are compared to determine the most optimum U-Net architecture. Since the input 483 dimension is fixed with the 16×16, the convolutional kernel sizes in three candidate models are three, 484 four, and five, respectively. The associated number of layers is set as sixteen, ten, and eight, 485 respectively. In addition, they are all designed with three concatenate connections to achieve extracted 486 feature fusion to improve the prediction performance. The comparison of total accuracy with different 487 U-Net architectures is shown in Figure 12. 488 489 Figure 12 Comparison of total accuracy with different U-Net architectures 490 Three candidate models achieve over 92% total prediction accuracy. The prediction performance 491 of models with a kernel size of three and four are nearly the same. After the 70-training epoch, these 492 two classifiers become stable. The candidate model with the kernel size of five nearly achieves the 493 best prediction performance from the start of the training, but it has some huge fluctuation in the first 494 50 training epochs. It achieves a final prediction performance of 92.78%, which is slightly higher than 495 the other two classifiers. The classification results for each bridge part and system are listed in Table 496 7. Except for the bridge superstructure, the model with kernel size 5 has higher classification accuracy 497 than the other two classifiers for the bridge substructure, deck, and system. It can be concluded that a 498 slightly larger convolutional kernel size helps to extract features at the regional bridge condition level 499 in the U-Net model. 500 Table 7 Prediction performance of different U-Net architectures 501 Models Prediction accuracy Bridge system Superstructures Substructures Deck Kernel=3, concatenate=3 81.12% 95.44% 91.02% 94.08% Kernel=4, concatenate=3 82.55% 95.90% 90.95% 94.99% Kernel=5, concatenate=3 85.94% 94.67% 94.73% 95.77% 4.6 Compared with other deep learning models 502 Four candidate deep learning models with different architectures are compared to illustrate the 503 effectiveness of the model. A candidate deep learning model is artificial neural networks (ANN) only 504 designed with neurons and back propagations. Since the U-Net models have the convolutional layers 505 050 100 150 200 85% 90% 95% Accuracy Epoch kernel=3 kernel=4 kernel=5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 18 - and encoder-decoder structure, the CNN and Encoder-decoder are also selected as the comparative 506 models. The training parameters for the model, such as the training period, batch size, and learning 507 rate, are all set to the same value. The comparison of total accuracy is shown in Figure 13. 508 509 Figure 13 Comparison of total accuracy with different deep learning models 510 The backpropagation mechanism improves the feature extraction performance to a limited extent, 511 thus, the ANN model only ends up with about 82% prediction accuracy. Since both of CNN and 512 Encoder-decoder have convolutional layers, the prediction performance has been significantly 513 improved to around 90%. Furthermore, the encoder-decoder structure does not contribute to the 514 classification performance in the issues of bridge condition level prediction. For the U-Net model, as 515 mentioned above, achieves the best classification performance from the start of the training. It is worth 516 noting that the concatenate connections contribute a lot to the feature extraction and condition level 517 classification. The classification comparison of each deep learning model for each bridge part and 518 system is summarized in Table 8. It is surprising that the prediction of the bridge system in ANN 519 achieves almost the same performance as that of the U-Net model. This might be inferred that the 520 ANN model can only focus on fitting a few results and has limited ability to extract global features. 521 The U-Net model integrating backpropagations, convolutional layers, encoder-decoder structure, and 522 concatenate connections achieves the best prediction performance in each bridge part and system. 523 Table 8 Prediction performance of different deep learning models 524 Models Prediction accuracy Bridge system Superstructures Substructures Deck ANN 85.76% 77.04% 84.64% 84.64% CNN 80.79% 94.38% 90.62% 93.49% Encoder-Decoder 80.47% 94.12% 89.45% 94.99% U-Net 85.94% 94.67% 94.73% 95.77% 5. Conclusions 525 This study offers a condition level deterioration modeling technique using the U-Net model to 526 increase the forecast accuracy of future structural situations. The bridge inspection data contains the 527 structural deterioration pattern, and the condition changes are closely related to the environment and 528 structural characteristics. The data acquired from years of regional bridge inspection reports back up 529 050 100 150 200 40% 60% 80% 100% Accuracy Epoch U-Net Encoder-Decoder CNN ANN 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 - 19 - the proposed strategy. Before training the model, the effect of bridge ages and superstructure types on 530 regional condition-related characteristics is explored, and the correlations between selected features 531 and structural conditions. Comparative studies are also used to validate the best architecture design 532 and efficacy. Several conclusions regarding the developed condition level deterioration modeling 533 method can be drawn: 534 1) The developed U-Net deterioration modeling method has superior prediction accuracy and 535 classification performance. It takes advantage of the inspection data to explore the regional 536 bridge condition deterioration characteristics. Since the U-Net model contains convolutional 537 layers, encoder-decoder structure, and concatenate connections to improve the feature 538 extraction efficiency, it outperforms other deep learning models (ANNs, CNNs, and 539 Encoder-Decoder). It brings out total accuracy of around 92.5%. 540 2) The proposed method has a good prediction robustness performance, validated under various 541 data missing rates. The overall prediction performance was slightly degraded when using 542 average interpolation to fill in the assumed missing data. When the missing data rate reaches 543 20%, the overall prediction performance decreases by approximately 4.75%. Data missing 544 impacts the bridge system most. It might be concluded that the model with the worst 545 prediction performance under the complete data also has the worst robustness. 546 3) Spearman correlation coefficients are revealed between selected regional key features and 547 structural condition levels. The strong correlations between these key regional features 548 indicate the inherent relationship between variables. Length significantly impacts structural 549 conditions in the relationship between the regional structural features and the structural 550 condition levels. ADT has a greater influence on the condition level of the bridge system. 551 The condition level distributions concerning bridge age for different bridge parts and systems 552 are disclosed. 553 Future work could improve the prediction accuracy of the bridge system by adjusting the 554 hyperparameters or introducing the attention mechanisms. This study mainly assesses the regional 555 bridges in individual aspects. A regional safety indicator should be proposed in the future to represent 556 the safety and serviceability of the total bridges or bridge network 557 Funding 558 This paper is supported by the National Natural Science Foundation of China (51978508), 559 Technology Cooperation Project of Shanghai Qizhi Institute (SYXF0120020109), and Transportation 560 Science and Technology Program of Shandong Province (2021B51). 561 References [1] K.W. Al Shboul, H.A. Rasheed, H.A. Alshareef, Intelligent approach for accurately predicting fatigue damage in overhead highway sign structures, Structures, 34 (2021) 3453-3463, doi: 10.1016/j.istruc.2021.09.090. [2] F. Alogdianakis, D.C. Charmpis, I. Balafas, Macroscopic effect of distance from seacoast on bridge deterioration - Statistical data assessment of structural condition recordings, Structures, 27 (2020) 319-329, doi: 10.1016/j.istruc.2020.05.052. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 20 [3] Y. Xia, X.M. Lei, P. Wang, G.M. Liu, L.M. Sun, Long-term performance monitoring and assessment of concrete beam bridges using neutral axis indicator, Structural Control & Health Monitoring, 27 (2020) doi: ARTN e2637 10.1002/stc.2637. [4] D.M. Frangopol, Y. Dong, S. Sabatino, Bridge life-cycle performance and cost: analysis, prediction, optimisation and decision-making, Structure and Infrastructure Engineering, 13 (2017) 1239-1257, doi: 10.1080/15732479.2016.1267772. [5] A. Panenka, F.M.N. Fangue, R. Rabe, H. Schmidt-Baumler, J. Sorgatz, Reliability assessment of ageing infrastructures: an interdisciplinary methodology, Structure and Infrastructure Engineering, 16 (2020) 698-713, doi: 10.1080/15732479.2019.1662063. [6] M. Rashidi, B. Samali, P. Sharafi, A new model for bridge management: Part A: condition assessment and priority ranking of bridges, Australian Journal of Civil Engineering, 14 (2015) 35-45, doi: 10.1080/14488353.2015.1092641. [7] S.Q. Wang, L.Y. Zhang, H. Su, J.S. Du, Time-dependent robustness-based condition assessment of RC bridges subjected to corrosion, Structures, 34 (2021) 4500-4510, doi: 10.1016/j.istruc.2021.10.061. [8] B. Tu, Z. Fang, Y. Dong, D.M. Frangopol, Time-variant reliability analysis of widened deteriorating prestressed concrete bridges considering shrinkage and creep, Engineering Structures, 153 (2017) 1-16, doi: 10.1016/j.engstruct.2017.09.060. [9] F. Pavel, R. Vacareanu, D. Marcu, Seismic performance assessment and rating for a flat-slab RC core wall structure in Bucharest, Romania, Structures, 31 (2021) 1006-1016, doi: 10.1016/j.istruc.2021.02.036. [10] J.C. Matos, P.J.S. Cruz, I.B. Valente, L.C. Neves, V.N. Moreira, An innovative framework for probabilistic-based structural assessment with an application to existing reinforced concrete structures, Engineering Structures, 111 (2016) 552-564, doi: 10.1016/j.engstruct.2015.12.040. [11] L.F.M. Sanchez, B. Fournier, D. Mitchell, J. Bastien, Condition assessment of an ASR-affected overpass after nearly 50 years in service, Construction and Building Materials, 236 (2020) doi: ARTN 117554 10.1016/j.conbuildmat.2019.117554. [12] I. Zambon, A. Vidovic, A. Strauss, J. Matos, Condition Prediction of Existing Concrete Bridges as a Combination of Visual Inspection and Analytical Models of Deterioration, Applied Sciences-Basel, 9 (2019) doi: ARTN 148 10.3390/app9010148. [13] N.J. Bertola, E. Bruhwiler, Risk-based methodology to assess bridge condition based on visual inspection, Structure and Infrastructure Engineering, (2021) 1-14, doi: 10.1080/15732479.2021.1959621. [14] J. Ghosh, K. Rokneddin, J.E. Padgett, L. Dueñas-Osorio, Seismic Reliability Assessment of Aging Highway Bridge Networks with Field Instrumentation Data and Correlated Failures, I: Methodology, Earthquake Spectra, 30 (2019) 795817, doi: 10.1193/040512eqs155m. [15] N. Galvão, J.C. Matos, D.V. Oliveira, R. Hajdin, Human error impact in structural safety of a reinforced concrete bridge, Structure and Infrastructure Engineering, 18 (2021) 836-850, doi: 10.1080/15732479.2021.1876105. [16] M. Li, G.F. Jia, Bayesian Updating of Bridge Condition Deterioration Models Using Complete and Incomplete Inspection Data, Journal of Bridge Engineering, 25 (2020) doi: Artn 04020007 10.1061/(Asce)Be.1943-5592.0001530. [17] D. Honfi, I. Bjornsson, O.L. Ivanov, J. Leander, Informed successive condition assessments in bridge maintenance, Journal of Civil Structural Health Monitoring, 10 (2020) 729-737, doi: 10.1007/s13349-020-00415-2. [18] A. Bagheri, O.E. Ozbulut, D.K. Harris, M. Alipour, A.Z. Hosseinzadeh, A hybrid experimental-numerical approach for load rating of reinforced concrete bridges with insufficient structural properties, Structure and Infrastructure Engineering, 15 (2019) 754-770, doi: 10.1080/15732479.2019.1572197. [19] E. Romano, L. Cascini, M. D'Aniello, F. Portioli, R. Landolfo, A simplified multi-performance approach to lifecycle assessment of steel structures, Structures, 27 (2020) 371-382, doi: 10.1016/j.istruc.2020.05.053. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 21 [20] X. Xu, Q. Huang, Y. Ren, H.B. Sun, Condition Assessment of Suspension Bridges Using Local Variable Weight and Normal Cloud Model, Ksce Journal of Civil Engineering, 22 (2018) 4064-4072, doi: 10.1007/s12205-018-1819-3. [21] X. Xu, Q. Huang, Y. Ren, D.Y. Zhao, D.Y. Zhang, H.B. Sun, Condition evaluation of suspension bridges for maintenance, repair and rehabilitation: a comprehensive framework, Structure and Infrastructure Engineering, 15 (2019) 555-567, doi: 10.1080/15732479.2018.1562479. [22] T.U. Saeed, Y. Qiao, S.K. Chen, K. Gkritza, S. Labi, Methodology for Probabilistic Modeling of Highway Bridge Infrastructure Condition: Accounting for Improvement Effectiveness and Incorporating Random Effects, Journal of Infrastructure Systems, 23 (2017) doi: Artn 04017030 10.1061/(Asce)Is.1943-555x.0000389. [23] B.X. Ge, S. Kim, Probabilistic service life prediction updating with inspection information for RC structures subjected to coupled corrosion and fatigue, Engineering Structures, 238 (2021) doi: ARTN 112260 10.1016/j.engstruct.2021.112260. [24] B.X. Ge, S. Kim, Determination of appropriate updating parameters for effective life-cycle management of deteriorating structures under uncertainty, Structure and Infrastructure Engineering, 17 (2021) 1284-1298, doi: 10.1080/15732479.2020.1809466. [25] B. Murali Krishna, V. Guru Prathap Reddy, M. Shafee, T. Tadepalli, Condition assessment of RC beams using artificial neural networks, Structures, 23 (2020) 1-12, doi: 10.1016/j.istruc.2019.09.014. [26] A. Gouda Mohamed, M. Marzouk, Building condition assessment using artificial neural network and structural equations, Expert Systems with Applications, 186 (2021) doi: 10.1016/j.eswa.2021.115743. [27] A.M. Ivankovic, A. Strauss, H. Sousa, European review of performance indicators towards sustainable road bridge management, Proceedings of the Institution of Civil Engineers-Engineering Sustainability, 173 (2020) 109-124, doi: 10.1680/jensu.18.00052. [28] Y.H. Huang, Artificial Neural Network Model of Bridge Deterioration, Journal of Performance of Constructed Facilities, 24 (2010) 597-602, doi: 10.1061/(Asce)Cf.1943-5509.0000124. [29] Z. Li, R. Burgueno, Using Soft Computing to Analyze Inspection Results for Bridge Evaluation and Management, Journal of Bridge Engineering, 15 (2010) 430-438, doi: 10.1061/(Asce)Be.1943-5592.0000072. [30] Z. Allah Bukhsh, I. Stipanovic, A. Saeed, A.G. Doree, Maintenance intervention predictions using entityembedding neural networks, Automation in Construction, 116 (2020) doi: 10.1016/j.autcon.2020.103202. [31] S. Mangalathu, G. Heo, J.S. Jeon, Artificial neural network based multi-dimensional fragility development of skewed concrete bridge classes, Engineering Structures, 162 (2018) 166-176, doi: 10.1016/j.engstruct.2018.01.053. [32] S. Mangalathu, K. Karthikeyan, D.-C. Feng, J.-S. Jeon, Machine-learning interpretability techniques for seismic performance assessment of infrastructure systems, Engineering Structures, 250 (2022) doi: 10.1016/j.engstruct.2021.112883. [33] M. Santamaria Ariza, I. Zambon, H. S. Sousa, J.A. Campos e Matos, A. Strauss, Comparison of forecasting models to predict concrete bridge decks performance, Structural Concrete, 21 (2020) 1240-1253, doi: 10.1002/suco.201900434. [34] H. Sun, H.V. Burton, H.L. Huang, Machine learning applications for building structural design and performance assessment: State-of-the-art review, Journal of Building Engineering, 33 (2021) doi: ARTN 101816 10.1016/j.jobe.2020.101816. [35] X.M. Lei, L.M. Sun, Y. Xia, Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks, Structural Health Monitoring-an International Journal, 20 (2021) 2069-2087, doi: Artn 1475921720959226 10.1177/1475921720959226. [36] H. Liu, Y.F. Zhang, Bridge condition rating data modeling using deep learning algorithm, Structure and Infrastructure Engineering, 16 (2020) 1447-1460, doi: 10.1080/15732479.2020.1712610. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 22 [37] G. Fiorillo, H. Nassif, Improving the conversion accuracy between bridge element conditions and NBI ratings using deep convolutional neural networks, Structure and Infrastructure Engineering, 16 (2020) 1669-1682, doi: 10.1080/15732479.2020.1725065. [38] J.S. Zhu, Y.L. Wang, Feature Selection and Deep Learning for Deterioration Prediction of the Bridges, Journal of Performance of Constructed Facilities, 35 (2021) doi: Artn 04021078 10.1061/(Asce)Cf.1943-5509.0001653. [39] H. Liu, J. Nehme, P. Lu, An application of convolutional neural network for deterioration modeling of highway bridge components in the United States, Structure and Infrastructure Engineering, (2021) 1-14, doi: 10.1080/15732479.2021.1979597. [40] J. Huyan, W. Li, S. Tighe, Z. Xu, J. Zhai, CrackU‐net: A novel deep convolutional neural network for pixelwise pavement crack detection, Structural Control and Health Monitoring, 27 (2020) doi: 10.1002/stc.2551. [41] K. Yang, Y.L. Ding, P. Sun, H.C. Jiang, Z.W. Wang, Computer vision-based crack width identification using FCNN model and pixel nonlinear calibration, Structure and Infrastructure Engineering, (2021) 1-12, doi: 10.1080/15732479.2021.1994617. [42] C. Dong, L. Li, J. Yan, Z. Zhang, H. Pan, F.N. Catbas, Pixel-Level Fatigue Crack Segmentation in Large-Scale Images of Steel Structures Using an Encoder-Decoder Network, Sensors (Basel), 21 (2021) doi: 10.3390/s21124135. [43] Z.H. Li, M.Q. Huang, P.X. Ji, H.M. Zhu, Q.B. Zhang, One-step deep learning-based method for pixel-level detection of fine cracks in steel girder images, Smart Structures and Systems, 29 (2022) 153-166, doi: 10.12989/sss.2022.29.1.153. [44] Y. Xia, X.M. Lei, P. Wang, L.M. Sun, A data-driven approach for regional bridge condition assessment using inspection reports, Structural Control & Health Monitoring, 29 (2022) doi: ARTN e2915 10.1002/stc.2915. [45] X.M. Lei, Y. Xia, L. Deng, L.M. Sun, A deep reinforcement learning framework for life-cycle maintenance planning of regional deteriorating bridges using inspection data, Structural and Multidisciplinary Optimization, 65 (2022) doi: ARTN 149 10.1007/s00158-022-03210-3. [46] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, Medical Image Computing and Computer-Assisted Intervention, Pt Iii, 9351 (2015) 234-241, doi: 10.1007/978-3-319-245744_28. [47] T.Y. Lin, P. Goyal, R. Girshick, K. He, P. Dollar, Focal Loss for Dense Object Detection, IEEE Trans Pattern Anal Mach Intell, 42 (2020) 318-327, doi: 10.1109/TPAMI.2018.2858826. [48] M.o.T.o.C. (MOT), Standards for technical condition evaluation of highway bridges (JTG/T H21-2011), in, China Communications Press Co., Ltd., Beijing, 2011. [49] Y. Xia, X.M. Lei, P. Wang, L.M. Sun, Artificial Intelligence Based Structural Assessment for Regional Shortand Medium-Span Concrete Beam Bridges with Inspection Information, Remote Sensing, 13 (2021) doi: ARTN 3687 10.3390/rs13183687. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 Manuscript ID: STRUCTURES-D-22-00902 Response to Reviewer Comments Response to Reviewer 1 Comments [General comments] It is a very interesting contribution in the field of performance assessment of bridge structures using neural network algorithms. The approach of the presented concept is coherent and comprehensible. ---------------------------------------------------------------------------------------------------------------- [Point 1] However, from my point of view, the discussions on the performance indicators and monitoring concepts are missing; these should at least be addressed in the introduction. See also: Comparison of forecasting models to predict concrete bridge decks performance. STRUCT CONCRETE. 2020; 21(4): 1240-1253. European review of performance indicators towards sustainable road bridge management. P I CIVIL ENG-ENG SU. 2020; 173(3): 109-124. [Response 1] Thank you for your valuable suggestions. We have added some discussions on the performance indicators and monitoring concepts. The presented two studies on the performance indicators provide us a holistic review of the research and operational indicators in the bridge management. In addition, different models (ANNs, MDP, hidden Markov model (HMM), and Semi-MDP) in forecasting the bridge deck performance are compared to show their performances. The HMM model provided a better representation in predicting the deck ratings than MDP and Semi-MDP, but the ANNs model achieved the best prediction performance. The added discussions are shown in Lines 80-82 and 88-91 in the revised manuscript. The added reference are: [27] A.M. Ivankovic, A. Strauss, H. Sousa. European review of performance indicators towards sustainable road bridge management. Proceedings of the Institution of Civil EngineersEngineering Sustainability. 173 (2020). 109-124. doi:10.1680/jensu.18.00052. [32] S. Mangalathu, K. Karthikeyan, D.C. Feng, J.S. Jeon. Machine-learning interpretability techniques for seismic performance assessment of infrastructure systems. Engineering Structures. 250 (2022). doi:10.1016/j.engstruct.2021.112883. ---------------------------------------------------------------------------------------------------------------- [Point 2] Furthermore, the discussion of human error and the influence of the quality of execution is missing for me, what effect does this already have on the available data and the evaluation? [Response 2] Thank you for your helpful suggestions. Data quality significantly impacts the assessment of the condition of bridges. The preserved human error and data missing existed in the established dataset impact the analysis accuracy and efficiency. Data pre-processing techniques, such as cleaning and regulation, are then implemented to purify the established database and highlight the correlated attributes. For the duplicate items, we choose to delete them to avoid information redundancy. For the missing or abnormal items, if it appears in bridge design parameters, the data value can be rectified by inspection reports of other years of the same bridge. If it appears in bridge condition information, it can be inferred with the help of data from the previous year and the year after. For example, when there is no maintenance action in the missing-data year, and the bridge Detailed Response to Reviewers Manuscript ID: STRUCTURES-D-22-00902 Response to Reviewer Comments condition level of the previous year and the year after is unchanged, the missing or abnormal condition value could be rectified as the condition level of the previous year. If it appears in bridge maintenance information, it might be inferred with the help of the data of the later year. For example, when the condition level in the current year is the same as the following year, we suppose that the missing bridge maintenance information should be no maintenance action; when the condition level in the current year is worse than the following year, we suppose the inspected bridge is performed with maintenance actions. Since the presented data pre-processing methods are not the contributions of this study, and they have been addressed in our previous studies (Ref. 44 and 45), the brief description of the pre-processing of data error are added in Lines 159-161 in the revised manuscript. ---------------------------------------------------------------------------------------------------------------- [Point 3] What is also not yet clear to me is the transition from performance indicators to key performance indicators, as they are treated in the following works, for example, and which should be addressed in this article. https://onlinelibrary.wiley.com/doi/pdf/10.1002/bate.201700104 https://www.scopus.com/record/display.uri?src=s&origin=cto&ctoId=CTODS_1428438 723&stateKey=CTOF_1428438724&eid=2-s2.0-85093818120 [Response 3] Thank you for your helpful suggestions. The first paper is in Germany and the author could not fully understand its content. The second link cannot be opened to see its content. In this study, there are two kinds of indicators. The first indictor is the “condition level” which is used to quantitively evaluate the structural condition of bridges. It belongs to the operational indicator that denotes in the Reference [27]. The second indicator is the statistical indicator that denotes the “key performance indicator” in this study. It is used to evaluate the prediction performance of the trained model. The key performance indicators are the accuracy, and F1-score. The explanation of evaluation metrics and condition levels are shown in Lines 241-247 and 261-265, respectively. ----------------------------------------------------------------------------------------------------------------