Definition of AI-based tools / D1.1

Definition of AI-based tools / D1.1 WP1, T1.1 Authors: Dr Evelyne El Masri (BUL); Dr Henrique de Melo Ribeiro (BUL); Mr Mohmmad Ali Asgar Abbas (BUL) Disclaimer Co-funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or REA. Neither the European Union nor the granting authority can be held responsible for them. Definition of AI-based tools 2 1. Technical references Project Acronym RECONSTRUCT Project Title A Territorial Construction System for a Circular LowCarbon Built Environment GA number 101082265 Project Coordinator Kathleen Blanco Instituto de Tecnología de la Construcción de Cataluña Head of Innovation Project Management Department [email protected] Project Duration June 2023 – May 2027 (48 months) Deliverable No. D1.1 Dissemination level* PU Work Package WP 1 - AI-based sourcing and characterization of materials and waste Task T1.1 - Data collection, design and validation of the AI-based sourcing modules Lead beneficiary 16. BUL Contributing beneficiary/ies 1.ITEC, 4.SYM, 5.UNIVPM, 6.COM, 8.SOR Due date of deliverable 30 November 2023 Actual submission date 05 December 2023 * PU – Public, fully open, e.g. web (Deliverables flagged as public will be automatically published in CORDIS project’s page) SEN – Sensitive, limited under the conditions of the Grant Agreement Classified R-UE/EU-R – EU RESTRICTED under the Commission Decision No2015/444 Classified C-UE/EU-C – EU CONFIDENTIAL under the Commission Decision No2015/444 Classified S-UE/EU-S – EU SECRET under the Commission Decision No2015/444 Definition of AI-based tools 3 v Date Beneficiary Author 1.0 29/09/2023 First internal draft (BUL) Dr Henrique de Melo Ribeiro (BUL); Mr Mohmmad Ali Asgar Abbas (BUL) 2.0 06/10/2023 First internal review (BUL) Dr Evelyne El Masri (BUL) 3.0 20/10/2023 Second internal draft (BUL) Dr Henrique de Melo Ribeiro (BUL); Mr Mohmmad Ali Asgar Abbas (BUL) 4.0 24/10/2023 Second internal review (BUL) Dr Evelyne El Masri (BUL) 5.0 27/10/2023 First external draft (BUL) Dr Evelyne El Masri (BUL), Dr Henrique de Melo Ribeiro (BUL); Mr Mohmmad Ali Asgar Abbas (BUL) 6.0 06/11/2023 Second external draft (ITeC) Jose Lucas, Kathleen Blanco, Vanessa Carolina Soteldo, Laura Silva (ITeC) 7.0 30/11/2023 Third internal draft (BUL) Dr Henrique de Melo Ribeiro (BUL); Mr Mohmmad Ali Asgar Abbas (BUL) 8.0 01/12/2023 Final Report (BUL) Dr Evelyne El Masri (BUL) Definition of AI-based tools 4 2. Table of contents 1. Technical references ................................................................... 2 2. Table of contents ......................................................................... 4 3. Introduction .................................................................................. 7 4. Literature Review ......................................................................... 8 4.1. Introduction .......................................................................................................... 8 4.2. Construction Demolition Waste Detection ........................................................ 8 4.2.1. Overview ............................................................................................................. 8 4.2.2. Hand-Crafted Feature Based .............................................................................. 8 4.2.3. Traditional Feature Learning ............................................................................... 9 4.2.4. Supervised Learning ........................................................................................... 9 4.2.5. Semi-Supervised Learning ............................................................................... 14 4.3. Construction Demolition Waste Quantification .............................................. 15 4.3.1. Overview ........................................................................................................... 15 4.3.2. Hand-Crafted Feature Based ............................................................................ 15 4.3.3. Traditional Feature Learning ............................................................................. 16 4.3.4. Supervised Learning ......................................................................................... 16 4.4. Construction Demolition Waste Satellite Geolocation ................................... 19 4.4.1. Overview ........................................................................................................... 19 4.4.2. Ideal Implementations ....................................................................................... 19 4.4.3. Building Footprint Detection and Methodology ................................................. 20 4.4.4. Plastic Function Identification and Methodology ............................................... 21 4.4.5. Expanding Horizons: Satellite Imagery in Construction .................................... 23 4.4.6. Limitations on Satellite data and ML Applications ............................................. 23 4.5. Conclusion ......................................................................................................... 24 5. Methodology .............................................................................. 26 5.1. Introduction ........................................................................................................ 26 5.2. Dataset ............................................................................................................... 26 5.2.1. CDW Data Collection ........................................................................................ 26 5.2.2. Satellite Data Collection .................................................................................... 27 5.2.3. Data Specifications ........................................................................................... 30 5.2.4. Data Labelling ................................................................................................... 31 5.3. AI-based Modules .............................................................................................. 33 5.3.1. Classification Module Design ............................................................................ 33 5.3.2. Quantification Module Design ........................................................................... 34 5.3.3. Evaluation and Validation Criteria ..................................................................... 35 5.4. Geolocation Via Satellite Data .......................................................................... 35 5.4.1. Module Design .................................................................................................. 35 5.4.2. Evaluation and Validation Criteria ..................................................................... 35 6. WP1 Workflow ........................................................................... 37 Definition of AI-based tools 5 6.1. Introduction ........................................................................................................ 37 6.2. First Phase ......................................................................................................... 37 6.2.1. Conducting Literature Review ........................................................................... 37 6.2.2. Initial Planning and Data Collection .................................................................. 37 6.2.3. Data Processing and Model Exploration ........................................................... 37 6.3. Second Phase .................................................................................................... 37 6.3.1. Planning and Collaboration ............................................................................... 37 7. Conclusions ............................................................................... 39 7.1. Feedback and Refinement ................................................................................ 39 7.2. Future Work ....................................................................................................... 39 8. References ................................................................................ 41 List of Tables Table 1: Hand-crafted feature-based algorithms .................................................................. 9 Table 2: Data Specifications by Sensor ............................................................................. 30 List of Figures Figure 1: CDW samples [31]. ............................................................................................. 10 Figure 2: Left): original image; (Middle): binarised and filtered image and (Right): segmented image [33]. .................................................................................................................. 10 Figure 3: Apparatus used to take images of recycled aggregates [33]. ............................. 11 Figure 4: GoPro mounting configuration above bin. (b) Location of GoPro Hero4 above bin [34]. ............................................................................................................................. 11 Figure 5: Segmentation results [35]. .................................................................................. 12 Figure 6: Confusion matrix on the test set [35]. ................................................................. 13 Figure 7: Inference results from YoloV7x model [37]. ........................................................ 14 Figure 8: (a) Photograph and (b) schematic Illustration of the prototype CDW sorting station comprised of (A) an RGB camera, (B) a robotic arm manipulator and (C) a conveyor belt [38]. ...................................................................................................................... 14 Figure 9: (a) RGB-D acquisition platform (b) Color image (c) Depth image (d) The contours and categories of objects are detected [38]. ............................................................... 17 Figure 10: (a) RGB-D concat model (b) RGB-D Ci-add model and RGB-D Ci-concat model [38]. ............................................................................................................................. 18 Figure 11: System flowchart of the localization module [60]. ............................................. 19 Figure 12: Example Image from dataset and label [63]...................................................... 20 Figure 13: Enlarged subscenes of Sentinel-2 imagery, superimposed with the ground truth map of the respective functions (colours) of plastic materials in polygons, and with filled polygons as a result of function assignment using VGG-13 model. Misclassifications are depicted when colours of polygons and fillings differ (e.g., top b2; bottom right b5) [63]. ............................................................................................................................. 22 Figure 14: CDW image samples collected from SOR and COM sites. .............................. 26 Figure 15. CDW sample distribution by site. ...................................................................... 27 Figure 16: AOI of Senitel-2 image of Sorigue Site. ............................................................ 28 Figure 17: Request builder interface. [74] .......................................................................... 29 Figure 18: Image example from Pleiades. [77] .................................................................. 29 Figure 19: Semantic segmentation vs instance segmentation. .......................................... 31 Definition of AI-based tools 6 Figure 20: CDW samples instance labelling outcomes using RGB-D images. Left: raw image; Middle: depth image; Right: Instance segmentation (coloured coded masks) [38]. ............................................................................................................................. 31 Figure 21: Dataset labelling per EU standard. ................................................................... 32 Figure 22: Mask R-CNN architecture illustration [78]. ........................................................ 34 Figure 23: Sample of CDW RGB and depth images [38]. .................................................. 34 Definition of AI-based tools 7 3. Introduction The escalating volume of Construction and Demolition Waste (CDW) alongside industrial waste has propelled the necessity for innovative solutions to repurpose these materials. The development of novel construction materials from CDW and industrial waste not only adheres to the principles of a circular economy but also significantly mitigates the environmental impact of the construction sector. Central to this initiative is the deployment of Artificial Intelligence (AI)-based tools, which facilitate the sourcing and characterisation of these waste materials. AI-based tools encompass a range of computational technologies that mimic human intelligence processes to autonomously perform tasks or solve complex problems. In the domain of CDW management, these tools are pivotal for identifying the sources of waste materials, collecting requisite data, and subsequently processing this data to derive actionable insights. The data, comprising satellite and image data samples, serve as the cornerstone for developing and training sophisticated image processing and machine learning algorithms. These algorithms are integral to the envisioned digital sourcing system, streamlining the identification, collection, and characterisation of CDW waste. Within Work Package 1 framework, series of digital systems will be deployed to automatically determine local CDW availability. This will be achieved by applying image processing techniques on satellite data to localize CDW, using CCTV image data to detect and quantify it, and employing data from onsite Hyperspectral Imaging (HSI) analysers to characterise CDW. The information garnered through these digital systems will be fed into a database of industrial activities, thereby enabling the automatic identification and ranking of circular pathways for waste valorisation. This report aims to clarify and identify the design, development, and implementation of AIbased tools in establishing a robust digital sourcing system. It delves into the intricacies of data collection, machine learning algorithm development, and the collaborative engagements with local industries. Through a detailed examination, this report aims to underscore the transformative potential of AI-based tools in steering the construction industry towards sustainable practices by leveraging CDW and industrial waste for the development of innovative construction materials. Definition of AI-based tools 8 4. Literature Review 4.1. Introduction Construction and demolition waste (CDW) significantly contributes to the overall waste stream [1, 2]. A World Bank report indicates that materials like rubble, concrete, and masonry, which are typical components of CDW, can represent up to 40% of the total solid waste generated in certain cities [3]. Given the substantial volume of CDW, the need for robust Construction Waste Management (CWM) cannot be overstated. Effective CWM requires accurate information regarding the composition of CDW. In countries such as Spain [4] and Brussels [5], where disposal fees vary based on the composition of CDW [6], interregional taxes, and type of site (e.g., permanent or temporary). Furthermore, during the treatment of construction waste at valorisation facilities, detailed information about waste material types and composition is crucial for automated sorting operations. 4.2. Construction Demolition Waste Detection 4.2.1. Overview Utilising computer vision (CV) for waste detection emerges as a promising strategy, especially given the ease and cost-effectiveness of photograph acquisition, as well as their aptness for analysing a vast array of waste materials. Research in this arena has been active for over two decades [7], focusing on identifying waste materials through images and enabling various waste management applications. These encompass household waste classification [8,9], bin-level detection [10], and material segregation [11]. In the initial stages, much research emphasis was placed on recognising municipal solid waste [12]. Early methodologies involved manually crafting image features and employing hand-crafted feature-based algorithms, traditional feature learning machine learning (ML) models, such as support vector machines (SVM) and k-nearest neighbours (k-NN), for classification. However, due to the inherent manual feature crafting, the models experienced limited adaptability to variations, which constrained their broader applications [13,14]. The reemergence of deep learning (DL) provides opportunities to address existing limitations by employing an end-to-end training strategy [15,16], such as the employment of supervised learning algorithms such as convolutional neural network (CNN). 4.2.2. Hand-Crafted Feature Based Linear Discriminant Analysis (LDA) commonly utilised as a linear classification technique, aims to optimise a linear transformation from a high-dimensional feature space to a compact subspace when provided with a set of data samples. Subsequently, a classifier is designated to distribute these samples into distinct classes [17]. Originating from Fisher's initial work on dichotomous discriminant analysis in 1936, LDA has proliferated into various adaptations and extensions, such as multiple discriminant analysis and quadratic discriminant analysis. As underscored in Table 1, six studies have employed LDA as classifiers, focusing on the recognition/detection of Residential and Municipal (RM) and Industrial and Commercial (IC) wastes. The algorithm's efficacy, reliant upon the specific waste types, task objectives, and features utilised, manifests accuracy rates that vary from 85% to 98%. Hand-crafted feature- Definition of AI-based tools 9 based algorithms utilising LDA in CV faces various limitations and constraints including labour intensive hand-engineering, lack of generalisation, and lack of adaptability and scalability since features as static and transfer learning is not possible. Table 1: Hand-crafted feature-based algorithms Study Waste Type Accuracy Faibish et al. (1997)[18] RM 90% Leitner et al. (2003) [17] IC 88.16% Tachwali et al. (2007)[19] RM 94.14% Ramli et al. (2008) [20] RM 98.3 Koyanaka and Kobayashi (2010)[21] IC 88% Koyanaka and Kobayashi (2011) [22] IC 85% 4.2.3. Traditional Feature Learning Traditional feature learning algorithms can be heavily reliant on the choice of features used. How these features are represented, may involve a distance metric, such as the Euclidean distance, in order to determine which instances are “nearest” to another. Modern approaches employ automated feature/representation learning, akin to deep learning. The Nearest Neighbour algorithm rigid assumptions about the relationship between input variables and output labels, assigning a queried point the class label of the nearest point from a sample set S in the feature space [23]. K-NN, a variant, assigns a class based on a majority vote from the k nearest points. Despite its simple mechanism, K-NN is widely studied and has often achieved over 85% accuracy in various studies [24], making it a predominant choice among traditional machine learning algorithms when the computation complexity of the dataset is not high. Moreover, in contrast, Support Vector Machine (SVM) are more suitable for larger highdimensional data, provided computational resources are available. It involves identifying the optimal hyperplane that most effectively separates data samples from different classes. This optimal hyperplane is characterised by maximising the distance from the nearest points or support vectors of all classes [25]. Extensive analysis and applications have been noted for the task of solid waste recognition using SVMs [26,27,28], notably Kuritcyn et al. (2015) [27], achieving an accuracy of 96.5% targeting CDW aggregates. Limitations to this approach in similarity with K-NNs involve lack of scalability dependant on the size of dataset, also associated with high model training times, and lack of domain adaptation within the realm of transfer learning. 4.2.4. Supervised Learning Within supervised learning, it can be observed a progression between probabilistic graphical model Bayesian network, and end-to-end neural artificial networks. Bayesian network leverages cause-and-effect relationships among variables. Particularly in the context of waste classification, it calculates the conditional probability of a sample being assigned to a type of waste, considering its visual features. Such features include colour [29, 30], shape Definition of AI-based tools 16 object categories that employ quantised 3D Spatial Pyramid Matching Kernels in point clouds. In this methodology, 3D SURF local descriptors [51], were extracted and quantised into 3D visual words using K-means clustering, followed by the application of a linear support vector machine (LinSVM) for object recognition. In contrast, Logoglu et al. (2016) [52] introduced two spatially enhanced local 3D descriptors for object recognition tasks, namely, Histograms of Spatial Concentric Surflet-Pairs and Coloured SPAIR (CoSPAIR). Experiments revealed that the newly introduced CoSPAIR descriptor surpassed the performance of existing state-of-the-art descriptors in recognition tasks. 3D shape point cloud descriptors despite providing robust classification capabilities are limited by computational efficiency, noise handling, scalability, multi-class classification, and management of scalability for larger datasets and dynamic scenarios. 4.3.3. Traditional Feature Learning While hand-crafted features can elevate accuracy in object recognition, their development necessitates a deep comprehension of domain-specific knowledge and requires adaptation for new data types. A significant limitation of hand-crafted features is their tendency to capture only a subset of features from raw data that are discriminative for object recognition, inadvertently omitting other potentially valuable cues during the design process [53]. To mitigate dependence on hand-crafted features, various methods, centred around feature learning, which inherently extract features directly from raw data, have been proposed in the context of RGB-D object recognition. The widespread use of single-layer convolutional operators for learning low-level features from image and depth modalities, in an unsupervised manner, is noteworthy. Blum et al. (2012) [54], introduced a convolutional k-means descriptor, which was devised to autonomously derive meaningful local feature descriptions. They computed feature responses, consolidated them into feature histograms, and employed a LinSVM for object recognition. Similarly, Cheng et al. (2015) [55] introduced a convolutional Fisher kernel approach for RGB-D object recognition. This method involved learning convolutional features from image and depth modalities via a single-layer CNN. Moreover, two LinSVMs were trained independently for the RGB and depth modalities, and their aggregated scores were utilised to predict the category. On the other hand, Li et al. (2015) [56] employed filters of multiple scales and various pooling granularities and integrated colour as an additional pooling domain to minimise sensitivity to spatial deformations. Thus, shallow architecture design limits the model’s capability to learn complex features, only effective in identifying linear boundaries due to the lack of depth. Undermining generalisation due to lack of applicability in scenarios such as complex CDW material features. 4.3.4. Supervised Learning In the context of supervised learning approaches, two paths are typically chosen: constructing an RGB-CNN and a Depth-CNN or fine-tuning two readily available CNNs using labelled RGB and depth data. Following this, a multimodal fusion strategy is implemented to combine the two modalities, after which a classifier is trained for recognition purposes. Predominantly, the greatest share of cutting-edge research has leaned on CNNs pre-trained on ImageNet, with a few exceptions, such as the studies found in [53, 57], where CNNs Definition of AI-based tools 17 were purposefully designed and trained anew. The strategy of employing pre-trained CNNs is rooted in two principal justifications. First, there is a pronounced disparity in the availability of sufficiently scaled training datasets between depth images and colour (RGB) images, with the latter being far more abundant. Second, the versatility of pre-trained CNNs has been demonstrated, showcasing their capability to be fine-tuned to new datasets and thus address varied tasks within computer vision [58, 59]. In order to utilise a pre-trained CNN effectively, the RGB modality does not require additional processing, as it maintains a specific input distribution akin to that of natural camera images. However, depth images must be transformed into the RGB domain to access the benefits of features learned from CNNs originally trained on RGB images. Li et al. (2021) [38], introduces RGB-D fusion models entitled RGB-D Ci-add / concat, improving mean average precision for CDW segmentation, by 1.33% to 1.72, and classification accuracy by 1.92% to approximately 2.27%. In addition to suggesting real-time model inference of 265 ms, for robotic operations. The 8 targeted classes include concrete, machine-made brick, fired brick, wood, plaster, plastic, ceramic, and carton. The dataset is composed of 3367 (640 x 640) RGB and depth images. The sensory input was collected in a controlled environment, composed of a conveyor belt, coaxial light source, RGB camera, and laser line-scanning sensor composed of a laser transmitter and two monochrome cameras, the apparatus is illustrated in Figure 9. Figure 9: (a) RGB-D acquisition platform (b) Color image (c) Depth image (d) The contours and categories of objects are detected [38]. Mask recurrent CNN (Mask R-CNN) is the employed model architecture design, with hyperparameter finetuning akin to ResNet101. Thus, equating in the comparison between 4 produced models: RGB, RGB-D concat, RGB-D Ci-add, RGB-D Ci-concat (Figure 10). Reporting improved aforementioned results from RGB model compared to RGB-D Ci-concat of mAP[50:95] 82.15%, referencing to the threshold average for the IOU between 0.5 and Definition of AI-based tools 18 0.95. Limitations for this study can be perceived based on a controlled environment of the deployed system, and lack of severe overlapping and aggregation of CDW, followed by use of high-end energy intensive GPUs (NVIDIA RTX 2080 Ti) as the inference device target. Figure 10: (a) RGB-D concat model (b) RGB-D Ci-add model and RGB-D Ci-concat model [38]. Chen et al. (2022) [60], propose a framework for automatic waste sorting on construction sites using robots. To address the challenges posed by the complex environment and diverse waste types (Figure 11). These robots utilise various technologies such as Simultaneous Localization and Mapping (SLAM), deep learning, and high-precision 3D object pickup strategies to achieve real-time navigation, accurate identification, and stable grasping of CDW. The sensorial input is composed of 2 RGB-D cameras and LiDAR, for CDW recognition and handling, in addition to mapping surrounding navigational area, respectively. Precise object recognition and identification model is based on Mask R-CNN architecture, achieving mAP[0.5:0.95] for segmentation and bounding-box of 0.683 and 0.657 respectively. Where in more challenging conditions (e.g., special density, and light condition) a small degradation can be observed. Definition of AI-based tools 19 Figure 11: System flowchart of the localization module [60]. 4.4. Construction Demolition Waste Satellite Geolocation 4.4.1. Overview Deep learning and satellite technology's combination has brought in disruptive possibilities, which are especially notable in the construction sector. The use of satellite imaging for trash detection and categorization in construction sites and the exploitation of high-resolution satellite multispectral imagery for construction processes serve as two key examples of this synergy. The former is essential for efficient trash circulation, whereas the latter is crucial for activities like identifying plastic functions on construction sites and detecting building footprints. These approaches, which are based on the interaction between satellite data and sophisticated algorithms, not only provide quick fixes but also suggest broader applications and ramifications in the field of building and construction. 4.4.2. Ideal Implementations Since 2018 [61], the realm of satellite imagery analysis has witnessed a significant surge in interest and attention within the research community. This burgeoning interest can be attributed to the myriad of potential applications that satellite imagery offers with the increased interest with Machine learning, especially in the fields of urban planning and construction. As researchers delved deeper into the capabilities of satellite imagery, two applications began to stand out due to their transformative potential for the industry. Firstly, the detection of building footprints from satellite images emerged as a game-changer for urban developers and planners. With the ability to accurately map and monitor the layout of buildings in vast urban areas, city planners could gain invaluable insights into urban sprawl, infrastructure development, and the overall growth trajectory of cities. This not only aids in better resource allocation but also in ensuring sustainable urban development. Showcasing solid technology and results produced by this advancement [62]. Secondly, the identification of plastic functions within land sites became another focal area of research. With the increasing emphasis on sustainable construction practices and waste Definition of AI-based tools 20 management, the ability to pinpoint and categorize different types of plastics on construction sites from satellite images can have profound implications. This can lead to better waste management strategies, recycling initiatives, and overall, an eco-friendlier approach to construction [63]. Together, these two applications underscore the transformative potential of satellite imagery analysis in reshaping the future of urban planning and sustainable construction. 4.4.3. Building Footprint Detection and Methodology High-resolution satellite multispectral imagery has been instrumental in offering insights into urban sprawl, infrastructure development, and construction progress. An ensemble modelling approach, fortified with deep learning enhancements, was adopted. This methodology involved a 2-sigma percentile normalization of the initial dataset, followed by the development of three distinct models that integrated data from OpenStreetMap. The UNet architecture, a brainchild originally conceived for biomedical image segmentation, was repurposed for this endeavour. To bolster its efficacy, batch normalization wrappers were incorporated. Benchmarking against the acclaimed solutions of the SpaceNet 2 competition, the methodology was tested on real satellite multispectral images from diverse urban landscapes, including Vegas, Paris, Shanghai, and Khartoum. The outcomes underscored the potency of the approach, marking a notable stride in building detection accuracy [62]. Figure 12: Example Image from dataset and label [63]. In the context of feature extraction from satellite imagery, the binary distance transformation, as elucidated by M. B. Neiva, A. Manzanera, and O. M. Bruno in "Binary distance transform to improve feature extraction" [62], plays a crucial role, particularly when dealing with diminutive building footprints, as highlighted in the experiment by Prathap, G. and Afanasyev, I [64]. This method emphasizes the intricacies of these footprints, ensuring their precise representation. Specifically, the transformation calculates a positive distance for pixels located inside buildings, while assigning a negative distance to those outside, optimizing the clarity and accuracy of the extracted features. An example can be seen above in Figure 12. Definition of AI-based tools 21 4.4.4. Plastic Function Identification and Methodology A recent 2023 paper titled “Can we use deep learning models to identify the functionality of plastics from space?” [63] shed light on the advancements in satellite analysis for object and pattern identification from space. A two-tiered workflow was devised to discern plastic functions on land surfaces using deep learning in tandem with Sentinel-2 satellite imagery. The first stage involved crafting a classification map to differentiate 10 unique plastic types via spaceborne hyperspectral PRISMA data provided by the Italian Space Agency (ASL). The subsequent stage harnessed deep learning algorithms to allocate functions to these classified plastics, leveraging RGB data from Sentinel-2 images. This process was refined through manual labelling of 1,645 plastic polygons. The Copernicus Sentinel-2 mission consists of a constellation of two polar-orbiting satellites (A/B) that are phased at 180 to each other and placed in the same sun-synchronous orbit. Multispectral images with various spectral and spatial resolutions are provided by the Sentinel multispectral instruments (MSI). The study illuminated five cardinal plastic functions integral to the construction industry: • Construction area plastic cover sheeting • Greenhouse structures • Photovoltaic panels (PVs) • Roofing materials • Sport field floorings CNNs are a highly effective choice for image classification tasks, as they have demonstrated state-of-the-art performance on many benchmark datasets [63]. A comparative analysis of three deep learning models - GoogLeNet, VGGNet, and ResNet - was conducted in the paper. The VGG-13 network emerged as the frontrunner, registering an overall accuracy of 78% on the test dataset. However, the model's prowess varied across specific plastic functions, with PVs and greenhouses achieving F1 scores of 0.85 and 0.77, respectively. Definition of AI-based tools 22 Figure 13: Enlarged subscenes of Sentinel-2 imagery, superimposed with the ground truth map of the respective functions (colours) of plastic materials in polygons, and with filled polygons as a result of function assignment using VGG-13 model. Misclassifications are depicted when colours of polygons and fillings differ (e.g., top b2; bottom right b5) [63]. In zoomed-in parts of Sentinel-2 images, (Figure 13: a1-a5) show ground truth maps with colours representing different plastic materials. In (b1-b5), these areas are filled based on predictions from the VGG-13 model. When the colours of the outlines and fillings don't match, it means there was a mistake in prediction (like in the top of b2 and bottom right of b5). This shows us the potential way we will be needing the images to be labelled in the future. The potential of deep learning models in recognizing plastic functions is evident, yet it's not without its challenges. One of the most significant hurdles is the limited availability of labelled data. For deep learning models to be trained effectively, they require a vast amount of labelled data. Given the diverse nature of plastic materials, especially when factoring in regional variations in their form and surrounding environments, a comprehensive range of samples is essential. However, the process of accurately labelling these samples is timeconsuming and labour-intensive. Definition of AI-based tools 23 Another challenge arises from the model initialization process. The study heavily relied on pre-trained weights, which, while beneficial in enhancing the model's accuracy, limits the input to real colour composites (RGB). Without the advantage of these pre-trained weights, the model's accuracy experiences a significant drop. To move away from the dependency on pre-trained weights, there's a pressing need for more training data. This would not only improve accuracy but also allow for the inclusion of additional spectral bands from Sentinel2 data and insights from hyperspectral analysis. Another consideration is in this paper, the geographical scope of the study, centred around the Shandong Province, poses questions about the model's applicability in diverse regions. While the datasets used were spatially independent, their confinement to a single province might limit the model's adaptability to other regions. Lastly, the model's dependency on hyperspectral imaging for prior knowledge of plastic distributions complicates its broader application. The process of acquiring and analysing hyperspectral data is not only time-intensive but also requires specialized expertise. Moreover, the necessity to consistently align the basic plastic classification map with Sentinel-2 images adds to the operational challenges. Looking ahead, it's imperative that future research addresses these challenges. Potential solutions could involve the development of end-to-end models that negate the need for prior knowledge of plastic distributions or methods to simplify the alignment of classification maps with Sentinel-2 images according to the author. 4.4.5. Expanding Horizons: Satellite Imagery in Construction The use of satellite photography goes far beyond just looking for urban planning and pattern detection. Deep learning approaches have made it easier to carefully extract building footprints from these photos, a feat that was long thought to be challenging [64]. The U-Net design, a breakthrough originally intended for biological image segmentation [65] but now with applications in the construction industry [66], is evidence of this. Looking at the wealth of information, we can say that an innovative solution for the construction industry can be developed as well. In hopes to utilise state-of-the-art Deep learning algorithms to develop a method to give suggestion to the user on the optimal allocation of the site to be able to give a better construction environment. This could include optimised valorisation routes and approximate site allocation. 4.4.6. Limitations on Satellite data and ML Applications Technical Limitations A study revealed that cloud cover significantly reduces the accuracy of satellite imagery, with thicker and darker clouds causing further reductions, impacting applications like environmental monitoring, disaster response, and urban planning [67]. Satellite remote sensing's effectiveness, especially in visible (VIS) and near-infrared (NIR) wavebands, is severely limited by the frequent presence of cloud cover, particularly in polar regions [68]. Persistent cloud cover during disasters can significantly impede post-event analysis by obscuring affected areas when optical data are required [69]. Even when Geostationary Definition of AI-based tools 24 weather satellites cannot monitor the entire planet due to their fixed position above a particular region of the Earth [68]. The accuracy of cloud detection based on deep learning is greatly affected by the number and quality of training samples, emphasizing the importance of selecting many representative and high-quality training samples [70]. Nowcasting warnings based on radar and satellite data is a complex task due to the large volume of data to be analysed. Deploying ML technologies for satellite imagery analysis also requires substantial resources, limiting their accessibility and usage, especially in low-budget contexts. The resource-intensive nature of current approaches necessitates domain knowledge, remote sensing and engineering expertise, access to imagery, and large computational resources [71]. Training deep learning models effectively requires a large amount of labelled data, especially from diverse regions and categories like roofs and sport fields. The spatial limitation of the datasets to one study region may hinder the models' generalizability and their ability to transfer to unknown areas [63]. Operational Limitations The application and synergistic combination of satellite data and machine learning techniques face a range of challenges. Nominally, a prominent issue has been the difficulty in identifying a relevant use case for satellite data that aligns with the project's specific needs. Despite the abundance of satellite imagery available, integration and alignment with overall goals has been a complex endeavour. In response to these challenges, the team is actively investigating the limitations and associated fees of using satellite data. This involves a thorough analysis of the costs and practical constraints to ensure that our approaches are feasible and align with the project's resources and objectives. As progress is made, a commitment to overcoming these obstacles is maintained by the team. Focus is placed on innovating and adapting strategies to efficiently meet the project's goals within the constraints of available resources. 4.5. Conclusion To conclude, automated construction and demolition waste detection and quantification is a critical step towards efficient CDW management. The application of computer vision and multimodal data for waste sorting has been conceptualised and under investigation for over two decades. Driven by the emerging deep learning techniques, the field is currently experiencing an unprecedented development. This chapter provides a critical review of academic research to understand the past, present, and future of the field of deep learning and computer vision based CDW detection and quantification. With primary focus on replicability, performance, and efficiency. Furthermore, the use of satellite imagery and deep learning for the construction industry has ushered in a new era. Construction project planning and execution are undergoing a revolution thanks to the ability to detect, categorize, and even suggest ideal site allocations based on satellite data. The effectiveness of this synergy is demonstrated by the accuracy Definition of AI-based tools 25 with which waste can be located and managed, building footprints can be located, and plastic functions can be identified. As time goes on, there are a plethora of potential uses for this technology in the construction industry. There are countless options available, from waste management to urban planning, infrastructure monitoring to site optimization. The fusion of deep learning and satellite imagery is poised to usher in a technological revolution for the construction industry, which faces unique difficulties and constantly changing demands. As technology continues to advance, it is anticipated that even more refined, efficient, and sustainable solutions will emerge, catering to the multifaceted demands of the construction realm. Definition of AI-based tools 32 Figure 21: Dataset labelling per EU standard. To facilitate the ease of labelling for the expert partners, BUL identified a labelling tool equipped with integrated AI segmentation models, which enable an assisted labelling process. This tool significantly streamlines the initial manual labelling efforts by providing suggestive outlines around distinct materials captured in the handheld camera images, thereby accelerating the categorization of concrete, brick, tile/ceramic, etc. This hybrid approach marries the speed of AI with the precision of human expertise, ensuring that the labels are both accurate and consistent across the dataset. By adopting this tool, the project not only fast-tracks the initial labelling phase but also sets a solid foundation for the semiautomated labelling processes envisioned in the subsequent stages. As the project progresses, the aim is to deploy a semi-automated labelling process by integrating machine learning algorithms. These algorithms can expedite the labelling process by automatically identifying and categorizing waste materials based on the initial labels provided by the expert partners. The semi-automated labelling process not only accelerates the data preparation but also reduces the manual effort required, thereby increasing the efficiency and scalability of the project. In the future, the hope is to continually refine the labelling process through the feedback received from the model's predictions and the expertise of our collaborators. This iterative feedback loop will help in improving the accuracy of the labels and, consequently, the performance of the machine learning models employed for waste identification and categorization. The collected data will be prepared for analysis through cleaning, organizing, and formatting to align with the requisites of the analytical tools and techniques to be employed. This phase may also encompass data augmentation to enrich the dataset, including the accurate linkage between different data collections like logs and geolocation tags. The meticulous labelling process lays a solid foundation for the subsequent phases of the project, ensuring that the models are trained on high-quality, accurately labelled data, which is pivotal for the success of the RECONSTRUCT waste management initiative. Definition of AI-based tools 33 As of the current phase of our project, the task of labelling the satellite data has not been initiated, given that we are in the process of securing commercial access to the highresolution imagery provided by Pleiades. The enhanced resolution offered by Pleiades is expected to significantly contribute to the accuracy and granularity of our data labelling efforts, which in turn, is crucial for the subsequent analysis and machine learning model training. Acquiring this commercial access is a priority, as it sets the stage for a comprehensive and precise labelling process, thereby ensuring that the ensuing stages of the project are built on a solid and reliable data foundation. The anticipation is that once the access to high-resolution imagery from Pleiades is secured, a more detailed strategy for satellite data labelling will be developed and implemented, leveraging the expertise of our collaborative partners and advanced labelling tools. 5.3. AI-based Modules 5.3.1. Classification Module Design Establishing models to classify the distinct types of CDW based on the extracted features from the data, enabling categorization of the waste into predefined classes, thereby aiding in efficacious management and disposal strategies. In the highlight of data complexity and novelty of state-of-the-art algorithms and accuracy requisites, deep learning algorithms such as Convolutional Neural Networks (CNN), YOLO, Faster-RCNN, and Mask R-CNN, are points of interest and take priority in regard to early experimental results as per the literature survey conducted. As a leadup from the established labelling techniques (semantic and instance segmentation), one such framework at the forefront of object detection and instance segmentation is Mask R-CNN. The genesis of Mask R-CNN is deeply rooted in a series of foundational technologies, beginning with the advent of Convolutional Neural Networks (CNNs), which significantly advanced image classification tasks by efficiently processing grid-like topology data. The journey progressed with the development of Region-based Convolutional Neural Networks (R-CNN), which extended the capabilities of CNNs to object detection by introducing region proposal techniques to identify potential object-containing areas within images. However, the computational burden of processing region proposals sequentially emerged as a bottleneck, leading to the inception of Fast R-CNN. Fast R-CNN ameliorated this issue by sharing computation across proposals through ROI Pooling, markedly accelerating the process. Subsequently, Faster R-CNN emerged, replacing the selective search strategy of R-CNN with a Region Proposal Network (RPN), thus achieving quicker proposal generation and enhanced accuracy. Yet, a gap existed as Faster R-CNN lacked the ability to precisely delineate object boundaries, a critical capability for instance segmentation tasks. Mask R-CNN bridged this gap by extending Faster R-CNN, adding a branch for predicting segmentation masks. A key innovation was the RoI Align module, which rectified the misalignment introduced by RoI Pooling, ensuring accurate segmentation. This Definition of AI-based tools 34 development not only retained efficiency and accuracy in object detection but also enabled pixel-level segmentation, marking a significant milestone in computer vision technology. Figure 22: Mask R-CNN architecture illustration [78]. The arrival of Mask R-CNN (Figure 22) has had a profound impact on various domains, notably in the management of CDW. The framework's capability to accurately segment and classify different materials makes it a capable tool for CDW detection. By delineating and identifying various waste materials such as bricks, concrete, wood, and plastics, Mask RCNN facilitates the efficient sorting and recycling processes crucial in modern waste management systems. 5.3.2. Quantification Module Design As per conducted survey and availability of representative data withing the context of quantification using mixed methods associated with primarily computer vison and depth sensing. The application of an end-to-end DL architecture is paramount when transfer learning and scalability is required. Thus, Mask R-CNN implementation is of interest since it comprises three main components: a backbone network for feature extraction, a Region Proposal Network (RPN) for suggesting regions of interest, and two heads for classification/regression and segmentation. A notable feature is the RoI Align technique, which rectifies the misalignment issue in RoI Pooling, ensuring precise alignment between the RoIs and the extracted features for accurate segmentation. Mask R-CNN is trained endto-end with a loss function that combines classification loss, bounding box regression loss, and mask binary classification loss. It has showcased state-of-the-art performance on various benchmarks and is well-suited for CDW quantification as it enables incorporation of additional information related to depth to the CDW samples, I.e., RGB-D (RGB and depth images) algorithmic based approach, as per illustration on Figure 23. Figure 23: Sample of CDW RGB and depth images [38]. Definition of AI-based tools 35 5.3.3. Evaluation and Validation Criteria Evaluation metrics are crucial for assessing the performance of instance segmentation models employed in construction and demolition waste classification. Among the notable metrics are Intersection over Union (IoU), and Mean Average Precision (mAP), where the average precision (AP) is calculated for each class and then averaging the APs over all classes. The precision is the ratio of true positive predictions and is calculated as follows: Where C is the number of classes and APc is the mean precision for class c. mAP, including mAP50 and mAP with a 0.75 IoU threshold, evaluates the accuracy of the segmentation model by comparing predicted segmentation masks with ground truth masks. On the other hand, IoU measures the overlap between predicted and ground truth masks to determine the correctness of predictions. Additionally, Mean IoU (mIoU) provides an average overlap assessment across all classes, calculated as follows: Where C is the number of classes and IoUc is the intersection over union for class c. Furthermore, the confusion matrix offers a detailed breakdown of true positive, true negative, false positive, and false negative predictions across different classes, providing a more granular insight into the model's performance. 5.4. Geolocation Via Satellite Data 5.4.1. Module Design The aim is to leverage satellite imagery for the development of a recommendation system that can provide optimized valorisation routes for construction and demolition sites. Satellite data, owing to its expansive coverage and frequent updates, holds promise in providing valuable insights into site conditions and surrounding infrastructure [79]. This data could be pivotal in making informed decisions regarding the valorisation routes on new sites and aiding existing sites in enhancing their site optimization. We intend to employ a deep learning model to analyse the satellite images and extract meaningful information regarding the sites. Among the models considered, U-Net is a potential candidate due to its proficiency in semantic segmentation tasks. However, recent advancements may have led to the emergence of more efficient models where some build on top of the U-net architecture [80]. Therefore, a thorough examination of the latest literature will be undertaken to ascertain the most suitable model for this task [80]. 5.4.2. Evaluation and Validation Criteria The evaluation and validation of the model are crucial to ensure the reliability and accuracy of the recommendations provided. Common evaluation metrics such as Precision, Recall, and F1-Score (F1) could be considered to measure the model's performance on a validation dataset. Moreover, Intersection over Union (IoU) might be employed to gauge the accuracy of the semantic segmentation, which is pivotal for accurate geolocation and route Definition of AI-based tools 36 optimization [80]. With overall accuracy (OA) and Kappa coefficient to be used to evaluate the final identification results [80]. The formulas to be used are as follows: In the future, the hope is to conduct a comprehensive validation by comparing the model's recommendations with ground-truth data collected from the SOR site. This comparison could provide a clear indication of the model's efficacy and the practicality of the recommendations provided. Additionally, feedback from domain experts and stakeholders could be incorporated to further refine the model and enhance the recommendation system [81]. Definition of AI-based tools 37 6. WP1 Workflow 6.1. Introduction As for the current progression and workflow of our research project, RECONSTRUCT, the initial phase focused on understanding the problem space and collecting preliminary data, while the ongoing second phase involves detailed planning and collaboration for long-term data acquisition and model implementation working towards the final product to be integrated with SIMBIOSY’s (SYM) platform. 6.2. First Phase 6.2.1. Conducting Literature Review Brunel University London (BUL) conducted a comprehensive literature review on existing solutions in the Construction and Demolition Waste (CDW) domain. This review was instrumental in identifying various approaches that have been previously employed in this space. These approaches and the detailed findings from the literature review can be found in the Literature Review section of this document, offering readers a deeper insight into the context and background of the project. 6.2.2. Initial Planning and Data Collection The initial phase began with engaging Spanish partners Sorigue (SOR) and COMSA (COM) to understand the problem space and the construction site layout to pinpoint the approach to be chosen based on the literature conducted before. Following these discussions, UNIVERSITA POLITECNICA DELLE MARCHE (UNIVPM) conducted a site survey in Spain, where they evaluated the site and gathered initial data essential for developing early prototype AI models. 6.2.3. Data Processing and Model Exploration Upon returning from Spain, the BUL team commenced labelling the acquired data, focusing on a semantic segmentation task, where they trained the Spanish Partners with semiautomated tools and provided support along the way to any queries. 6.3. Second Phase 6.3.1. Planning and Collaboration The second phase involved planning for long-term data acquisition, which included selecting appropriate sensors and discussing practical issues like network connectivity and electricity with Spanish partners. On the other side, Brunel University London (BUL), leading Work Package 1 (WP1), initiated dialogue with VRIJE UNIVERSITEIT BRUSSEL (VUB) to align on the needs of the Brussels site, where the final demonstration will take place. Additionally, BUL included the WP4 lead, SOCIETAT ORGÀNICA (SOO), in the discussions, leveraging Definition of AI-based tools 38 their experience with the Brussels site and their role in utilizing the data produced by BUL's AI models. Definition of AI-based tools 39 7. Conclusions In this project, we have embarked on a path to innovatively address Construction and Demolition Waste (CDW) management through automated detection and quantification leveraging deep learning, computer vision, and route valorisation via satellite imagery. The envisioned framework considers to capitalizes on an array of contemporary machine learning architectures like Convolutional Neural Networks (CNN), YOLO, Faster-RCNN, and Mask R-CNN for the classification and quantification of CDW and the potential use of U-net derivative for the satellite valorisation route recommendation. Through rigorous data collection from various sources including satellite imagery, groundbased observations, and thermal imaging, we aimed to build a robust dataset that can fuel the precise training of these models. However, the journey thus far has unveiled both strides and setbacks which are vital for moulding the subsequent phases of this project. 7.1. Feedback and Refinement In this crucial phase of the project, the feedback loop generated through our collaborative partners proved to be invaluable, shedding light on the multifaceted challenges that lay ahead facilitated by Spanish survey, particularly in the realm of data labelling. Initially, an unexpected delay in the data collection from the Spanish sites significantly impinged on our ability to move forward with constructing a prototype for the classification and quantification modules. This delay presented a tangible setback, underscoring the critical dependency of the project milestones on the timely availability and accuracy of data. Moreover, our anticipation to augment the dataset with high-resolution imagery from Pleiades faced a hurdle due to delays in securing commercial access to the imagery. This delay not only paused the labelling process but also underscored the complexity and the prerequisite resources essential for obtaining high-quality data, pivotal for the accuracy and reliability of our envisioned models. The feedback received from our collaborative partners underscored the pivotal role of accurate data labelling in building reliable and effective models. It highlighted the necessity for a well-orchestrated, comprehensive strategy to overcome the labelling bottleneck, which is now understood to be a critical path in our project timeline. The insights gathered through this feedback loop have been instrumental in not only understanding the intricacies and dependencies intertwined in the data labelling process but also in re-evaluating and finetuning our project plan to better align with the ground realities and the resources at our disposal. 7.2. Future Work Looking ahead, the roadmap is delineated with several pivotal milestones. Foremost is the resolution of the data labelling impediment which is fundamental for advancing to the prototype development phase. The future envisions not only the creation of a reliable prototype but also the continuous refinement of the models through an iterative feedback loop with our expert partners. Additionally, a more detailed strategy for satellite data labelling Definition of AI-based tools 40 will be developed to ensure a solid foundation for the machine learning models. The aspiration to expand the data collection to more geographical locations and integrating more advanced labelling and machine learning techniques reflects the evolutionary nature of this project. The end goal remains to contribute a robust, efficient, and scalable solution to the CDW management realm which will be integrated to the IRIS SaaS Digital Platform and demonstrate the solution at the Belgium site. This journey, albeit with its share of challenges, has set a robust groundwork and the lessons learned are instrumental for steering the project towards its envisioned goal. Through persistent efforts, collaborative engagements, and an iterative refinement approach, the horizon seems promising for achieving a breakthrough in automated CDW management. Definition of AI-based tools 41 8. References 1. Bao, Z., Lu, W., Chi, B., Yuan, H. and Hao, J., 2019. Procurement innovation for a circular economy of construction and demolition waste: Lessons learnt from Suzhou, China. Waste Management, 99, pp.12-21. 2. W. Lu, J. Lou, C. Webster, F. Xue, Z. Bao, B. Chi Estimating construction waste generation in the Greater Bay Area, China using machine learning Waste Manag., 134 (2021), pp. 78-88 3. Hoornweg, D. and Bhada-Tata, P., 2012. What a waste: a global review of solid waste management. 4. G. Rodríguez, C. Medina, F.J. Alegre, E. Asensio, M.I. Sánchez de Rojas, Assessment of Construction and Demolition Waste plant management in Spain: in pursuit of sustainability and eco-efficiency,Journal of Cleaner Production, Volume 90,2015,Pages 16-24,ISSN 0959-6526,https://doi.org/10.1016/j.jclepro.2014.11.067. 5. The Brussels Times (June 2022). https://www.brusselstimes.com/236071/brusselsconstruction-waste-increases-along-with-recyclingrates#:~:text=Friday,%2010%20June%202022%20Credit:,La%20Capitale%20reported 6. Yuan, L., Lu, W. and Xue, F., 2021. Estimation of construction waste composition based on bulk density: A big data-probability (BD-P) model. Journal of Environmental Management, 292, p.112822. 7. Faibish, S., Bacakoglu, H. and Goldenberg, A.A., 1997, April. An eye-hand system for automated paper recycling. In Proceedings of International Conference on Robotics and Automation (Vol. 1, pp. 9-14). IEEE. 8. C. Srinilta, S. Kanharattanachai Municipal solid waste segregation with CNN 2019 5th International Conference on Engineering, Applied Sciences and Technology (ICEAST) (2019), pp. 1-4 9. J. Yang, Z. Zeng, K. Wang, H. Zou, L. Xie GarbageNet: a unified learning framework for robust garbage classification IEEE Trans. Artif. Intell. (2021) 10. F. Aziz, H. Arof, N. Mokhtar, N.M. Shah, A.S.M. Khairuddin, E. Hanafi, M.S.A. Talip Waste level detection and HMM based collection scheduling of multiple bins PLoS One, 13 (2018) 11. T.J. Lukka, T. Tossavainen, J.V. Kujala, T. RaikoZenRobotics Recycler–Robotic sorting using machine learning Proceedings of the International Conference on Sensor Based Sorting (SBS) (2014), pp. 1-8 12. G. Sauve, K. Van Acker The environmental impacts of municipal solid waste landfills in Europe: a life cycle assessment of proper reference cases to support decision making J. Environ. Manag., 261 (2020), Article 110216 13. C. Bircanoğlu, M. Atay, F. Beşer, Ö. Genç, M.A. Kızrak RecycleNet: intelligent waste sorting using deep neural networks 2018 Innovations in Intelligent Systems and Applications (INISTA) (2018), pp. 1-7 14. Z. Chen, Z. Hebin, W. Yanbo, L. Binyan, L. Yu A vision-based robotic grasping system using deep learning for garbage sorting 2017 36th Chinese Control Conference (CCC) (2017), pp. 11223-11226