Advancing unsupervised anomaly detection with normalizing flow and multi-scale ensemble learning

Contents lists available at ScienceDirect Engineering Applications of Artificial Intelligence journal homepage: www.elsevier.com/locate/engappai Research paper Advancing unsupervised anomaly detection with normalizing flow and multi-scale ensemble learning Miguel Campos-Romero a,b,∗, Manuel Carranza-García b, José C. Riquelmeb aLMI Technologies Inc., 9200 Glenlyon Parkway, Burnaby, V5J 5J8, British Columbia, Canada bDivision of Computer Science, University of Sevilla, Av. Reina Mercedes, s/n, Seville, 41012, Spain ARTICLE INFO Keywords: Anomaly detection Novelty detection Deep learning Normalizing flow Unsupervised learning ABSTRACT Visual anomaly detection plays a crucial role in manufacturing to ensure product quality by identifying image patterns that deviate from the expected ones. Existing methods that rely on distribution estimation struggle with the complexity of real-world images, resulting in complex and inefficient procedures. This study leverages normalizing flow techniques to address the cold start anomaly detection problem, where no prior examples of anomalies are available during the training phase. In such scenarios, models must learn exclusively from defect-free images and still accurately identify anomalies. We propose a novel unsupervised multi-scale and multi-semantic normalizing flow model, enhanced with an ensemble of neural networks, to detect anomalies based on their feature distributions. Our model estimates the likelihood of non-defective features, identifying anomalies as out-of-distribution values. Extensive experiments on three state-of-the-art anomaly detection datasets demonstrate that our proposal achieves superior AUROC performance and improves computational efficiency compared to existing approaches. Furthermore, we validate the robustness and adaptability of our proposal through low-shot training experiments using only 20% of available training data, highlighting its potential as an efficient solution for cold start anomaly detection. 1. Introduction Anomaly detection in computer vision refers to identifying pattern images that deviate from what is considered standard or expected. This task is crucial in industrial automation, ensuring product quality in large-scale production while maintaining low operational costs. Although humans can detect visual anomalies naturally, computational methods face the challenge of the cold start problem, where there is no prior knowledge or examples of anomalies for model training. To address this problem, current state-of-the-art unsupervised approaches use deep representations of nominal defect-free images for training. These techniques include the use of autoencoders (Bergmann et al.,2019;Tang et al.,2020), generative models (Song et al.,2021; Akcay et al.,2019), embeddings (Defard et al.,2021), or distribution analysis approaches (Gudovskiy et al.,2022;Rudolph et al.,2021). In this context, the training process involves learning the characteristics of defect-free images. During inference, if an image exhibits features that deviate from these learned characteristics, it is detected as a potential anomaly. An evaluation of the latest generation of models reveals limitations that impact response rate due to high computational costs and difficulty in accurately identifying various types of anomalies. For instance, ∗Corresponding author at: Division of Computer Science, University of Sevilla, Av. Reina Mercedes, s/n, Seville, 41012, Spain. E-mail addresses: [email protected] (M. Campos-Romero), [email protected] (M. Carranza-García), [email protected] (J.C. Riquelme). autoencoders often produce many false positives due to information loss in the bottleneck, as shown in Fig. 1. Alternative patch-based reconstruction approaches aim to address these limitations but can miss anomalies with long-range dependencies because they focus on locally restricted regions, as evidenced in Fig. 2 (Bergmann et al.,2020). Adversarial generative models (Yang et al.,2023) and embedding techniques (Roth et al.,2022) also have high computational demands, while existing distribution-based methods (Rippel et al.,2020) can still benefit from further enhancements to effectively handle the complex distribution of images for anomaly detection. In this work, we propose the InAI (Industrial Anomaly Inspection) method to address these challenges. Our approach centers on framing the cold start anomaly detection problem as a maximum likelihood estimation (MLE) task focused on nominal images. This entails assigning higher likelihood scores to new defect-free images than those containing defects, facilitating clear differentiation. Our proposed architecture follows a novel encoder–decoder scheme. The encoder is a convolutional neural network (CNN) that captures global semantic information with a complex and unknown distribution. The decoder is another CNN that leverages normalization flow (NF) techniques to map the distribution from the base space to a target https://doi.org/10.1016/j.engappai.2024.109088 Received 5 April 2024; Received in revised form 17 June 2024; Accepted 1 August 2024 Engineering Applications of Articial Intelligence 137 (2024) 109088 Available online 16 August 2024 0952-1976/© 2024 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ). M. Campos-Romero et al. Fig. 1. Example of qualitative results of an autoencoder model on the MVTec LOCO AD dataset. The first row displays a defect-free image and its reconstruction; the second row shows a defective image with a green bounding box around the defect and its reconstruction. The autoencoder struggles with detail accuracy, causing false positives in the defect-free image and missing the true defect in the defective image. latent Gaussian space with well-known characteristics. This enables optimal MLE optimization during training and accurate likelihood score assignment during inference. We adopt a semantic multi-scale approach, utilizing different extraction points in the encoder to reinforce result robustness with varying levels of abstraction and receptive fields. Finally, we also employ an ensemble of encoders to improve anomaly detection performance. To our knowledge, this research represents the first successful integration of NF, multi-scale, multi-semantic, and model ensemble techniques into a single and efficient end-to-end two-dimensional approach. Our method establishes a new benchmark in processing speed and anomaly detection metrics in three state-of-the-art datasets: MVTec Anomaly Detection (MVTec AD) (Bergmann et al.,2021), MVTec Logical Constrains Anomaly Detection (MVTec LOCO AD) (Bergmann et al., 2021), and Surface Defect Saliency of Magnetic Tile (SDMT) (Huang et al.,2020). Our method is also compared to other state-of-the-art methods in a low-shot training regime, where only 20% of available training data is used. The results demonstrate that InAI outperforms current methods while preserving high computational efficiency. In summary, the key contributions of this work are: •A new unsupervised anomaly detection method using state-of-the- art NF techniques, outperforming existing approaches at lower computational requirements. •A novel integration of ensemble, semantic multiscale, and NF techniques in a flexible encoder–decoder architecture, adaptable to any existing convolutional network configuration. •A comprehensive experimental study optimizing encoder settings for NF functions in anomaly detection, demonstrating superior adaptability and reliability in low-shot training regimes. This study is organized as follows: Section 2reviews the relevant literature. Section 4elaborates on the proposed methodology. Section 5 outlines the materials and metrics used for testing our method and reports our study’s results. Section 7presents the main findings of this work, discusses potential limitations, and suggests new avenues for future research. Finally, Section 8concludes the article. 2. Related work Anomaly detection plays a critical role in various domains, including finance (Andor and Bohák,2017;Ardizzi et al.,2018), cybersecurity (Dixit and Silakari,2021;Anthi et al.,2021), and healthcare (Luo et al.,2023;Hammad et al.,2022). Traditionally, anomaly detection has relied on supervised learning paradigms, requiring careful data preparation to train a mathematical model that can distinguish between anomalous and non-anomalous data (Kanishima et al.,2022;Fernandez et al.,2022). In academia, the absence of datasets explicitly designed for anomaly detection has led to the adaptation of other conventional datasets for this task. For instance, CIFAR-10 (Krizhevsky et al.,2009) and MNIST (LeCun et al.,1998) have been widely used for anomaly detection by excluding certain classes from the training phase to use them as anomalies. The release of the MVTec AD dataset (Bergmann et al.,2020) marked an important milestone in the field, enabling the development of robust measures that resulted in substantial advances in anomaly detection methodologies. Although the literature on anomaly detection is extensive, it can be categorized into four groups: methods based on reconstruction, embeddings, generative adversarial networks (GANs), and distribution analysis. Table 1 provides an overview of key methods within each group, aiding in understanding the distinct characteristics and relationships among various approaches. 2.1. Reconstruction-based methods Reconstruction-based anomaly detection techniques utilize a studentteacher scheme, in which the student network is trained to imitate a teacher’s output using defect-free images. This methodology facilitates the identification of anomalies in regions where the student’s output diverges from the teacher’s output. The U-ST method (Bergmann et al., 2020) employs multiple encoders with different receptive fields to ensure detection across a wide range of anomalies. GCAD (Bergmann et al.,2022a) incorporates nine models to enhance the potential for divergence, thereby facilitating anomaly detection. The AST model (Rudolph et al.,2023) refines this approach by introducing asymmetric learning, where the student and teacher have asymmetric architectures, increasing divergence in the presence of defects. The ITrain model (Cai et al.,2023) introduces the use of an inductive transformer as a student, designed to encode and decode the teacher’s output, inducing high error in defect regions. AEKD (Wu et al.,2024a) integrates an autoencoder as a student, which fails to reconstruct anomalous regions correctly, making them easier to identify. TFA-Net (Luo et al.,2024) uses a nominal feature template for anomaly detection, ensuring that features analogous to the template are used, thereby identifying defects as empty regions. Alternative approaches improve performance by enhancing existing methods. For instance, Ristea et al. (2022) proposes a generic selfsupervised predictive convolutional attentive block, enhancing any method by minimizing the reconstruction error of the masked area in the receptive field. 2.2. Embedding-based methods Embedding-based approaches leverage K-nearest neighbors (kNN) techniques (Peterson,2009) to compute relative distances to nominal images, distinguishing between defective and non-defective images. SPADE (Cohen and Hoshen,2020) pioneered this technique by introducing a multilevel semantic memory bank, calculating the maximum Euclidean distance from the center of the nominal feature cluster for a new image to be considered defect-free. PaDiM (Defard et al.,2021) proposes a patch-based approach, replacing Euclidean distance with Mahalanobis distance and showing that a random sub-manifold suffices for correct anomaly detection, thereby saving computational costs. Patchcore (Roth et al.,2022) reverts to Euclidean distance and employs a multilevel memory bank with the concept of submanifolds (Defard Engineering Applications of Articial Intelligence 137 (2024) 109088 2 M. Campos-Romero et al. Fig. 2. Example of qualitative results of a patch-based model on the MVTec AD dataset (Bergmann et al.,2020). The left image shows a printing defect, while the right displays inference results on patches with receptive fields (RF) of∈ {17,33,65}. Larger receptive fields cause the model to miss the defect. et al.,2021), integrating active learning (Sener and Savarese,2018) for more robust anomaly detection with higher efficiency. CFA (Lee et al.,2022) models feature points using a hypersphere approach, grouping features by density to estimate a minimum radius separating non-defective from defective features. A refined version of Patchcore also emerges with ensemble learning techniques to enhance robustness (Roth et al.,2022). SSMRKD (Tong et al.,2023) uses reverse knowledge distillation, self-supervised learning, attention mechanisms, and a memory bank to improve anomaly detection in subsequent kNN searches. 2.3. Generative adversarial methods Generative adversarial methods synthesize anomalies from nominal images, facilitating a pseudo-supervised learning paradigm for anomaly detection. CutPaste (Li et al.,2021) generates synthetic anomalous images by cutting and overlaying nominal image regions. DRAEM (Zavrtanik et al.,2021) advances anomaly synthesis by incorporating Perlin noise, adding complexity for more realistic and diverse anomalies. MemSeg (Yang et al.,2023), a U-Net architecture with attention mechanisms, multi-scale feature fusion, and a memory bank, also introduces a sophisticated anomaly generation method based on DRAEM, with a transparency factor for better synthesis. 2.4. Distribution-based methods Distribution-based methods focus on the distribution of nominal images, operating on the premise that defect-free images differ in distance from anomaly images relative to a theoretical Gaussian distribution. Mahalanobis-AD (Rippel et al.,2020) fits a multivariate Gaussian distribution to nominal data, detecting anomalies using the Mahalanobis distance. Despite using kNN methods, Defard et al. (2021) contributes by presenting Mahalanobis distance at the image patch level and demonstrating the effectiveness of using a sub-manifold. Recent techniques have integrated flow-based networks, modeling nominal data as a standard distribution. DifferNet (Rudolph et al.,2021) was the first to adopt this methodology. CFlow (Gudovskiy et al.,2022) conditions the model on a positional encoding vector (Vaswani et al.,2017), extending DifferNet for segmentation. CS-Flow (Rudolph et al.,2022) adopts a convolutional strategy to improve segmentation, resizing input data to three scales to enhance semantic information. AFlow (Wu et al.,2024b) incorporates SE attention mechanisms (Hu et al.,2018) and refines features for anomaly detection. Finally, variational autoencoders (Ikeda et al.,2018) can estimate image density for anomaly detection by maximizing the evidence lower bound (ELBO) of a nominal distribution. However, they may face challenges in fully capturing the complexities of the data distribution due to their approximation of the true posterior. 2.5. Our proposed method: InAI Our InAI methodology aligns closely with distribution-based approaches, leveraging normalizing flow (NF) techniques. NF models belong to the probabilistic generative model family and were introduced by Tabak and Turner (2013). Dinh et al. (2015) extended this research with a novel method for estimating nonlinear independent components in images. Dinh et al. (2017) marked significant advances in flow-based networks, achieving robust imaging and density estimation results. Kingma and Dhariwal (2018) established new imaging and density estimation benchmarks. Our interest in NF arises from its ability to model flexible distributions and accurately estimate image data density, modeling the nominal distribution in a well-characterized Gaussian space for effective anomaly detection. In contrast to our approach, existing NF-based methods, such as DifferNet and CFlow, convert multidimensional features into vectors, losing the original spatial relationships and limiting segmentation performance. Although CFlow extends DifferNet for segmentation by conditioning the model on positional encoding vectors, this increases computational costs. CS-Flow preserves spatial relationships with a convolutional strategy, resizing images to three scales to capture multi-scale semantic information, significantly increasing computational demands. Furthermore, generative adversarial methods perform well in detecting structural anomalies involving unexpected patterns but face limitations with logical anomalies (Bergmann et al.,2022a), which are present in the second dataset addressed in this study. This limitation is due to adversarial learning methods’ difficulty in effectively synthesizing logical anomalies for training. Our InAI proposal learns exclusively from nominal images, interpreting anomalies as deviations, significantly expanding its applicability. Despite the effectiveness of kNN-based methods, they are computationally demanding, especially with large datasets. PatchCore tries to address it with active learning (Sener and Savarese,2018) but is insufficient in practice. Moreover, existing reconstruction-based methods must use several student networks to reduce false positives, increasing computational costs and complicating training with multiple models. In contrast, as shown in Section 5, InAI offers superior performance, providing a more straightforward and robust alternative. In summary, our proposed InAI method is distinguished in several key aspects from existing methods: Use of NF for enhanced anomaly detection. Unlike other distribution-based methods (Rippel et al.,2020;Defard et al.,2021), our approach employs NF to estimate the probability of an image having defects within a Gaussian latent space. This imparts well-known attributes to images, facilitating more precise anomaly detection. Preservation of spatial relationships. Unlike other flow-based methods (Rudolph et al.,2021) or embedding-based methods (Lee et al.,2022), our method maintains the original spatial relationships of the images. This eliminates the need to reconstruct local positions in the final output, thereby preserving the integrity of spatial information. Improved efficiency for real-time applications. InAI adopts a fully convolutional scheme like CS-Flow but eliminates the computational overhead of resizing multiple inputs, cross-referencing, and unifying multiple scales. Unlike methods such as U-ST or GCAD, which rely on a multi-student approach, InAI operates as a single network. Other methods, such as PatchCore, PaDiM, or Mahalanobis-AD, require the computation of Euclidean or Mahalanobis distances to determine the nearest neighbors of each input image relative to a large bank of nominal features. This process requires many computations, especially if the training set is large. In contrast, our InAI methodology Engineering Applications of Articial Intelligence 137 (2024) 109088 3 M. Campos-Romero et al. Table 1 Overview of existing anomaly detection approaches. Category Method and description Reconstructionbased U-ST (Bergmann et al.,2020): Uses students CNNs with different receptive fields. GCAD (Bergmann et al.,2022a): Extends U-ST with complex CNN architectures. AST (Rudolph et al.,2023): Introduces asymmetric learning. ITrain Cai et al. (2023): Utilizes student transformers. AEKD (Wu et al.,2024a): Employs student Autoencoders. TFA-Net (Luo et al.,2024): Introduces nominal feature template. Embeddingbased SPADE (Cohen and Hoshen,2020): Introduces using a memory bank. PaDiM (Defard et al.,2021): Combines patch-based approach with Mahalanobis distance. Patchcore (Defard et al.,2021): Extends SPADE with active learning. CFA (Lee et al.,2022): Extends PaDiM with a hypersphere approach. SSMRKD (Tong et al.,2023): Applies reverse knowledge distillation and self-attention mechanisms. Generative Adversarial CutPaste (Li et al.,2021): Generates synthetic anomalous images through region cutting and overlaying. DRAEM (Zavrtanik et al.,2021): Introduces Perlin noise for anomaly synthesis. MemSeg (Yang et al.,2023): Combines kNN and anomaly synthesis mechanisms. Distributionbased Mahalanobis-AD (Rippel et al.,2020): Introduces multivariate Gaussian distribution for anomaly detection. DifferNet (Rudolph et al.,2021): Adopts flow-based networks for anomaly detection. CFlow (Gudovskiy et al.,2022): Introduce positional encoding to extend DifferNet for segmentation. CS-Flow (Rudolph et al.,2022): Extends DifferNet with a cross-convolutional strategy for segmentation. AFlow (Wu et al.,2024b): Incorporates SE attention mechanisms. InAI (ours): Incorporates dequantization, new ActNorm2D normalization, and advanced affine transformation. avoids these complexities, positioning it as an interesting candidate for real-time anomaly detection applications. Broader anomaly detection. Unlike adversarial generative models such as MemSeg, InAI learns anomaly detection using exclusively defect-free nominal images, conceptualizing anomalies as deviations from these data. This approach significantly broadens the model’s applicability, as it does not rely on prior knowledge of specific anomalies. Component refinement. Our findings indicate that complex strategies such as spatial resizing (Rudolph et al.,2022) or positional encoding (Gudovskiy et al.,2022) may limit the performance of NF for anomaly detection. Therefore, we focused on optimally integrating components that have demonstrated high performance in density estimation (Ho et al.,2019;Hoogeboom et al.,2021;Kingma and Dhariwal,2018), leading to new state-of-the-art performance in unsupervised anomaly detection. 3. Theoretical background This section introduces the main concepts used in our study: maximum likelihood estimation (MLE), Kullback–Leibler (KL) divergence, and the change of variable formula. We also explain how these concepts enable training a model using normalizing flow techniques through MLE in an unsupervised manner. 3.1. MLE Maximum Likelihood Estimation (MLE) is a method used to estimate the parameters of a statistical model. Given a set of observations {𝑥1, 𝑥2,…, 𝑥𝑛}and a probability distribution 𝑝(𝑥|𝜃)parameterized by 𝜃, the likelihood function 𝐿(𝜃)is defined as: 𝐿(𝜃) = 𝑛 ∏ 𝑖=1 𝑝(𝑥𝑖|𝜃)(1) The log-likelihood simplifies computation: log 𝐿(𝜃) = 𝑛 ∑ 𝑖=1 log 𝑝(𝑥𝑖|𝜃)(2) The objective of MLE is to find the optimal parameter 𝜃that maximizes the log-likelihood function:  𝜃= arg max 𝜃 𝑛 ∑ 𝑖=1 log 𝑝(𝑥𝑖|𝜃)(3) 3.2. KL divergence Kullback–Leibler (KL) Divergence is a measure of how one probability distribution 𝑄diverges from a reference probability distribution 𝑃. It is defined as: 𝐷KL[𝑃∥𝑄] = E𝑃(𝑥)[log 𝑃(𝑥) 𝑄(𝑥)](4) The KL divergence is non-negative and equals zero only when the distributions 𝑃and 𝑄are identical. 3.3. Change of variable formula The change of variable formula is essential for transforming an observed variable 𝑥∈𝑋into a latent variable 𝑧∈𝑍using a bijective (invertible) transformation 𝑓∶𝑋→𝑍. If 𝑝𝑍(𝑧)is the prior probability distribution of 𝑧, the probability distribution of 𝑥,𝑝𝑋(𝑥), can be derived using the change of variable formula (Dinh et al.,2015,2017;Bell and Sejnowski,1995;Deco and Brauer,1994): 𝑝𝑋(𝑥) = 𝑝𝑍(𝑓(𝑥)) |||||det (𝜕𝑓(𝑥) 𝜕𝑥𝑇)|||||(5) Here, ||||det (𝜕𝑓 (𝑥) 𝜕𝑥𝑇)||||is the absolute value of the determinant of the Jacobian matrix of the transformation 𝑓. 3.4. Relation to our framework Training an NF model involves minimizing the divergence between the model and target distribution. This can be achieved by training the model using MLE. We often minimize the reverse KL divergence because it allows us to sample easily from the model distribution 𝑝𝑋(𝑥;𝜃) and evaluate it with respect to the target density 𝑝∗ 𝑋(𝑥), which in our case is a standard Gaussian distribution. The reverse KL divergence is given by: 𝐷KL[𝑝𝑋(𝑥;𝜃) ∥ 𝑝∗ 𝑋(𝑥)] = E𝑝𝑋(𝑥;𝜃)[log 𝑝𝑋(𝑥;𝜃) 𝑝∗ 𝑋(𝑥)] =E𝑝𝑋(𝑥;𝜃)[log 𝑝𝑋(𝑥;𝜃) − log 𝑝∗ 𝑋(𝑥)] =E𝑝𝑋(𝑥;𝜃)[log 𝑝𝑋(𝑥;𝜃)]+𝐶(6) Since the target density 𝑝∗ 𝑋(𝑥)does not depend on the model parameters 𝜃, it can be replaced by a constant 𝐶. This shows that the term inside the reverse KL is just the logarithm of the change of variable formula. Consequently, fitting the model distribution 𝑝𝑋(𝑥;𝜃)to the Engineering Applications of Articial Intelligence 137 (2024) 109088 4 M. Campos-Romero et al. target 𝑝∗ 𝑋(𝑥)by minimizing the reverse KL divergence is equivalent to training the model by maximizing the likelihood. Therefore, we can train the model on an observable dataset of defect-free images to identify samples with defects in the out-of-distribution values. 4. Methodology This section elaborates on our proposed Industrial Anomaly Inspection (InAI) method, which is presented in Fig. 3. In summary, InAI is a CNN model incorporating NF techniques within an Encoder–Decoder architecture to detect anomalies. The InAI methodology begins with the Encoder, which transforms the input images into deep feature representations, decreasing the computational cost and improving the efficiency compared to processing raw images. The Decoder then maps these encoded feature map representations from their original space, with an unknown distribution, into a Gaussian latent space, with a well-known distribution. The input image data are RGB images. We employ the notation 𝑑 to denote an individual image, characterized as a three-dimensional matrix in R3×ℎ×𝑤. In this representation, ℎand 𝑤correspond to the height and width, while the three refer to the image encoding. We use 𝐷to denote the entire dataset, while 𝑑∈ {𝐷𝑡𝑟𝑎𝑖𝑛, 𝐷𝑡𝑒𝑠𝑡}specifies if an image belongs to the training or testing dataset. The training data is composed of 𝑁pairs of data labels, denoted by (𝑑𝑖, 𝑦𝑖)𝑁 𝑖=1, where 𝑑𝑖is a batch of input images with |𝑑𝑖|≥1. In this set, ∀𝑑∈𝐷𝑡𝑟𝑎𝑖𝑛 ∶𝑦= 0 indicates the images have no defects. In contrast, the testing data 𝐷𝑡𝑒𝑠𝑡 is composed by 𝑇data label mask triples (𝑑𝑡, 𝑦𝑡, 𝑚𝑡)𝑇 𝑡=1, with (𝑦𝑡, 𝑚𝑡) ∈ {0,1}. Here, ∀𝑑∈𝐷𝑡𝑒𝑠𝑡 has |𝑑𝑡|= 1, with one unique associated label 𝑦𝑡and mask 𝑚𝑡. The label can have either zero (0) or one (1) value indicating the presence of defects in the image. The mask indicates the anomalous regions within the image. 4.1. Encoder The Encoder is the initial part of our proposed model and plays a crucial role in generating high-level feature maps for the Decoder. It consists of a CNN pre-trained on ImageNet (Krizhevsky et al.,2017) that produces deep and meaningful representations of the input images. These representations, containing information about the image’s color, shape, edges, and other relevant features, serve as input data for training our Decoder. CNNs trained on real-world natural images, such as those in the ImageNet dataset, extract robust features for image analysis. The deeper layers of these CNNs tend to encode more specific semantic information related to the problem for which they were initially trained. In contrast, intermediate layers encode more generic, semantically meaningful features, which is ideal for training our Decoder. This approach aims to address the limitations highlighted in Kirichenko et al. (2020), where it is shown that NF tends to focus on fine-grained details, degrading anomaly detection. Given the lack of prior information about anomalies, training a CNN from scratch is not feasible. However, using a pre-trained ImageNet CNN enables us to capture high semantic value information efficiently, enhancing the NF model’s performance by avoiding focusing on irrelevant details. Consequently, the Encoder is specifically designed to optimize the generation of these deep representations. The internal engine of the Encoder is composed of selected CNN models from the EfficientNet (Tan and Le,2019) and ResNet (Zagoruyko and Komodakis,2016;He et al., 2016) architectures. These pre-trained CNNs are initialized with parameters 𝜆that minimize the Kullback–Leibler (KL) divergence (Kullback, 1951)𝐷𝐾𝐿(𝑃∥𝑄)between the training distribution 𝑄represented by ImageNet and the Encoder distribution 𝑃. Given an input 𝑑∈𝐷, the Encoder processes it through its internal CNN, generating a set of deep features 𝐹= {𝑥𝑖}𝑗 𝑖=1, with 𝑗denoting the maximum number of feature maps produced by the CNN. Each feature map 𝑥𝑖∈𝐹is represented as 𝑥𝑖∈R𝑐𝑖×ℎ𝑖×𝑤𝑖, with 𝑐𝑖,ℎ𝑖, and 𝑤𝑖 representing the number of channels, height, and width of the feature map at hierarchical level 𝑖. Not all feature maps in 𝐹are utilized by the Decoder. Our approach involves selecting a subset of these features, denoted as 𝑋, based on an extensive experimental study. Therefore, 𝑋 ⊆ 𝐹 and 𝑋≠∅. This subset 𝑋is represented as {𝑥𝑖∈𝑋}𝑗∗ 𝑖=1, where 𝑗∗is the cardinality of 𝑋. The selection of the optimal 𝑋from 𝐹is detailed in Section 5. In summary, the Encoder function performs two main steps. First, it uses its internal CNN to extract deep features 𝐹from the input image 𝑑. Second, it selects a subset 𝑋from these features to be used as input for the Decoder. This methodology ensures that only the most relevant and informative features, as determined by our experimental study, are used to train the Decoder, hence enhancing the performance and efficiency of our anomaly detection method. 4.2. Decoder The Decoder is a probabilistic generative model based on normalizing flows. NF is an application of diffeomorphism parameterized by a neural network that allows building complex distributions, transforming simpler distributions through a series of invertible and differentiable mappings (Kingma and Dhariwal,2018;Dinh et al., 2017;Durkan et al.,2019). This bijective mapping ensures a one- to-one correspondence, enabling the transformation of the complex distribution inherent in the feature maps extracted by the Encoder into a Gaussian distribution. The well-known properties of Gaussian distributions facilitate effective anomaly detection. Let 𝐺be our Decoder mapping the data 𝑋= {𝑥𝑖}𝑗∗ 𝑖=1, 𝑥𝑖∈R𝑐𝑖×ℎ𝑖×𝑤𝑖 with complex distribution 𝑝𝑋, into a set of latent features 𝑍= {𝑧𝑖}𝑗∗ 𝑖=1, 𝑧𝑖∈R𝑐𝑖×ℎ𝑖×𝑤𝑖with standard Gaussian distribution 𝑝𝑍=𝑁(0, 𝐼), such that: 𝐺∶𝑋𝜃 ←←←←←←→ 𝑍, ∀𝑥∈𝑋, ∃!𝑧∈𝑍(7) To optimize this bijection for our anomaly detection problem, it is necessary to find the optimal parameters 𝜃∗that minimize a loss function , as described in Section 4.3, such that: 𝜃∗= arg min 𝜃(𝐺)(8) Instead of using NF, one might consider obtaining a similar approach with random linear or convolutional transformations. However, unlike NF, these transformations do not guarantee the creation of isomorphic spaces. The Johnson–Lindenstrauss lemma (Dasgupta and Gupta,2003) suggests that a high-dimensional space can be embedded in a lower-dimensional space using random linear or convolutional transformations. Nevertheless, such an approach compromises both the preservation of dimensionality and the invertibility of the transformation, which are fundamental concepts of our proposed methodology. As illustrated in Fig. 3, the decoder architecture consists of two main components: a dequantization layer and a series of affine coupling blocks, which are described below. 4.2.1. Dequantization NFs rely on the change of variables formula, which is naturally defined in continuous space. However, the feature maps consist of discrete values, which concentrate the probability mass in the discrete points, resulting in an undesired Dirac Delta distribution (Ho et al., 2019;Hoogeboom et al.,2021). To mitigate this problem, we introduce a dequantization procedure that spreads the concentrated probability mass into surrounding values, thus approximating a continuous space. Given a feature value 𝑣of 𝑥𝑖in the final 𝑖th feature map and denoting the dequantization operation as 𝑄, the dequantized point 𝑞=𝑄(𝑣)is computed as follows: 𝑄(𝑣) = 𝑣+𝑢, where 𝑢is a random value ∈ [0,1𝑒−5]. This random value acts as noise, propagating feature values around neighboring values without losing the original semantic coherence. Engineering Applications of Articial Intelligence 137 (2024) 109088 5 M. Campos-Romero et al. Fig. 3. Graphical representation of the InAI method, highlighting its encoder–decoder scheme. The encoder converts 𝑑𝑖∈𝐷𝑡𝑟𝑎𝑖𝑛 into a set 𝐹𝑖= {𝑥𝑖,𝑘}𝑗 𝑘=1 of feature maps. By concatenating affine coupling blocks, the decoder maps the subset 𝑋𝑖∶𝑋𝑖⊂ 𝐹𝑖to a latent Gaussian space 𝑍𝑖, addressing the MLE problem. The test images 𝑑𝑡∈𝐷𝑡𝑒𝑠𝑡 are processed similarly during inference. The notation 𝑥𝑖,𝑗 refers to the 𝑗th feature of the 𝑖-batch of images. For the affine coupling blocks, 𝑥represents the input vector, (𝑥𝑎, 𝑥𝑏)are two halves, 𝑠and 𝑡are random functions for scale and translation, 𝑒is the exponential function, and 𝑥′is the concatenation of the transformations. The final heat map shows likelihood scores in Gaussian space, with hotspots indicating faulty regions. 4.2.2. Affine coupling block The affine coupling block is the basic unit responsible for mapping 𝑋to 𝑍. The mapping 𝐺of the Decoder is achieved through a series of simpler, smoother, and bijective transformations, denoted by 𝑔. These 𝑔transformations, in turn, consist of even more basic invertible transformations, which are the affine coupling blocks denoted by 𝑓. This basic unit incorporates the following elements: an affine transformation 𝐴; an ActNorm2D normalization layer with parameters 𝑠𝑔, 𝑡𝑔, and activation function 𝜓; and a permutation operation 𝐾. For any input 𝑥, the affine coupling block 𝑓∶R𝑐×ℎ×𝑤→R𝑐×ℎ×𝑤is formalized as follows: 𝑓(𝑥) = 𝐾 ⊙ 𝛹(𝑠𝑔)⊙ 𝐴(𝑥) + 𝑡𝑔(9) Therefore, 𝐺(𝑋, 𝜃)={𝑔𝑖(𝑥𝑖, 𝜃𝑖)}𝑗∗ 𝑖=1 where 𝑗∗denotes the number of feature maps. Furthermore, each 𝑔𝑖(𝑥𝑖, 𝜃𝑖)is composed of a sequence of affine coupling blocks, such that 𝑔𝑖(𝑥𝑖, 𝜃𝑖) = 𝑓𝑖,1(𝑥𝑖, 𝜃𝑖,1)◦𝑓𝑖,2(⋅, 𝜃𝑖,2)◦… ◦𝑓𝑖,𝑘(⋅, 𝜃𝑖,𝑘). In general, the 𝑘th transformation maps the output of 𝑘−1- th transformation. Here, ◦denotes the composition operation, and 𝑘represents the length of the transformation sequence. The global transformation 𝐺parameterized by 𝜃includes all the parameters in each 𝑔𝑖: 𝜃= 𝑗∗ ⋃ 𝑖=1 𝜃𝑖∈𝑔𝑖(10) Similarly, the parameter 𝜃𝑖is composed of the parameters in the 𝑖th sequence of 𝑘transformations 𝑓𝑖,𝑡: 𝜃𝑖= 𝑘 ⋃ 𝑡=1 𝜃𝑖,𝑡 ∈𝑓𝑖,𝑡 (11) This hierarchical composition allows the model to build complex mappings from the input feature maps 𝑋to the latent space 𝑍, while maintaining the invertibility and smoothness required for effective anomaly detection. Given that every function 𝑓𝑖,𝑡 ∶R𝑐𝑖×ℎ𝑖×𝑤𝑖→ R𝑐𝑖×ℎ𝑖×𝑤𝑖has an inverse 𝑓−1 𝑖,𝑡 ∶R𝑐𝑖×ℎ𝑖×𝑤𝑖→R𝑐𝑖×ℎ𝑖×𝑤𝑖, it follows that 𝑔𝑖and, by extension, 𝐺are invertible. Next, we describe the operations that build our affine coupling block: affine transformation, ActNorm2D, and 1 ×1 convolution. Affine transformation. An affine transformation is a bijective geometric transformation that plays a crucial role in our affine coupling block. It is primarily responsible for mapping the components from 𝑋 to 𝑍. Given any 𝑥∈𝑋such that 𝑥∈R𝑐×ℎ×𝑤, our affine transformation 𝐴(𝑥)is described as: 𝑥=𝑥𝑎∥𝑥𝑏 𝑥𝑎=𝑥0∶(𝑐∕2) 𝑥𝑏=𝑥(𝑐∕2)+1∶𝑐 𝑥′ 𝑎=𝑥𝑎⊙ 𝑒𝑠1(𝑥𝑏)+𝑡1(𝑥𝑏) 𝑥′ 𝑏=𝑥𝑏⊙ 𝑒𝑠2(𝑥′ 𝑎)+𝑡2(𝑥′ 𝑎) 𝑥′=𝑥′ 𝑎∥𝑥′ 𝑏 (12) Engineering Applications of Articial Intelligence 137 (2024) 109088 6 M. Campos-Romero et al. The input variable is divided into two disjoint subsets, 𝑥0∶(𝑐∕2) and 𝑥(𝑐∕2)+1∶𝑐, which represent the first and second halves of the input data along the depth dimension, respectively. The affine transformation alternately acts on these halves, keeping one-half intact while modifying the other. Here, ⊙denotes the Hadamard product, and 𝑠(⋅)and 𝑡(⋅) are two small residually connected subnetworks performing scaling and translation. The final result 𝑥′is obtained by concatenating the transformed subsets 𝑥′ 𝑎and 𝑥′ 𝑏. ActNorm2D. Next, the activation normalization layer is applied using parameters 𝑠𝑎𝑐𝑡 and 𝑡𝑎𝑐𝑡, and the Softplus activation 𝜓to smooth the output. Normalization techniques were already proposed in Dinh et al. (2017), introducing an averaging-based batch normalization variant for small mini-batches to alleviate training instability issues. Subsequently, Kingma and Dhariwal (2018) proposed activation normalization (ActNorm) with data-dependent initialization, operating per channel. Our work extends that concept with a data-independent initialization Activation that includes the spatial dimension, hence 2D. 1x1 Convolution. Following each affine transformation, ensuring the independent transformation of each component is essential, given that the transformation of one half depends on the other. To achieve this, we alternate the way transformations occur. While (Dinh et al., 2017) proposed masked convolution to ensure different transformation patterns, it requires a complex design. Alternatively, Kingma and Dhariwal (2018) proposed a trainable invertible convolution layer to permute the elements. However, computing the inverse of matrices is computationally expensive, and there is a risk of encountering a non-invertible matrix during training. Our proposal introduces a 1 ×1 convolutional layer parameterized by a kernel 𝐾that belongs to the special orthonormal group. This represents a linear transformation with an inverse function parameterized by the transpose kernel 𝐾𝑇. This strategy ensures efficient computation and maintains invertibility throughout the training process, contributing to the stability and effectiveness of our model. The ablation studies provided in Appendix C analyze the impact of each of these components in the model performance. 4.3. Prior estimation This section defines the loss function and the training process of our Decoder for anomaly detection. As detailed in Section 4.2, our aim is to find the optimal parameters 𝜃∗that transform each 𝑥∈𝑋with a complex density 𝑝𝑋into a latent variable 𝑧∈𝑍with a standard Gaussian density 𝑝𝑍=𝑁(0, 𝐼). Given the observed training data, which are features from defectfree images such that 𝑥∈𝑋with density 𝑝𝑋, and assuming some latent variable 𝑧∈𝑍with density 𝑝𝑍, the objective is to map 𝑋 to 𝑍through the diffeomorphism application 𝐺∶𝑋𝜃∗ ←←←←←←←←←←→ 𝑍, where 𝑋=𝐺−1(𝑍). By finding the optimal parameters 𝜃∗, we can establish an optimal mapping between 𝑋and 𝑍. These parameters 𝜃∗parameterizing this mapping are determined by minimizing the loss function with a standard normal Gaussian prior, which is derived from the change-of-variable formula as explained in Section 3: 𝑝𝑋(𝑋) = 𝑝𝑍(𝐺(𝑋, 𝜃)) |||||det (𝜕𝐺(𝑋, 𝜃) 𝜕𝑋𝑇)|||||(13) Here, 𝜕𝐺(𝑋,𝜃) 𝜕𝑋𝑇represents the Jacobian matrix of the transformation, ensuring that the total probability of both distributions is bounded to 1, that is, ∫𝑝𝑋(𝑋), 𝑑𝑋 =∫𝑝𝑍(𝑍), 𝑑𝑍 = 1. To formulate a suitable loss function, we transform the probability density function (PDF) into the negative log-likelihood (NLL), which converts the problem into a minimization task: − log 𝑝𝑋(𝑋) = − log 𝑝𝑍(𝐺(𝑋, 𝜃)) − log |||||det (𝜕𝐺(𝑋, 𝜃) 𝜕𝑋𝑇)|||||(14) From an information theory perspective, we express Eq. (14) in terms of the Bit-Per-Dimension (BPD) loss function, quantifying the number of bits required to encode a particular sample within our Gaussian distribution. A lower BPD signifies a higher likelihood. To compute the BPD metric, we change the base of the NLL from natural base-e logarithm to base-2 logarithm, enabling NLL measurement in terms of bits per dimension: log2(𝑋) = log𝑒(𝑋) log𝑒(2) (15) Since log𝑒(2) = 1 log2(𝑒), Eq. (15) can be rewritten as: log2(𝑋) = log𝑒(𝑋) log2(𝑒)(16) Then, we substitute Eq. (14) into the log𝑒(𝑋)term of Eq. (16). This allows us to express the NLL as a base two logarithm. The term (∏𝑑𝑖)−1 denotes the reciprocal of the observed data’s dimensionality (equivalent to the product of its dimensions), used to normalize our final loss function: (𝐺) = − log 𝑝𝑋(𝑋) log2(𝑒)(∏𝑑𝑖)−1 (17) 4.4. Posterior estimation This section describes the process of identifying anomalies in new incoming images once the model has completed the training phase. During inference, the optimized 𝜃∗parameters obtained after training are used by our Decoder 𝐺(𝑋, 𝜃∗)to produce an output  𝑍. This output is a set of mutually independent latent features, represented as  𝑍= 𝑧1,…, 𝑧𝑗∗, such that 𝑧𝑖⟂𝑧𝑗,∀𝑖≠𝑗. These features are then used to construct an NLL score map, denoted as 𝜂, to detect anomalous pixels within unseen images. In our proposed model, the task of anomaly detection is framed as evaluating the alignment between a predicted multivariable random variable 𝑧 ∈ 𝑍and the target density 𝑝𝑍(𝜇, 𝛴) = 𝑁(0, 𝐼)learned during training. As discussed in Section 4.3, the training process involves solving an MLE problem by minimizing the NLL using only nominal images. By evaluating the likelihood of 𝑧 under the standard Gaussian density 𝑝𝑍=𝑁(0, 𝐼)learned by the model, anomalous images can be identified as deviating significantly from this distribution. We employ the PDF associated with the assumed multivariate Gaussian density 𝑁(𝜇, 𝛴)for the 𝐷-dimensional random variable 𝑧 to calculate these likelihood scores. The PDF is given by Eq. (18), where (2𝜋)−𝐷∕2 and |𝛴|−1∕2 are normalization terms, and the term (𝑧 −𝜇)𝑇𝛴−1(𝑧 −𝜇) represents the squared Mahalanobis distance. 𝑝𝑍(𝑧) = (2𝜋)−𝐷∕2|𝛴|−1∕2𝑒−1 2(𝑧−𝜇)𝑇𝛴−1(𝑧−𝜇)(18) In the context of our model, the Mahalanobis distance measures the deviation of 𝑧 from the learned target distribution 𝑍∼𝑝𝑍=𝑁(0, 𝐼). Hence, in Eq. (18),𝜇∈𝑅𝐷∶𝜇= 0 and 𝛴∈𝑅𝐷×𝐷∶𝛴= 𝐼. Therefore, the Mahalanobis distance simplifies to the L2 norm or Euclidean distance ‖𝑧‖2from 𝑧 to the origin, as shown in Eq. (19). 𝑀(𝑧, 𝑍) = √(𝑧 −𝜇)𝑇𝛴−1(𝑧 −𝜇) = √𝑧𝑇𝑧 =‖𝑧‖2(19) Consequently, when expressed in terms of the NLL, the PDF (Eq. (18)) is proportional to Eq. (20) as follows: − log 𝑝𝑍(𝑧) = 1 2𝑀2(𝑧, 𝑍) = 1 2‖𝑧‖2 2(20) To generate the final score map 𝜂for anomaly detection, we calculate the deviation at the pixel level from the learned distribution, considering each local position (𝑎, 𝑏)in 𝑧. This deviation is quantified using the NLL scores stored in the 𝜂map with dimensions 𝐻×𝑊. Regions associated with anomalies will have a high NLL, indicating a greater distance from the learned distribution. In contrast, regions without defects will have a low NLL, suggesting proximity to the learned distribution. To obtain 𝜂, we first compute an NLL score map 𝑚for each 𝑧 using the aggregation function 𝑓add ∶𝑅𝑐×ℎ×𝑤→𝑅ℎ×𝑤. This function Engineering Applications of Articial Intelligence 137 (2024) 109088 7 M. Campos-Romero et al. aggregates the NLL scores at each spatial location (𝑎, 𝑏)along the channels in 𝑧. Using Eq. (20) at the pixel level, we compute the NLL score map 𝑚for each 𝑧 using the following formula: 𝑚(𝑎,𝑏)=𝑓add (𝑧)(𝑎,𝑏)=1 2 𝑐 ∑ 𝑘=1 ‖𝑧(𝑘,𝑎,𝑏)‖2 2(21) Applying Eq. (21) to each 𝑧 yields a set of score maps {𝑓add(𝑧1), …, 𝑓add(𝑧𝑗∗)} = {𝑚1,…, 𝑚𝑗∗}. The next step is to unify these 𝑗∗score maps. However, each 𝑚has its own spatial dimensions ℎ×𝑤, which must be normalized before unification. To achieve this, we utilize a bilinear interpolation function, represented as , which maps each pixel from its original position 𝑚(𝑎,𝑏)to a new position 𝑚(𝑎′,𝑏′), such that ∶𝑅ℎ×𝑤→𝑅𝐻×𝑊: (𝑎′, 𝑏′) = (𝑚)(𝑎,𝑏)(22) Finally, the averaging function 𝑓avg ∶𝑅𝑗∗×𝐻×𝑊→𝑅𝐻×𝑊is used. This function calculates the final NLL score map, denoted as 𝜂, by averaging the 𝑗∗interpolated maps at each spatial location (𝑎′, 𝑏′), as follows: 𝜂(𝑎′,𝑏′)=𝑓avg (𝑚1,…, 𝑚𝑗∗)=1 𝑗∗ 𝑗∗ ∑ 𝑖=1 𝑚𝑖(𝑎′,𝑏′)(23) After generating the final NLL map 𝜂, two decision-making functions are introduced, represented as 𝐹seg ∶𝑅2→{0,1} and 𝐹det ∶𝑅𝐻×𝑊× 𝑅→{0,1}. The function 𝐹seg, given a specific threshold 𝜏, operates at the pixel level, allowing for segmentation. In contrast, 𝐹det works at the image level, classifying an entire input image as anomalous or not. This distinction between pixel-level and image-level anomaly detection improves the flexibility of our model. The threshold 𝜏is determined as the minimum log-likelihood over the entire training distribution of defect-free images. 𝐹seg applies a thresholding operation to each pixel location (𝑎′, 𝑏′) of an image, using its corresponding NLL score 𝜂(𝑎′,𝑏′)and a defined threshold 𝜏. This function assigns a value of 1 to pixels representing an anomalous region and a 0 to non-defective pixels. This results in a binary mask that highlights the areas with defects. 𝐹seg(𝜂(𝑎′,𝑏′), 𝜏) = {1,if 𝜂(𝑎′,𝑏′)≥𝜏 0,otherwise (24) The second function, 𝐹det, evaluates the anomaly status of an entire image. This is achieved by comparing the maximum NLL score across all pixels within the image to a predefined threshold 𝜏. If the NLL score of any pixel surpasses or equals this threshold, the image is considered anomalous, and the function returns a 1. Conversely, if no pixel’s NLL score crosses the threshold, the image is classified as non-anomalous, prompting the function to output a 0. 𝐹det(𝜂, 𝜏) = ⎧ ⎪ ⎨ ⎪ ⎩ 1,if max (𝑎′,𝑏′)(𝜂(𝑎′,𝑏′))≥𝜏 0,otherwise (25) 5. Experimental study Implementation details. To determine the hyperparameters of our model, we performed manual adjustments based on empirical results and existing evidence. For optimization, we chose Adam’s optimizer with an L2 penalty value of 1𝑒−5, an epsilon value of 1𝑒−8, and betas set to (0.9,0.99), as recommended by Kingma and Ba (2014). We tested various learning rates {3 × 10−2,3 × 10−3,2 × 10−5} and batch sizes {16, 32, 64}. Ultimately, we settled on a learning rate of 3 × 10−3 and a batch size of 32, as they offered no substantial changes. Regarding the decoder configuration, we also experimented with different clamp values and observed the best performance with a clamp value of 1.9. We found that performance improved up to 6 blocks for the number of coupling blocks. Ablation studies of these hyperparameters are provided in Appendix C. Furthermore, we performed experiments with different input image resolutions {256,343,434,521,603}, followed by center cropping to {224,300,380,456,528} to maintain an aspect ratio of 1.4. The performance remained consistent across different resolutions, so we opted for the smaller size to reduce computational demand. The results of these experiments are presented in Table 3. Images were standardized using the mean and standard deviation calculated from the ImageNet dataset. The training was performed for a maximum of 800 epochs and stopped if the AUROC did not improve after 150 epochs. For instance, in our system with an NVIDIA RTX3090 GPU, the complete training process of the MVTec AD lasted approximately 4 days. Metrics. The performance of our method is evaluated using the area under the receiver operating characteristic curve (AUROC) metric. AUROC is the metric used in this research field primarily due to its ability to evaluate a model’s ability to accurately distinguish between anomalous (defective) and normal (anomaly-free) instances, essentially measuring how well the model identifies both classes equally. It is computed as the ratio 𝑇 𝑃 𝑅 𝐹 𝑃 𝑅 , where TPR signifies the True Positive Rate, and FPR denotes the False Positive Rate. This metric assesses the performance of both image-level anomaly detection and pixel-level segmentation. Additional metrics are provided in Appendix B. Datasets. The reliability of our method is evaluated using the MVTec Anomaly Detection (Bergmann et al.,2021), MVTec Logical Constraints Anomaly Detection (Bergmann et al.,2022a), and Surface Defect Saliency of Magnetic Tile (SDMT) (Huang et al.,2020) datasets. 5.1. Experiments on MVTec anomaly detection dataset Details. The MVTec Anomaly Detection (MVTec AD) dataset is the most popular multi-object and multi-defect anomaly dataset for 2D inspection. It incorporates 15 sub-datasets, totaling 3629 training images and 1725 test images, all with a resolution of 1024x1024. The training set consists exclusively of defect-free images, while the test sets combine defective and non-defective images. The dataset is divided into two main image types: textures (with five sub-datasets: carpet, grid, leather, tile, wood) and objects (with ten sub-datasets: bottle, cable, capsule, hazelnut, metal nut, pill, screw, toothbrush, transistor, zipper). It includes 73 types of defects in total, averaging five per sub-dataset. This study evaluates both the anomaly detection (image-level) and segmentation (pixel-level) performance for defects in test images across all 15 sub-datasets. Fig. 4 shows one example from each sub-dataset. Baseline models. We compare our InAI method with SPADE, U- ST, GCAD, DifferNet, ITran, PaDiM, Mahalanobis-AD, DMU-TransNet, DRAEM, SSMRKD, CFlow, AFlow, CS-Flow, TFA-Net, AEKD, CFA, Mem- Seg, and Patchcore. For the compared models, we consider the best configuration as reported by the corresponding authors. Further details are available in Section 2or the corresponding studies. Description. Section 5.1.1 evaluates the Decoder performance using various Encoder configurations. These configurations generate diverse deep representations of incoming images from different CNN architectures and hierarchical levels. In Section 5.1.2, our method is benchmarked against other state-of-the-art methodologies. Section 5.1.3 offers a qualitative analysis of our method. This analysis involves a visual exploration and discussion of the results obtained. Finally, Section 5.1.4 tests our system in a low-shot regime, using fewer training data than other methodologies to simulate real-world conditions of data scarcity. 5.1.1. Quantitative analysis The first part of the analysis evaluates the performance using different ResNet and EfficientNet backbones as the Encoder. Semantic information is extracted from specific hierarchical levels related to the bottleneck to study their influence on the Decoder’s performance. This focus is crucial, as semantic information provides varying levels of abstraction depending on the network’s depth. The goal is to identify the Engineering Applications of Articial Intelligence 137 (2024) 109088 8 M. Campos-Romero et al. Fig. 4. One sample of each subset in the MVTec Anomaly Detection dataset. Table 2 EfficientNet AUROC(%) detection/segmentation performance. Encoder Block 2 Block 3 Block 5 Block 7 EfficientNet-B0 87.7/93.0 92.3/94.5 98.5/96.0 98.1/91.7 EfficientNet-B3 89.0/83.2 94.2/95.1 98.9/95.9 97.4/90.6 EfficientNet-B4 87.9/93.0 95.2/95.4 99.2/96.5 97.5/91.3 EfficientNet-B5 87.2/93.5 94.6/95.5 99.1/96.0 97.5/90.5 EfficientNet-B6 89.6/93.5 96.0/96.0 99.1/96.3 98.1/91.6 Table 3 EfficientNet AUROC(%) detection/segmentation performance with their original input size. Encoder Input size Block 3 Block 5 Block 7 EfficientNet-B0 224 92.3/94.5 98.5/96.0 98.1/91.7 EfficientNet-B3 300 89.9/93.9 99.0/96.4 98.3/94.3 EfficientNet-B4 380 92.2/93.6 99.2/96.6 98.0/94.7 EfficientNet-B5 456 91.2/92.6 98.7/96.2 98.6/95.8 EfficientNet-B6 528 91.3/92.4 98.8/96.3 98.1/95.7 optimal point for obtaining the most beneficial semantic information for the anomaly inspection task. Table 2 reports anomaly detection and segmentation performance with different architectures and EfficientNet blocks. The performance is presented in tuple form, with the first value representing AUROC detection (image level) and the second AUROC segmentation (pixel level), expressed as a percentage. Block 5 consistently achieves the best performance for all architectures. EfficientNet-B4 achieves the highest image-level AUROC value of 99.2%. For segmentation, Block 5 in EfficientNet-B4 achieves the highest pixel-level AUROC value of 96.5%. These results suggest that the semantic information captured in Block 5 of EfficientNet-B4 is the most effective for our visual anomaly inspection task. These experiments were performed using an image resolution of 224 ×224 to ensure comparability with future experiments. However, EfficientNet models were originally trained at higher resolutions. Our hypothesis was that EfficientNet models would perform better at the native resolutions, potentially yielding more useful intermediate semantic information and improving decoder performance. Table 3 reports the results. Our findings remain consistent, with Block 5 of EfficientNet- B4 providing the best performance. However, using higher resolutions did not enhance anomaly inspection performance, even when using the resolutions for which each model was optimized. Further experiments included using various ResNet architectures, with the results detailed in Table 4. Wide-ResNet50 in Stage 3 achieves Table 4 ResNet AUROC(%) detection/segmentation performance. Encoder Stage 1 Stage 2 Stage 3 Stage 4 ResNet18 90.8/93.2 96.0/94.9 98.1/96.1 92.6/89.0 Wide-ResNet50 95.5/95.4 98.6/96.8 99.1/96.9 94.2/92.6 Wide-ResNet101 95.7/96.2 98.0/97.7 98.9/97.3 94.8/91.2 the best detection AUROC, with a value of 99.1%. Wide-ResNet101 in Stage 2 achieved the best segmentation AUROC, with 97.7%. In conclusion, Wide-ResNet50 is the best-performing ResNet architecture for anomaly detection, while Wide-ResNet101 outperforms the other models in anomaly segmentation. These findings underscore the significance of semantic information across hierarchical levels for decoder performance. Each level contains information at varying levels of abstraction. To leverage this information effectively, we introduce lateral connections between the encoder and decoder, enabling the training and estimation of anomalies from multiple abstraction levels. As explained in Section 4.4, the final decision is determined by averaging across all levels. Based on the preceding results (Tables 2–4), we progressively added layers from best to worst performance. The performance begins to decrease after adding more than two layers per model. Optimal detection is achieved using the final output maps of Stage 2 and Stage 3 for ResNet and Blocks 5 and 7 for EfficientNet. Such results are reported in Table 5. Wide-ResNet50 achieves the highest anomaly detection performance of 99.8%. For segmentation, Wide-ResNet101 outperforms the other ResNet architectures, scoring 98.4%. Among EfficientNet backbones, both EfficientNet-B4 and EfficientNet-B5 exhibit identical anomaly detection performance at 99.2%. However, EfficientNet-B4 surpasses the other EfficientNet architectures in segmentation performance, achieving the highest score of 96.1%. Interestingly, while ResNets demonstrate a significant performance improvement when combining different semantic information, EfficientNet models do not show a substantial increase in performance. Nevertheless, EfficientNet models demonstrate faster convergence. These results support our previous hypothesis: adding more semantic information significantly influences decoder performance, facilitating the discrimination between defective and defect-free images. However, relying solely on the semantic information of a single network poses challenges, as it tends to be biased towards ImageNet (at the last hierarchical levels) or exhibits limited abstraction (at the first hierarchical levels). Thus, we propose another hypothesis: leveraging Engineering Applications of Articial Intelligence 137 (2024) 109088 9 M. Campos-Romero et al. Fig. 12. Qualitative results in SMTD. Each row displays three examples of each defect type. Fig. 13. Qualitative results in SDMT showcasing representative examples of false negatives and false positives. Table 11 Computational efficiency comparison of anomaly detection methods on an NVIDIA RTX 3090 GPU. Empty values denote information not available. Methods CFlow AST CFA ST GDCA Patchcore Patchcore Ensemble InAI Latency [ms] ↓309 83.0 11.4 – – 20.3 118.1 1.9 Throughput [img/s] ↑3 50 87 – – 42 8 506 Memory usage [MB] ↓1142 618 637 + kNN + M 1077 555 637 + kNN + M 1335 + kNN + M 362 GFLOPs ↓281 199 42 + kNN 4468 416 42 + kNN 159 + kNN 47 requires more GFLOPs than PatchCore or CFA but has significantly lower latency and higher throughput. More findings from this analysis are presented in Fig. 15, illustrating the relationship between latency and AUROC. The graphs depict this relationship for the MVTec AD dataset (left) and the MVTec LOCO AD dataset (right). Moving further to the left indicates lower latency while moving higher up the graph indicates better AUROC performance. Our method distinctly positions itself as the best-performing technique, achieving the optimal latency-AUROC balance. 7. Discussion and implications Summary of the research. This study develops a novel unsupervised CNN model for anomaly detection. The model adopts an encoder– decoder structure, integrating a pre-trained CNN in ImageNet at the encoder and normalizing flow techniques at the decoder. Through extensive experimentation, we analyzed the performance of various ResNet and EfficientNet backbones as encoders, examining the influence of specific hierarchical levels related to the bottleneck on Engineering Applications of Articial Intelligence 137 (2024) 109088 16 M. Campos-Romero et al. Fig. 14. Qualitative results in SDMT. For the Uneven defect type, the model segments a thin line around the edges where the uneven part begins, contrasting with the technician-labeled ground truth mask. Fig. 15. Computational analysis on the MVTec AD and MVTec AD LOCO in an Nvidia RTX3090. decoder performance. The goal was to identify the optimal configuration to obtain the most beneficial semantic information for our anomaly inspection task. Key findings. With a single-scale approach, the results indicate that Block 5 consistently achieves the best performance across all EfficientNet architectures, with EfficientNet-B4 achieving the highest AUROC value for anomaly detection. While adopting a multi-scale, multi-semantic approach, we observe ResNet’s superiority over EfficientNet. Despite conducting experiments with EfficientNet at higher resolutions, ResNet’s performance remains unmatched. We also propose a ResNet and EfficientNet ensemble, resulting in further improvements. Implications. Our findings highlight the importance of selecting the appropriate hierarchy level to optimize decoder performance. The superior performance of EfficientNet-B4, coupled with its lower computational cost compared to ResNet, positions it as a viable choice for industrial deployment, where real-time responsiveness is often imperative. Moreover, the lack of performance improvement at higher resolutions suggests that a resolution of 224 ×224 may suffice for anomaly inspection, a crucial insight for resource-constrained environments. Additionally, our model demonstrates robust performance in low-shot tasks, a crucial capability for real-world applications where extensive image datasets may not be readily available at project inception. Limitations. While the InAI method sets new benchmarks in visual anomaly detection, several challenges remain. Since the method relies on maximizing the density of non-defective training data, the model may have difficulty identifying anomalies if these nominal images exhibit complex patterns. On the other hand, NF assumes some Gaussian distribution exists, which may not always hold in more complex scenarios. Furthermore, detecting very small anomalies presents difficulties, as increasing image resolution may not necessarily improve performance and may burden the model’s capacity to handle larger distribution volumes. Future research. Future studies could explore methodologies to better fit the model to Gaussian distributions, such as parameterizing theoretical Gaussians with learnable parameters. Additionally, researchers could implement flow matching techniques (Lipman et al., Engineering Applications of Articial Intelligence 137 (2024) 109088 17 M. Campos-Romero et al. 2022), which have been shown to improve density estimation in highresolution images. Exploring 3D anomaly detection (Bergmann et al., 2022b), enhancing performance on datasets such as MVTec LOCO AD, or developing more specialized low-shot methods could also yield significant contributions to the field. Broader impact. Finally, although InAI focuses on anomaly detection, our research has strong connections to other topics related to generative data. We hope that our findings, architecture, methodology, limitations, and potential solutions will open new avenues and ideas to researchers. For instance, conditioning flows to feature maps with variational dequantization techniques could be investigated. 8. Conclusions This paper presents InAI, a novel unsupervised method for anomaly detection in industrial image data. InAI is a probabilistic generative model based on NF that can be trained exclusively on defect-free image data to detect and localize defective regions in new images. Our extensive evaluation demonstrated that InAI outperforms existing state- of-the-art methods, achieving higher anomaly detection and equally robust segmentation scores. We further showcased the effectiveness of our approach in low-shot training scenarios, where InAI achieves significantly superior performance compared to competing methods despite using fewer training samples. Additionally, to validate the generalizability of our method beyond the MVTec AD dataset, we applied it to two other datasets, which also involve detecting logical anomalies. Once again, InAI exhibited superior performance compared to state- of-the-art models. Overall, InAI offers reliable accuracy in detecting anomalies, making it a promising solution for ensuring product quality in industrial manufacturing applications. CRediT authorship contribution statement Miguel Campos-Romero: Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Software, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Manuel Carranza-García: Writing – review & editing, Supervision, Methodology. José C. Riquelme: Writing – review & editing, Supervision, Funding acquisition. Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Miguel Campos-Romero reports financial support was provided by LMI Technologies. Miguel Campos-Romero reports a relationship with LMI Technologies that includes: employment. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Data availability The datasets used are in the public domain. I made available a file, which is linked in the manuscript, to allow the researcher to reproduce our results. Acknowledgments I thank Chi Ho Ng (LMI Technologies), Rafael Zorrilla Robles (TKH Security), and Robert-Jan Sips (TKH AI) for supporting this research. Funding This research has been supported by LMI Technologies Inc.; Grant PID2020-117954RB-C22 funded by MICIU/AEI/10.13039/501100011 033; Grant TED2021-131311B-C21 funded by MICIU/AEI/10.13039/ 501100011033 and by the European Union NextGenerationEU/PRTR. Appendix A. AUROC results for MVTec AD This section presents the AUROC scores achieved by our model across all sub-datasets within the MVTec AD dataset, which are detailed in Section 5.Tables A.12–A.14 (summarized in Tables 2–4) present our experimental results using each semantic level of the backbone networks of the encoder. Tables A.15–A.16 (summarized in Table 5) report the AUROC performance of our method when including lateral connections to leverage semantic information across multiple levels of abstraction. Table A.17 (summarized in Table 6) describes the AUROC performance of our final ensemble-based proposal. Table A.18 (summarized in Table 8) shows the performance under low-shot training conditions. The analysis on the MVTec LOCO AD dataset, being smaller, is fully reported in Section 5. Appendix B. Complementary metrics for MVTec challenges This section presents complementary metrics for the main datasets used in our study: MVTec AD and MVTec LOCO AD. Regarding the detection task, in addition to AUROC, we also calculated the Area Under the Precision-Recall Curve (AUPR). The AUPR metric is crucial in industrial applications as it allows optimization of thresholds that align with specific business requirements. For instance, optimizing precision while maintaining a recall of at least 99% is common, ensuring a maximum error rate of 1% in defect detection. Tables B.19 and B.20 present AUPR alongside the best F1 scores, precision, and recall. The AUPR metrics provided are based on training optimized for AUROC, the primary metric for our challenge. While superior AUPRs, leading to higher F1 scores, precision, and recall, can be achieved with training tailored specifically for these metrics, this is often not desirable in practice. The AUPR metrics presented are based on training optimized for AUROC, the primary metric for our challenge. While better AUPRs, and consequently higher F1 scores, precision, and recall, can be achieved with training designed specifically for these metrics, this is not desirable in practice. Furthermore, the F1 emphasizes the balance between precision and recall. However, accurately identifying true positives (anomalies) and negatives (non-anomalies) is equally important in anomaly detection. The predefined AUROC metric is better suited to achieve this balance. Regarding the segmentation task, we also calculated the Jaccard coefficient, also known as Intersection over Union (IoU), which measures the accuracy of spatial detections. The Jaccard coefficient ranges from 0 to 1 and is widely used in object detection tasks to measure the accuracy between bounding boxes. In our study, we apply it at the pixel level to assess the similarity between model mask predictions and ground truth masks. However, evaluating true negatives (pixels correctly identified as non-defective) is equally important in our context. We also introduced the AUPRO (An and Cho,2015) metric to address this. AUPRO evaluates both true positives and negatives and operates at the region level, unlike IoU and AUROC. This ensures that detecting small anomalies is as important as detecting large ones, and it penalizes predictions that exceed certain false positive percentages, typically set at 30%. These results are reported in Tables B.21 and B.22. Appendix C. Ablation study on decoder configuration This section provides ablation experiments that evaluate the impact of different components of the model’s decoder design on task performance. For all experiments, we employ the Wide-ResNet50 backbone in the encoder (refer to Table 6) due to its widespread adoption in the field (Roth et al.,2022;Rudolph et al.,2021;Gudovskiy et al.,2022). Fig. C.16 displays performance curves for the detection and segmentation tasks across different experiments. The curves represent the average AUROC performance over epochs across individual datasets. Each experiment, distinguishable by the figure legend, involves replacing a specific component of our proposed model. Engineering Applications of Articial Intelligence 137 (2024) 109088 18 M. Campos-Romero et al. Table A.12 ResNets’ AUROC(%) detection/segmentation at each hierarchical level. Hierarchy level Model Carpet Grid Leather Tile Wood Bottle Cable Capsule Hazelnut Metalnut Pill Screw Toothbrush Transistor Zipper Layer 1 ResNet18 98.4/97.3 98.8/98.9 100.0/99.6 96.0/94.1 99.9/97.0 99.8/97.0 85.7/91.8 80.1/95.0 95.4/94.9 90.5/95.4 78.9/88.9 77.6/94.7 93.1/85.1 75.7/70.1 92.0/98.0 Wide-ResNet50 98.8/97.4 99.8/99.2 100.0/99.6 99.8/97.8 100.0/97.6 97.6/96.6 85.8/93.0 94.2/97.7 98.7/94.5 98.3/97.9 86.9/95.7 83.9/92.5 100.0/93.2 90.0/79.2 98.4/98.9 Wide-ResNet101 99.3/97.4 99.7/99.2 100.0/99.6 99.6/97.7 100.0/97.2 99.2/98.2 86.4/93.3 93.5/98.0 94.9/96.4 94.0/98.4 90.6/97.1 98.1/97.0 96.1/95.2 84.9/78.4 99.0/99.2 Layer 2 ResNet18 98.8/98.4 98.8/98.6 100.0/99.4 98.7/89.3 98.6/91.8 100.0/96.3 90.3/90.6 92.8/98.0 91.4/96.1 97.8/91.1 93.5/95.6 92.7/96.3 97.8/95.2 92.5/87.4 96.7/98.5 Wide-ResNet50 99.0/98.6 99.2/99.1 100.0/99.4 99.8/95.0 99.8/95.9 100.0/98.3 89.6/92.5 98.0/98.8 99.2/96.1 100.0/97.1 97.2/97.3 98.2/99.2 100.0/97.7 98.7/87.6 99.8/98.9 Wide-ResNet101 99.4/98.8 98.9/99.3 100.0/99.5 99.4/97.2 99.8/96.8 100.0/97.8 94.5/96.3 95.9/98.8 95.7/98.5 99.2/98.3 95.7/98.5 95.9/99.4 98.9/97.2 99.3/89.2 98.1/99.2 Layer 3 ResNet18 99.8/98.7 95.5/95.5 100.0/98.8 98.7/91.2 99.7/91.0 100.0/96.0 98.6/96.0 94.6/97.6 99.9/97.0 99.4/93.8 94.2/97.3 94.7/97.1 98.6/97.7 99.9/96.3 97.9/97.0 Wide-ResNet50 99.9/98.7 98.7/96.8 100.0/99.1 99.5/94.2 100.0/92.6 100.0/97.2 99.7/96.1 96.8/98.1 100.0/98.5 100.0/96.3 96.9/97.7 97.4/97.8 97.5/97.8 100.0/97.0 99.4/95.9 Wide-ResNet101 99.6/98.5 99.2/97.0 100.0/99.0 99.4/95.0 99.5/91.7 100.0/97.3 99.7/97.7 94.9/98.2 99.9/97.9 100.0/96.8 98.1/98.4 97.8/98.5 96.7/98.3 100.0/98.0 98.7/97.4 Layer 4 ResNet18 92.3/93.0 67.8/71.9 99.9/92.1 98.9/83.6 98.6/76.3 99.8/91.0 94.5/88.5 88.2/92.6 98.5/94.5 96.7/94.8 89.2/93.2 81.7/82.3 94.4/96.2 94.4/97.4 94.6/87.8 Wide-ResNet50 98.3/95.0 86.5/87.3 100.0/96.2 99.0/91.0 99.5/84.3 100.0/93.7 91.0/88.6 91.5/95.9 98.5/95.6 96.1/93.6 86.1/95.5 73.0/88.4 100.0/95.7 96.7/95.5 97.1/92.1 Wide-ResNet101 97.7/96.0 91.8/85.9 100.0/96.7 98.7/91.6 98.6/83.3 99.8/90.8 91.0/84.9 92.1/93.9 98.3/93.0 94.2/91.1 92.4/95.5 82.8/83.7 91.4/92.8 94.7/96.9 97.9/92.1 Table A.13 EfficientNets’ AUROC(%) detection/segmentation at each hierarchical level. Hierarchy level Model Carpet Grid Leather Tile Wood Bottle Cable Capsule Hazelnut Metalnut Pill Screw Toothbrush Transistor Zipper Block 2 B0 96.8/95.7 95.7/97.4 100.0/99.6 98.6/96.5 99.9/96.4 99.1/97.2 73.0/86.9 72.8/94.0 92.1/94.5 91.1/93.8 83.1/94.7 64.6/93.8 92.5/89.0 66.0/67.4 89.8/98.5 B3 97.3/95.5 93.9/99.0 100.0/99.6 99.2/96.3 100.0/95.8 99.0/96.7 81.2/83.4 76.4/95.8 94.3/94.4 93.5/96.3 86.9/96.2 57.6/92.1 92.2/85.2 69.7/73.5 94.3/97.9 B4 98.8/96.6 94.0/98.9 100.0/99.6 99.1/95.8 100.0/96.4 99.0/96.2 77.8/86.0 77.3/95.3 97.6/96.1 90.3/81.8 89.3/96.0 56.4/93.0 78.6/93.0 68.2/72.1 92.6/98.4 B5 98.0/96.7 96.7/98.9 100.0/99.7 99.0/96.6 100.0/96.1 99.8/96.2 74.2/86.7 73.9/95.2 95.9/96.1 89.1/87.7 77.3/95.3 52.8/93.4 88.1/91.4 71.6/73.5 92.1/98.4 B6 98.8/97.7 98.5/99.2 100.0/99.6 99.0/96.5 100.0/97.0 98.6/97.0 73.6/85.0 84.2/97.2 94.0/96.8 93.3/78.7 81.9/95.8 60.9/92.0 88.3/91.8 76.8/78.8 96.7/98.8 Block 3 B0 92.4/90.1 99.9/99.0 100.0/99.5 99.6/95.6 99.7/93.5 100.0/97.2 92.4/90.1 91.1/97.9 96.1/94.6 93.8/91.9 93.2/95.2 48.1/92.1 90.8/96.4 87.7/86.1 99.6/98.9 B3 99.7/97.8 99.7/98.9 100.0/99.6 99.8/94.9 99.6/93.2 100.0/97.7 89.3/89.4 91.0/98.2 97.2/94.2 97.6/92.0 89.0/94.6 68.4/91.7 92.2/95.3 91.7/90.2 97.8/98.9 B4 99.8/98.4 99.7/99.1 100.0/99.5 99.3/94.5 99.2/94.8 100.0/97.2 90.9/89.7 86.6/98.2 91.0/92.3 97.7/90.6 91.0/95.1 90.7/95.8 96.4/96.2 88.8/90.0 97.4/98.9 B5 100.0/96.3 99.1/99.1 100.0/99.5 99.6/94.8 98.9/94.2 100.0/96.7 85.2/89.3 93.1/98.3 88.5/93.5 96.8/92.7 94.9/95.7 74.9/94.6 96.7/96.0 93.4/92.6 97.7/98.8 B6 99.9/98.4 99.7/99.0 100.0/99.4 99.6/95.3 99.1/94.8 99.9/96.5 92.8/90.4 91.8/98.4 90.3/92.5 98.5/92.5 95.1/95.0 82.2/97.6 97.8/96.8 95.2/94.4 97.7/99.0 Block 5 B0 99.3/98.0 99.8/97.3 100.0/98.7 99.4/92.3 99.6/91.8 100.0/96.1 97.2/95.0 97.4/97.8 99.3/94.9 99.4/93.0 96.3/96.4 96.3/97.6 93.1/97.1 100.0/96.2 99.9/97.6 B3 100.0/98.3 99.7/96.5 100.0/99.0 99.9/91.4 99.7/91.3 100.0/96.1 99.4/95.2 97.6/97.6 99.4/94.5 99.3/93.4 96.1/95.2 96.8/98.5 94.7/96.7 100.0/97.1 100.0/97.5 B4 100.0/98.6 99.9/99.0 100.0/99.1 100.0/92.5 99.9/94.1 100.0/96.5 99.7/96.1 96.4/97.4 98.1/95.3 99.3/91.9 99.6/96.1 99.3/99.0 96.4/97.3 100.0/97.1 99.8/97.8 B5 100.0/98.3 99.9/97.4 100.0/98.9 99.1/91.5 99.6/92.7 100.0/96.5 99.7/95.8 95.7/97.5 99.2/94.4 99.5/93.7 99.2/95.5 99.3/98.7 95.3/96.0 100.0/96.7 100.0/97.1 B6 99.9/98.2 99.9/96.3 100.0/99.1 99.9/93.6 99.9/94.4 100.0/96.2 100.0/96.6 96.4/97.8 97.2/95.2 99.9/91.8 98.3/95.3 98.8/98.9 96.9/96.8 100.0/96.8 99.7/97.5 Block 7 B0 99.6/91.0 97.2/88.8 100.0/86.7 98.5/85.4 99.8/80.7 100.0/91.8 99.9/92.1 93.7/93.8 100.0/96.4 99.4/95.2 93.7/96.2 95.3/91.7 96.1/95.8 99.0/97.3 99.7/93.3 B3 99.8/93.2 98.5/85.4 100.0/92.8 99.0/84.5 99.3/82.8 100.0/88.2 97.2/88.0 93.5/93.4 99.4/93.5 99.1/94.7 89.6/94.6 91.7/89.3 97.8/94.8 98.3/97.4 98.0/86.7 B4 100.0/94.1 99.5/93.8 100.0/94.5 99.2/86.1 99.8/83.7 100.0/88.6 98.7/89.0 89.4/93.3 98.7/93.0 97.7/93.5 91.2/94.5 95.6/89.2 95.6/89.2 98.8/97.7 98.6/89.4 B5 99.8/94.2 99.1/83.5 100.0/92.9 98.3/87.6 99.1/82.7 100.0/90.6 97.4/87.0 91.5/94.4 99.1/94.7 98.3/93.0 91.5/91.9 96.2/86.5 96.1/94.1 98.0/96.6 98.8/87.8 B6 100.0/93.9 98.7/83.8 100.0/94.4 99.1/87.6 99.8/85.4 100.0/90.7 98.0/89.1 91.3/95.1 99.9/95.3 99.5/91.6 94.4/94.0 95.2/90.0 97.5/95.9 99.3/97.2 99.1/90.2 Table A.14 EfficientNets’ AUROC(%) detection/segmentation at each hierarchical level using larger input size. Hierarchy level Model Resolution Carpet Grid Leather Tile Wood Bottle Cable Capsule Hazelnut Metalnut Pill Screw Toothbrush Transistor Zipper Block 3 B3 300 99.6/97.2 98.9/99.3 100.0/99.6 99.4/95.9 99.9/94.2 99.3/97.1 86.9/86.1 92.2/97.9 94.0/93.8 97.5/89.1 55.0/92.9 55.0/92.9 96.9/95.3 76.2/78.9 98.3/98.8 B4 380 99.2/97.2 99.7/99.4 99.9/99.5 99.1/95.4 99.7/93.4 99.8/95.1 82.9/85.5 93.1/97.5 90.8/93.2 97.0/92.8 84.9/92.9 74.3/93.3 98.9/94.7 66.5/76.1 97.2/98.3 B5 456 99.0/96.0 99.8/99.4 100.0/99.5 98.9/94.2 98.2/92.8 99.0/90.2 81.5/84.3 93.4/97.2 87.2/90.5 95.6/94.3 83.8/90.8 60.1/92.0 98.9/93.8 75.7/75.4 96.4/98.4 B6 528 95.6/93.6 100.0/99.4 100.0/99.3 98.8/94.1 99.3/94.5 95.0/91.9 76.5/83.2 92.8/96.7 88.1/90.5 97.6/95.6 87.2/91.1 75.9/90.5 92.5/94.5 71.9/73.2 98.7/98.4 Block 5 B3 300 100.0/98.8 99.2/98.3 100.0/99.1 99.9/92.0 100.0/92.3 100.0/96.8 99.4/95.3 98.8/98.3 98.5/95.8 98.7/92.7 95.4/95.5 97.0/99.1 98.6/96.7 100.0/96.6 99.7/98.2 B4 380 100.0/99.0 99.8/99.0 100.0/99.2 100.0/93.5 100.0/94.0 99.9/96.8 99.1/95.1 97.4/98.4 99.1/94.7 99.5/93.1 96.9/94.7 97.2/99.1 100.0/97.6 99.1/96.3 100.0/98.8 B5 456 100.0/99.2 100.0/99.2 100.0/99.4 99.8/93.7 99.9/94.3 99.9/97.0 94.4/93.3 99.1/98.1 96.8/92.0 99.6/93.0 96.8/95.5 97.8/99.3 96.4/95.6 100.0/95.2 99.8/98.6 B6 528 100.0/99.0 100.0/99.3 100.0/99.3 99.3/94.1 99.4/92.9 100.0/95.0 97.9/93.1 99.4/98.3 93.7/94.3 99.8/94.0 95.4/95.8 97.3/99.1 99.7/96.1 100.0/94.6 99.9/99.0 Block 7 B3 300 100.0/100.0 99.7/99.7 100.0/100.0 99.8/99.8 100.0/100.0 100.0/100.0 99.4/99.4 96.2/96.2 99.6/99.6 99.6/99.6 94.6/94.6 94.6/94.6 95.8/95.8 98.6/98.6 96.9/96.9 B4 380 100.0/96.5 99.1/93.4 100.0/95.3 99.9/93.7 99.9/88.4 100.0/93.8 99.1/92.4 89.4/93.3 99.6/95.8 99.7/95.5 95.4/96.5 94.2/96.0 96.1/96.8 98.1/97.4 99.9/96.0 B5 456 100.0/98.6 99.1/94.0 100.0/98.5 99.4/94.4 100.0/89.5 100.0/94.8 98.1/93.1 94.1/97.2 100.0/97.6 100.0/95.0 96.3/96.9 96.5/97.6 95.6/96.8 99.9/97.1 99.8/96.4 B6 528 100.0/97.5 99.9/94.2 100.0/97.2 99.3/93.2 99.7/90.9 100.0/93.4 97.8/94.4 90.9/97.4 98.3/96.5 99.4/93.5 95.6/96.6 98.6/98.3 93.6/97.3 99.0/97.5 99.4/97.1 Table A.15 ResNets’ AUROC(%) detection/segmentation with lateral connections. Sub-datasets Carpet Grid Leather Tile Wood Bottle Cable Capsule Hazelnut Metalnut Pill Screw Toothbrush Transistor Zipper ResNet18 99.9/98.9 99.5/98.4 100.0/99.5 99.8/93.9 99.6/93.6 100.0/97.6 99.4/96.8 96.0/98.7 98.6/98.0 99.9/96.4 97.6/98.3 98.6/98.8 99.7/97.8 99.8/96.5 98.4/98.8 Wide-ResNet50 100.0/99.0 100.0/99.2 100.0/99.5 100.0/96.8 100.0/95.6 100.0/98.9 99.7/97.5 98.8/99.0 100.0/98.3 100.0/97.6 99.9/98.5 99.0/99.4 100.0/98.9 100.0/97.3 99.9/99.2 Wide-ResNet101 99.9/99.0 99.1/99.1 100.0/99.5 99.7/97.2 99.6/95.9 100.0/98.1 99.9/98.4 98.1/98.9 100.0/98.5 100.0/98.1 99.4/98.9 98.1/99.3 99.7/98.1 100.0/97.8 99.5/99.2 Table A.16 EfficientNets’ AUROC(%) detection/segmentation with lateral connections. Models Sub-datasets Carpet Grid Leather Tile Wood Bottle Cable Capsule Hazelnut Metalnut Pill Screw Toothbrush Transistor Zipper B0 99.8/98.0 99.4/97.2 100.0/98.0 98.6/92.1 99.9/89.3 100.0/95.8 99.7/95.4 96.6/96.9 100.0/97.1 99.8/95.3 97.9/97.1 97.6/96.8 94.2/97.0 100.0/97.3 99.7/96.9 B3 100.0/97.3 99.5/95.7 100.0/98.8 99.7/91.5 99.8/88.9 100.0/96.2 99.7/95.5 95.8/96.9 100.0/96.3 99.8/94.3 96.5/96.5 97.0/97.8 97.2/96.1 100.0/97.4 99.2/96.2 B4 100.0/97.8 99.7/95.4 100.0/98.8 99.5/93.3 99.7/92.1 100.0/95.4 99.8/95.7 96.3/97.2 99.8/95.5 99.8/93.2 97.8/96.3 98.8/98.5 96.7/97.3 99.9/97.4 99.8/97.4 B5 99.9/97.7 99.3/95.8 100.0/98.8 98.8/92.1 99.4/89.7 100.0/95.7 99.9/95.7 96.0/97.3 100.0/96.7 99.8/92.8 97.6/95.9 98.6/98.5 94.4/96.6 100.0/97.8 100.0/96.8 B6 100.0/98.0 99.7/96.0 100.0/98.9 99.4/91.9 99.5/91.1 100.0/95.8 100.0/96.0 96.4/97.7 100.0/96.9 99.9/93.5 98.1/96.2 99.1/98.7 98.9/96.8 100.0/97.8 100.0/96.6 Table A.17 Ensemble’s AUROC(%) detection/segmentation. Task Datasets Carpet Grid Leather Tile Wood Bottle Cable Capsule Hazelnut Metalnut Pill Screw Toothbrush Transistor Zipper Detection 100.0 100.0 100.0 100.0 100.0 100.0 99.7 100.0 100.0 100.0 99.7 99.5 100.0 100.0 100.0 Segmentation 99.3 99.3 99.5 97.1 96.0 98.1 97.5 99.1 98.4 98.2 98.7 99.5 98.7 97.6 99.2 Engineering Applications of Articial Intelligence 137 (2024) 109088 19 M. Campos-Romero et al. Table A.18 Low-shot AUROC(%) anomaly detection and segmentation with an ensemble of EfficientNet and Wide-ResNet. Task DA Carpet Grid Leather Tile Wood Bottle Cable Capsule Hazelnut Metalnut Pill Screw Toothbrush Transistor Zipper Detection No 100.0 99.9 100.0 99.9 100.0 100.0 96.6 98.8 100.0 100.0 99.3 96.5 100.0 99.8 99.9 Segmentation 99.4 99.2 99.5 97.0 96.6 98.5 96.7 99.0 98.7 98.5 98.9 99.3 98.9 97.0 99.2 Detection Yes 100.0 99.8 100.0 99.9 100.0 100.0 98.4 99.9 100.0 100.0 99.3 98.7 100.0 99.8 99.9 Segmentation 99.4 99.2 99.5 97.0 96.6 98.5 96.9 99.0 98.7 98.5 98.9 99.3 98.9 97.0 99.2 Table B.19 AUPR, F1-score, precision, and recall detection metrics for MVTec AD obtained with our InAI ensemble proposal. Metrics MVTec AD Bottle Cable Capsule Carpet Grid Hazelnut Leather Metalnut Pill Screw Tile Toothbrush Transistor Wood Zipper Mean AUPR 100.0 99.6 100.0 100.0 100.0 100.0 100.0 100.0 99.9 99.8 100.0 100.0 100.0 100.0 100.0 99.9 Precision 100.0 97.8 100.0 100.0 100.0 100.0 100.0 100.0 100.0 98.4 100.0 100.0 100.0 100.0 100.0 99.8 Recall 100.0 97.8 100.0 100.0 100.0 100.0 100.0 100.0 98.6 99.2 100.0 100.0 100.0 100.0 100.0 99.7 F1-Score 100.0 97.8 100.0 100.0 100.0 100.0 100.0 100.0 99.3 98.7 100.0 100.0 100.0 100.0 100.0 99.7 Table B.20 AUPR, F1-score, precision, and recall detection metrics for MVTec LOCO AD obtained with our InAI ensemble proposal. Metrics MVTec LOCO AD Breakfast box Juice bottle Pushpin Screw bag Splicing connector Mean AUPR 92.5 99.6 85.9 90.1 93.8 92.4 Precision 79.7 97.4 66.0 74.6 85.9 84.4 Recall 86.1 97.0 86.6 96.9 82.4 89.9 F1-Score 82.8 97.2 74.9 84.3 84.1 84.7 Table B.21 Additional segmentation metrics on the MVTec AD with our InAI ensemble proposal. Metrics MVTec AD Bottle Cable Capsule Carpet Grid Hazelnut Leather Metalnut Pill Screw Tile Toothbrush Transistor Wood Zipper Mean Jaccard coefficient 0.57 0.38 0.27 0.44 0.3 0.41 0.41 0.54 0.27 0.22 0.53 0.20 0.34 0.41 0.54 0.39 AUPRO(%) 95.6 88.0 95.9 97.7 98.8 92.1 99.0 89.3 95.7 96.1 96.7 93.3 91.1 93.6 98.4 94.7 Table B.22 Additional segmentation metrics on the MVTec LOCO AD with our InAI ensemble proposal. Metrics MVTec LOCO AD Breakfast box Juice bottle Pushpin Screw bag Splicing connector Mean Jaccard coefficient 0.19 0.30 0.08 0.04 0.08 0.1 AUPRO(%) 83.6 87.7 78.9 70.5 71.2 78.4 The legend denotes our proposed InAI model as the ‘‘Full model’’ (blue). In contrast, the other entries represent ablation experiments: dequantization versus non-dequantization (green); InAI versus iConv (1 ×1 invertible convolution) permutation (yellow); InAI affine transformation versus alternative affine transformation (red); and ActNorm2D vs ActNorm (black); ActNorm2D vs non-normalization (purple). The implementation of the ActNorm and iConv layer follows the official author guidelines (Kingma and Dhariwal,2018). The results indicate that dequantization had a positive impact on model performance. Furthermore, the iConv permutation, despite being a trainable case of the linear transformation performed by the InAI permutation, did not yield better results. Regarding the affine transformations, we experimented with an alternative affine transformation inspired by the CFlow model. This could be equivalent to running half of our affine transformation. However, this alternative did not show better results. When comparing normalization techniques, the ActNorm2D layer demonstrated superior performance, consistently enhancing detection and segmentation accuracy. Another important aspect of the decoder is the number of coupling blocks, denoted as 𝑘in our study. We conducted experiments to determine this optimal number, which implies finding how many transformations are necessary to push the nominal (defect-free training) base distribution to a standard Gaussian. Table C.23 reports the results of evaluating the proposed decoder on the texture sub-datasets from the MVTec AD challenge, indicating that six is the optimal number of Table C.23 AUROC(%) detection performance of InAI in relation to the number of coupling blocks (𝑘) in the decoder. Results are based on the MVTec AD texture datasets. Coupling Blocks (𝑘): 2 4 6 8 10 AUROC: 97.3 99.1 99.8 99.1 96.7 coupling blocks. As the number of transformations increases, the decoder becomes more expressive and capable of effectively modeling the desired Gaussian distribution. However, beyond a certain point, additional transformations begin to degrade the Gaussian structure, causing defect-free images to become indistinguishable from defective ones and leading to a drop in AUROC detection performance. Essentially, increasing the number of transformations is equivalent to increasing the depth of the network. In our design, each layer minimally impacts the computational load, as the most expensive operation is handled by the small residual subnet consisting of three convolutions (as explained in Section 4.) Each 𝑡-transformation requires calculating the Jacobian determinant, which is computationally inexpensive. The affine transformation ensures a triangular matrix, resulting in computational complexity growing only as 𝑂(𝑘), Furthermore, we explored the impact of different hyperparameter configurations. Table C.24 reports the decoder performance under different clamp and epsilon values. The clamp value scales the activations Engineering Applications of Articial Intelligence 137 (2024) 109088 20 M. Campos-Romero et al. Fig. C.16. Performance curves of design ablation experiments on the MVTec dataset. The curves represent the average AUROC(%) performance over epochs across the classes. Table C.24 AUROC(%) detection performance of InAI for MVTec AD in relation to clamp and epsilon hyperparameters of the decoder. Clamp: 1.2 1.4 1.6 1.8 1.9 1.9 2.0 2.5 3.0 Epsilon: 1e-8 1e−7 1e-8 AUROC: 99.1 99.2 99.7 99.7 99.8 99.3 99.6 99.5 99.5 to different limits, stabilizing training and preventing values from exploding in the exponential of the affine transformation (Eq. (12)). On the other hand, excessively small clamp values restrict the model from optimal training. While typically a learnable parameter, we observed faster convergence by fixing it. Specifically, we found a clamp value of 1.6 and an epsilon value of 1𝑒− 8 to be optimal. References Akcay, S., Atapour-Abarghouei, A., Breckon, T.P., 2019. GANomaly: Semi-supervised anomaly detection via adversarial training. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11363 LNCS, pp. 622–637. http://dx.doi.org/10.1007/978-3- 030-20893-6_39. An, J., Cho, S., 2015. Variational autoencoder based anomaly detection using reconstruction probability. Special Lect. IE 2 (1), 1–18. Andor, G., Bohák, A., 2017. Identifying events in financial time series – A new approach with bipower variation. Finance Res. Lett. 22, 42–48. http://dx.doi.org/10.1016/j. frl.2016.11.003. Anthi, E., Williams, L., Rhode, M., Burnap, P., Wedgbury, A., 2021. Adversarial attacks on machine learning cybersecurity defences in industrial control systems. J. Inf. Secur. Appl. 58, http://dx.doi.org/10.1016/j.jisa.2020.102717. Ardizzi, G., De Franceschis, P., Giammatteo, M., 2018. Cash payment anomalies and money laundering: An econometric analysis of Italian municipalities. Int. Rev. Law Econ. 56, 105–121. http://dx.doi.org/10.1016/j.irle.2018.08.001. Bell, A.J., Sejnowski, T.J., 1995. An information-maximization approach to blind separation and blind deconvolution. Neural Comput. 7 (6), 1129–1159. Bergmann, P., Batzner, K., Fauser, M., Sattlegger, D., Steger, C., 2021. The MVTec anomaly detection dataset: A comprehensive real-world dataset for unsupervised anomaly detection. Int. J. Comput. Vis. 129 (4), 1038–1059. http://dx.doi.org/10. 1007/s11263-020-01400-4. Bergmann, P., Batzner, K., Fauser, M., Sattlegger, D., Steger, C., 2022a. Beyond dents and scratches: Logical constraints in unsupervised anomaly detection and localization. Int. J. Comput. Vis. 130 (4), 947–969. http://dx.doi.org/10.1007/ s11263-022-01578-9. Bergmann, P., Fauser, M., Sattlegger, D., Steger, C., 2020. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In: Procedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 4182–4191. http://dx.doi.org/10.1109/CVPR42600.2020.00424. Bergmann, P., Jin, X., Sattlegger, D., Steger, C., 2022b. The MVTec 3D-AD Dataset for Unsupervised 3D Anomaly Detection and Localization. In: Farinella, G.M., Radeva, P., Bouatouch, K. (Eds.), Proceedings of the 17th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP 2022, Volume 5: VISAPP. pp. 202–213. http://dx.doi.org/ 10.5220/0010865000003124. Bergmann, P., Löwe, S., Fauser, M., Sattlegger, D., Steger, C., 2019. Improving unsupervised defect segmentation by applying structural similarity to autoencoders. In: VISIGRAPP 2019 - Proceedings of the 14th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 5, pp. 372–380. http://dx.doi.org/10.5220/0007364503720380. Cai, X., Xiao, R., Zeng, Z., Gong, P., Ni, Y., 2023. Itran: A novel transformer-based approach for industrial anomaly detection and localization. Eng. Appl. Artif. Intell. 125, 106677. http://dx.doi.org/10.1016/j.engappai.2023.106677. Campos-Romero, M., et al., 2022. Train split Surface Defect Salliency of Magnetic Tile dataset. https://drive.google.com/file/d/ 1nwXIg0dfs0FXhQRWUTqU1UEDVEdadbWN/view. (Accessed 17 June 2024). Cohen, N., Hoshen, Y., 2020. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.02357. Dasgupta, S., Gupta, A., 2003. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures Algorithms 22 (1), 60–65. Deco, G., Brauer, W., 1994. Higher order statistical decorrelation without information loss. Adv. Neural Inf. Process. Syst. 7. Defard, T., Setkov, A., Loesch, A., Audigier, R., 2021. Padim: A patch distribution modeling framework for anomaly detection and localization. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12664 LNCS, pp. 475–489. http://dx.doi.org/ 10.1007/978-3-030-68799-1_35. Dinh, L., Krueger, D., Bengio, Y., 2015. NICE: Non-linear independent components estimation. In: 3rd International Conference on Learning Representations, ICLR 2015 - Workshop Track Proceedings. Dinh, L., Sohl-Dickstein, J., Bengio, S., 2017. Density estimation using real NVP. In: 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings. Dixit, P., Silakari, S., 2021. Deep learning algorithms for cybersecurity applications: A technological and status review. Comp. Sci. Rev. 39, http://dx.doi.org/10.1016/j. cosrev.2020.100317. Durkan, C., Bekasov, A., Murray, I., Papamakarios, G., 2019. Neural spline flows. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA. Fernandez, A., Bella, J., Dorronsoro, J.R., 2022. Supervised outlier detection for classification and regression. Neurocomputing 486, 77–92. http://dx.doi.org/10. 1016/j.neucom.2022.02.047. Gudovskiy, D., Ishizaka, S., Kozuka, K., 2022. CFLOW-AD: Real-time unsupervised anomaly detection with localization via conditional normalizing flows. In: Proceedings - 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. WACV 2022, pp. 1819–1828. http://dx.doi.org/10.1109/WACV51458.2022.00188. Hammad, M., Abd El-Latif, A.A., Hussain, A., Abd El-Samie, F.E., Gupta, B.B., Ugail, H., Sedik, A., 2022. Deep learning models for arrhythmia detection in IoT healthcare applications. Comput. Electr. Eng. 100, http://dx.doi.org/10.1016/j.compeleceng. 2022.108011. He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778. Ho, J., Chen, X., Srinivas, A., Duan, Y., Abbeel, P., 2019. Flow++: Improving flowbased generative models with variational dequantization and architecture design. In: Chaudhuri, K., Salakhutdinov, R. (Eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA. In: Proceedings of Machine Learning Research, vol. 97, pp. 2722–2730. Hoogeboom, E., Cohen, T., Tomczak, J.M., 2021. Learning discrete distributions by dequantization. In: Third Symposium on Advances in Approximate Bayesian Inference. Hu, J., Shen, L., Sun, G., 2018. Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7132–7141. Engineering Applications of Articial Intelligence 137 (2024) 109088 21 M. Campos-Romero et al. Huang, C., Guan, H., Jiang, A., Zhang, Y., Spratling, M., Wang, Y.F., 2022. Registration based few-shot anomaly detection. In: European Conference on Computer Vision. Springer, pp. 303–319. Huang, Y., Qiu, C., Yuan, K., 2020. Surface defect saliency of magnetic tile. Vis. Comput. 36 (1), 85–96. Ikeda, Y., Tajiri, K., Nakano, Y., Watanabe, K., Ishibashi, K., 2018. Estimation of dimensions contributing to detected anomalies with variational autoencoders. arXiv preprint arXiv:1811.04576. Kamoona, A.M., Gostar, A.K., Wang, X., Easton, M., Bab-Hadiashar, A., Hoseinnezhad, R., 2024. Anomaly detection of defect using energy of point pattern features within random finite set framework. Eng. Appl. Artif. Intell. 130, 107706. Kanishima, Y., Sudo, T., Yanagihashi, H., 2022. Autoencoder with adaptive loss function for supervised anomaly detection. 207, pp. 563–572. http://dx.doi.org/10.1016/j. procs.2022.09.111, Kingma, D.P., Ba, J., 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. Kingma, D.P., Dhariwal, P., 2018. Glow: Generative flow with invertible 1×1 convolutions. In: Advances in Neural Information Processing Systems, vol. 2018-December, pp. 10215–10224. Kirichenko, P., Izmailov, P., Wilson, A.G., 2020. Why normalizing flows fail to detect out-of-distribution data. In: Advances in Neural Information Processing Systems, vol. 33, pp. 20578–20589. Krizhevsky, A., Hinton, G., et al., 2009. Learning multiple layers of features from tiny images. Krizhevsky, A., Sutskever, I., Hinton, G.E., 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM 60 (6), 84–90. Kullback, S., 1951. Kullback-Leibler divergence. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86 (11), 2278–2323. http://dx.doi.org/10.1109/ 5.726791. Lee, S., Lee, S., Song, B.C., 2022. CFA: Coupled-hypersphere-based feature adaptation for target-oriented anomaly localization. IEEE Access 10, 78446–78454. http://dx. doi.org/10.1109/ACCESS.2022.3193699. Li, C.L., Sohn, K., Yoon, J., Pfister, T., 2021. CutPaste: Self-supervised learning for anomaly detection and localization. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 9659–9669. http: //dx.doi.org/10.1109/CVPR46437.2021.00954. Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M., Le, M., 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Luo, G., Xie, W., Gao, R., Zheng, T., Chen, L., Sun, H., 2023. Unsupervised anomaly detection in brain MRI: Learning abstract distribution from massive healthy brains. Comput. Biol. Med. 154, http://dx.doi.org/10.1016/j.compbiomed.2023.106610. Luo, W., Yao, H., Yu, W., 2024. Template-based feature aggregation network for industrial anomaly detection. Eng. Appl. Artif. Intell. 131, 107810. http://dx.doi. org/10.1016/j.engappai.2023.107810. Peterson, L.E., 2009. K-nearest neighbor. Scholarpedia 4 (2), 1883. Rippel, O., Mertens, P., Merhof, D., 2020. Modeling the distribution of normal data in pre-trained deep features for anomaly detection. In: Proceedings - International Conference on Pattern Recognition. pp. 6726–6733. http://dx.doi.org/10.1109/ ICPR48806.2021.9412109. Ristea, N.C., Madan, N., Ionescu, R.T., Nasrollahi, K., Khan, F.S., Moeslund, T.B., Shah, M., 2022. Self-supervised predictive convolutional attentive block for anomaly detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2022-June, pp. 13566–13576. http://dx.doi.org/10.1109/CVPR52688.2022.01321. Roth, K., Pemula, L., Zepeda, J., Scholkopf, B., Brox, T., Gehler, P., 2022. Towards total recall in industrial anomaly detection. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2022-June, pp. 14298–14308. http://dx.doi.org/10.1109/CVPR52688.2022.01392. Rudolph, M., Wandt, B., Rosenhahn, B., 2021. Same same but DifferNet: Semisupervised defect detection with normalizing flows. In: Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision. WACV 2021, pp. 1906–1915. http://dx.doi.org/10.1109/WACV48630.2021.00195. Rudolph, M., Wehrbein, T., Rosenhahn, B., Wandt, B., 2022. Fully convolutional crossscale-flows for image-based defect detection. In: Proceedings - 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. WACV 2022, pp. 1829–1838. http://dx.doi.org/10.1109/WACV51458.2022.00189. Rudolph, M., Wehrbein, T., Rosenhahn, B., Wandt, B., 2023. Asymmetric studentteacher networks for industrial anomaly detection. In: Proceedings - 2023 IEEE Winter Conference on Applications of Computer Vision. WACV 2023, pp. 2591–2601. http://dx.doi.org/10.1109/WACV56688.2023.00262. Sener, O., Savarese, S., 2018. Active learning for convolutional neural networks: A core-set approach. In: 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings. Song, J., Kong, K., Park, Y.I., Kim, S.G., Kang, S.J., 2021. AnoSeg: anomaly segmentation network using self-supervised learning. arXiv preprint arXiv:2110. 03396. Tabak, E.G., Turner, C.V., 2013. A family of nonparametric density estimation algorithms. Comm. Pure Appl. Math. 66 (2), 145–164. Tan, M., Le, Q.V., 2019. EfficientNet: Rethinking model scaling for convolutional neural networks. In: 36th International Conference on Machine Learning, ICML 2019, vol. 2019-June, pp. 10691–10700. Tang, T.W., Kuo, W.H., Lan, J.H., Ding, C.F., Hsu, H., Young, H.T., 2020. Anomaly detection neural network with dual auto-encoders GAN and its industrial inspection applications. Sensors (Switzerland) 20 (12), http://dx.doi.org/10.3390/s20123336. Tong, G., Li, Q., Song, Y., 2023. Two-stage reverse knowledge distillation incorporated and self-supervised masking strategy for industrial anomaly detection. Knowl.-Based Syst. 273, 110611. http://dx.doi.org/10.1016/j.knosys.2023.110611. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30. Wu, Q., Li, H., Tian, C., Wen, L., Li, X., 2024a. AEKD: Unsupervised auto-encoder knowledge distillation for industrial anomaly detection. J. Manuf. Syst. 73, 159–169. http://dx.doi.org/10.1016/j.jmsy.2024.02.001. Wu, X., Mao, G., Xing, S., 2024b. Unsupervised anomaly detection in images using attentional normalizing flows. Eng. Appl. Artif. Intell. 127, 107369. http://dx.doi. org/10.1016/j.engappai.2023.107369. Yang, M., Wu, P., Feng, H., 2023. MemSeg: A semi-supervised method for image surface defect detection using differences and commonalities. Eng. Appl. Artif. Intell. 119, 105835. http://dx.doi.org/10.1016/j.engappai.2023.105835. Zagoruyko, S., Komodakis, N., 2016. Wide residual networks. In: Wilson, R.C., Hancock, E.R., Smith, W.A.P. (Eds.), Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. Zavrtanik, V., Kristan, M., Skočaj, D., 2021. Draem-a discriminatively trained reconstruction embedding for surface anomaly detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8330–8339. Zhou, W., Wu, S., Wang, Y., Zuo, L., Yi, Y., Cui, W., 2024. DMU-TransNet: Dense multi-scale U-shape transformer network for anomaly detection. Measurement 229, 114216. http://dx.doi.org/10.1016/j.measurement.2024.114216. Engineering Applications of Articial Intelligence 137 (2024) 109088 22