Explaining transformer-based next activity prediction by using attention scores

Käppel, Martin; Ackermann, Lars; Jablonski, Stefan; Härtl, Simon Article — Published Version Explaining transformer-based next activity prediction by using attention scores Process Science Provided in Cooperation with: Springer Nature Suggested Citation: Käppel, Martin; Ackermann, Lars; Jablonski, Stefan; Härtl, Simon (2025) : Explaining transformer-based next activity prediction by using attention scores, Process Science, ISSN 2948-2178, Springer International Publishing, Cham, Vol. 2, Iss. 1, https://doi.org/10.1007/s44311-025-00018-4 This Version is available at: https://hdl.handle.net/10419/323686 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by/4.0/ Open Access © The Author(s) 2025. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. RESEARCH Käppeletal. Process Science (2025) 2:11 https://doi.org/10.1007/s44311-025-00018-4 Process Science Explaining transformer-based nextactivity prediction byusing attention scores Martin Käppel1*, Lars Ackermann2, Stefan Jablonski3 and Simon Härtl3 Abstract Predictive business process monitoring aims to enhance process execution by providing real-time predictions about the future evolution of a process instance. In recent years, several deep learning approaches have been established as state of the art for various predictive tasks, including those based on the transformer architecture. The transformer architecture is equipped with a powerful attention mechanism that assigns attention-based importance scores to each input element, guiding the model to focus on the most relevant parts of the sequence, regardless of their position. This capability leads to more accurate and contextually grounded predictions. However, like most deep learning models, transformers largely operate as a black box, making it challenging to trace how specific features influence the model’s predictions. In this paper, we conduct a series of experiments to examine the role of attention scores in a transformer-based next activity prediction model. Specifically, we investigate whether these scores provide meaningful explanations for the model’s decisions. Our findings reveal that attention scores can indeed serve as effective explanations. Building on these insights, we propose two novel, global, graph-based explanation approaches that illustrate the model’s understanding of the process’s control flow. Our evaluation using various metrics on both real-world and synthetic event logs demonstrates that these explainers effectively capture the model’s decision-making process. By improving interpretability, these insights not only enhance process participants’ confidence in predictive models but also offer a valuable foundation for refining model performance. Furthermore, our investigation into the reliability of attention scores offers valuable insights into how transformer models encapsulate temporal and sequential dependencies in prediction tasks. Keywords: Predictive process monitoring, Transformer, Attention mechanism, Explainability Introduction In recent years, predictive business process monitoring(Maggi etal. 2014; Grigori etal. 2004) has experienced remarkable growth, driven by advances in artificial intelligence (Weinzierl etal. 2024), and has established itself as a subfield in process mining (Di Francescomarino and Ghidini 2022). In contrast to post-mortem analyses of event data, which focus on past and current events, predictive business process monitoring provides runtime support for the execution of a business process by making various predictions *Correspondence: mar[email protected] 1 Chair of Digital Industrial Service Systems, FriedrichAlexander-University ErlangenNuremberg, Fürther Str. 248, 90429 Nuremberg, Germany 2 Department for Computer Science, Hof University of Applied Sciences, Alfred-Goppel-Platz 1, 95028 Hof, Germany 3 Institute for Computer Science, University of Bayreuth, Universitätsstraße 30, 95447 Bayreuth, Germany Page 2 of 42 Käppeletal. Process Science (2025) 2:11 about the future evolution of the process instance (Di Francescomarino and Ghidini 2022). This includes, among others, behavior-related predictions (e.g., next activity (Pasquadibisceglie etal. 2019; Camargo etal. 2019; Evermann etal. 2017)), outcome-related predictions (e.g., outcome (Kratsch et al. 2021)), and time-related predictions (e.g., remaining time (Verenich etal. 2019)). From a business perspective, early knowledge about the future of a process instance offers significant advantages, as it permits improved resource and time planning, better preparation of upcoming steps, and early identification of potential problems (MárquezChamorro etal. 2017; Maggi etal. 2014). The latter, for example, allows process participants to take timely corrective actions to mitigate risks (Di Francescomarino and Ghidini 2022). Typically, predictive business process monitoring approaches involve constructing predictive models based on historical event log data captured by information systems (Grigori etal. 2004). These models are then applied to ongoing process instances to generate valuable predictive insights (Maggi etal. 2014). In recent years, deep learning approaches have been established as state of the art for predictive tasks for mainly two reasons: First, they have proven to outperform traditional machine learning models like decision trees or support vector machines for different targets (e.g., next activity (Mehdiyev etal. 2020), outcome (Kratsch etal. 2021), remaining time (Verenich etal. 2019)) in terms of accuracy and earliness of prediction. Second, they are completely data-driven so that they no longer require an explicit representation of the underlying process model (Senderovich etal. 2019). As a consequence, various kinds of deep learning architectures have been employed, among others Convolutional Neural Networks (Pasquadibisceglie etal. 2019), Long Short Term Memory Neural Networks (LSTM) (Evermann etal. 2017; Camargo etal. 2019), or more recently transformer architectures (Bukhsh etal. 2021). Despite their strong predictive performance, deep learning models largely operate as black boxes, limiting insights into their reasoning and decision-making processes (Nauta etal. 2023). This lack of transparency is a significant factor in why people distrust these models and, hence, marks a major obstacle to their usage in practice (Carvalho etal. 2019). The field of explainable artificial intelligence (XAI) aims to address this issue by developing techniques (so-called explainers) that explain the decisions of machine learning models (Carvalho etal. 2019; Nauta etal. 2023). In predictive business process monitoring both task-agnostic explanation techniques and techniques tailored towards the specific needs of predictive business process monitoring are applied (Weinzierl etal. 2020). In general, explainers are categorized as either local or global based on their explanation scope. While local explainers aim to explain the prediction for a single process instance, global explainers seek to find an explanation to uncover how a model makes decisions across a collection of instances (e.g., an event log). Hence, global approaches provide insights into how a model makes decisions in general, whereas local approaches focus on explaining single predictions. Previous research in predictive business process monitoring has predominantly concentrated on developing local explainers, so the development of process-specific global explainers is still in its infancy. Hence, the hypothesis proposed in the landmark work of Evermann etal. (2017) – that deep learning models for next activity prediction inherently learn the underlying process structure – remains largely untested. Peeperkorn etal. investigated this hypothesis Page 3 of 42 Käppeletal. Process Science (2025) 2:11 specifically for LSTMs, finding that these models sometimes “struggle to learn process model structure”(Peeperkorn etal. 2023). Nevertheless, examining this hypothesis for other deep learning architectures remains a valuable research direction (Peeperkorn etal. 2023), especially given that LSTMs often face challenges with long sequences, as they tend to pay less attention to elements appearing earlier in the sequence. The transformer architecture (Vaswani etal. 2017) addresses this issue with a powerful attention mechanism, which assigns importance scores (so-called attention scores) to each element of the input sequence, guiding the model to focus on the most relevant parts of the sequence, regardless of their position (Bukhsh etal. 2021; Vaswani etal. 2017). This architecture has led to new state-of-the-art approaches across multiple research fields (Vaswani etal. 2017) (e.g., large language models such as GPT-4 (OpenAI etal. 2024) and BERT (Devlin etal. 2019)). As this architecture also demonstrates strong performance in predictive business process monitoring by effectively predicting the next activity (Bukhsh etal. 2021), this paper explores the following research question: Can the attention scores within the transformer architecture provide insights into whether the trained prediction model has developed an understanding of a process’s control flow? This article is an extended and revised version of a previously published conference paper (Käppel etal. 2024). In Käppel etal. (2024), we conducted initial experiments to examine whether the aforementioned attention scores could serve as a solid basis for developing XAI approaches. We also proposed two novel, global, transformer-specific explanation approaches and conducted a quantitative evaluation using real-world event logs. This extended version expands on our previous work by presenting additional experiments to gain deeper insights into the reliability of attention scores. Furthermore, we extend our evaluation to include synthetically generated event logs from process models, enabling a direct comparison between extracted explanation rules and corresponding process models. We also present an embodiment of this approach as a software tool and conduct a qualitative analysis of the extracted explanations. The rest of the paper is organized as follows: Backgroundsection introduces basic terminology and the fundamentals of the transformer architecture. After positioning our work with respect to related research (Related worksection), we conduct a series of experiments to investigate whether attention scores can serve as an explanation (Prestudy: the relevance of attention scoressection). Building on these findings, we present the proposed global explainers (Explanation approachessection). In Evaluationsection, we perform a qualitative and quantitative evaluation of our approach on both real-world event logs and synthetically generated event logs. Concluding remarkssection discusses potential limitations and implications for theory and practice, while Conclusion and future worksection provides directions for future research. Background In this section, we first introduce key concepts and notations from the field of process mining that are essential for understanding the remainder of the paper. Next, we explain the functionality of the transformer architecture, with a strong focus on its attention mechanism, which plays a pivotal role in the proposed explanation approaches. Page 4 of 42 Käppeletal. Process Science (2025) 2:11 Event logs andnextactivity prediction A business process is a sequence of activities and decisions carried out to deliver a valuable outcome to the customer (Dumas etal. 2018). Each execution of such a business process is called a process instance or a case(van der Aalst 2016)). Modern IT systems record and store information about process executions in the form of event logs – sets of timestamped events that include various event attributes encapsulating information about the execution of activities (van der Aalst 2016)). In the following, we denote the set of activities of a business process as A and define an event formally as follows: Definition 1 An event is as a tuple e=(a,c,t,d1, ..., dm) , where a∈A is the executed activity, c is a case identifier indicating the process instance to which the event belongs, t is the timestamp of execution, and d1, ..., dm represent the data payload, i.e., optional event attributes related with the execution of activity a. Thus, at a minimum, an event contains the following event attributes: a case identifier, the executed activity, and the timestamp of execution (van der Aalst 2016)). Accordingly, we use functions πa(e) , πc(e) , πt(e) , and πd1(e),...,πdm(e) to access the activity, case identifier, timestamp, and the data payload of an event e(van der Aalst 2016)). All events belonging to the same process instance can be temporally ordered by their timestamp into a so-called trace(van der Aalst 2016)): Definition 2 A trace is a non-empty, finite sequence of events σ=�e1,...,en� such that, for 1≤i<j≤n , the following conditions hold: • all events are ordered according to their timestamp (i.e., πt(ej)≥πt(ei) ) and • all events belong to the same process instance (i.e., πc(ej)=πc(ei) ). The length of a trace, denoted by |σ| , refers to the number of events within that trace. Based on this definition, we can define an event log as follows (van der Aalst 2016)): Definition 3 An event log L is a set of traces of the same business process. The size of the event log, denoted by |L| , is defined as the number of traces contained in L. To represent process instances at different points in time, we utilize the prefixes of a trace. Definition 4 Let σ=�e1,...,en� be a trace and r∈{1, ...,n−1} . The prefix of a trace σ of length r is defined as a function hd that returns the first r events of σ , i.e., hd(σ ,r)=�e1,...,er� . Each prefix thus consists of a sequence of consecutive events from the start of the trace up to a specific point, capturing the process execution up to that moment. A next activity prediction model receives a record of a running process instance (i.e., a prefix) as input and predicts the most likely subsequent activity. We therefore define next activity prediction formally as a function (Rama-Maneiro etal. 2021): Page 5 of 42 Käppeletal. Process Science (2025) 2:11 Definition 5 Let σ=�e1,...,er� be a prefix of length r. Next activity prediction is defined as a function  that predicts for the activity of the next event er+1 , which is not yet known. Thus, a next activity prediction approach aims to train a prediction model that approximates this function  using a given event log. Most predictive business process monitoring approaches employ machine learning algorithms to learn the function  . In this paper, we do not define a new architecture but instead use the transformer architecture proposed by Bukhsh etal. (2021). Since this architecture considers solely the activity event attribute and ignores the remaining event attributes of a prefix, we can – without loss of generality – represent traces and prefixes simply as sequences of activities. Thus, we denote a trace or a prefix as follows: σ=�πa(e1),...,πa(en)� . Transformer architecture andattention mechanism The transformer architecture was originally introduced in Vaswani etal. (2017) as a sequence-to-sequence model built on an encoder-decoder architecture equipped with a robust attention mechanism. Sequence-to-sequence modeling refers to tasks that convert an input sequence into an output sequence of potentially different lengths, such as language translation (Zhao etal. 2023). Since its introduction, the transformer architecture has also been adapted for sequence-to-vector tasks, where an input sequence is processed to output a fixed-size vector. Such adaptations are particularly relevant for classification or regression tasks, including next activity prediction or remaining time prediction (Bukhsh etal. 2021). This adaptation is achieved by omitting the decoder part of the architecture, which is why this variant is also called an encoder-only transformer(Devlin etal. 2019). In this work, we focus on describing this modified transformer architecture as applied to next activity prediction. Specifically, we employ the process transformer architecture outlined in Bukhsh etal. (2021) (see Fig.1). The following sections describe the key components of the architecture relevant to this study: the input, the attention mechanism, and the output of the transformer. For details on other aspects of the architecture, we refer the reader to Vaswani etal. (2017) and Bukhsh etal. (2021). Input The transformer receives a sequence of activities σ=(a1,...,an) as input. This input is then embedded into a high-dimensional space of dimension dm∈N>0 1 by converting each element in the sequence into a so-called embedding vector(Vaswani etal. 2017). As a result, for each element ai of the sequence, two embedding vectors of dimension dm are generated: a vector representing the element itself (so-called input embedding) xi in ∈R 1×dm and a vector representing the position (so-called position embedding) xi pos ∈R 1×dm . Following common practice in transformer implementations, these embedding vectors are represented as row vectors. Because the transformer processes the entire input sequence in parallel and does not inherently capture positional information, position embeddings are essential to avoid losing the order of the elements. Both vectors are then summed component-wise, 1 In the architecture used in this paper dm is set to 36 (Bukhsh etal. 2021) Page 6 of 42 Käppeletal. Process Science (2025) 2:11 resulting in a continuous representation of the input element that captures positional, semantic, and syntactic properties (Vaswani etal. 2017). This resulting vector, denoted as xi∈ R 1×dm , serves as input for the transformer block (Vaswani etal. 2017). Notably, these embeddings are automatically learned during training, allowing the model to dynamically adapt to the semantic and positional characteristics of the input data. Transformer block The heart of the transformer architecture, and central to its capabilities, is the multi-head self-attention mechanism. This specialized form of attention significantly enhances the model’s ability to capture relationships within sequences. In general, an attention mechanism allows deep learning models to dynamically focus on different parts of an input sequence when generating output. The core idea behind attention is to assign varying scores to different elements in the input, indicating how much focus the model should pay them when performing a task. Hence, it allows to prioritize most relevant information leading to more accurate and contextual output. Attention mechanism can be integrated into various deep learning architectures. In the transformer architecture introduced in Vaswani etal. (2017), a refined version of the attention mechanism, known as self-attention is proposed. This mechanism enables the model to attend to all elements within the input sequence simultaneously, enhancing its ability to capture both local and global dependencies. Self-attention mechanism: In the self-attention mechanism, the embedding vector xi is utilized in three distinct ways: as a query vector qi , a key vector ki , and a value vector vi . Fig. 1 Overview about the transformer architecture proposed in Bukhsh et al. (2021) Page 7 of 42 Käppeletal. Process Science (2025) 2:11 These vectors are obtained by multiplying the embedding vector xi with the projections matrices WQ ∈R d m ×d k , WK∈ R d m ×dk , and WV∈ R d m ×dv : Because WQ , WK , and WV transform xi in a lower-dimensional space, they are referred to as projections. These matrices are automatically learned during training. According to Vaswani etal. (2017) dk and dv are typically set to the same value. In order to enhance computational efficiency, these vectors are packed row-wise into matrices Q∈ R n×dk , K∈Rn×d k , and V∈Rn×dv (Vaswani etal. 2017). We then employ Q, K, and V to calculate an attention score that reflects the relative importance of each element (query) in relation to the others in the input sequence. Specifically, each element in the input sequence is compared to the currently considered element by computing the matrix product QKT∈ R n×n . The values of the resulting n×n matrix are scaled by dividing by d k and then normalized by applying a row-wise softmax operation: This normalization mitigates the risk of vanishing gradients and improves training efficiency (Vaswani etal. 2017). The resulting matrix Mσ is called the attention score matrix or simply the attention matrix2. Notably, the dimension of the attention score matrix depends on the length of the input sequence, resulting in attention score matrices of varying sizes for input sequences of different lengths. Finally, each value vector vi (contained in V) is multiplied with the attention scores: This step is intended to reduce the impact of value vectors that receive very low attention scores. The resulting matrix Attσ is the output of the self-attention mechanism. Multi-Head Attention: To capture diverse relationships and patterns between the elements in the input sequence, we use h independent, parallel attention mechanisms, referred to as attention heads or heads. These heads are then combined into a so-called multi-head attention(Vaswani etal. 2017): with headj σ=Attj σ . In this formula, the attention score matrices from each attention head j are concatenated (i.e., the matrices are arranged side by side) resulting in a n×hdv matrix. This concatenated matrix is then multiplied by a learned weight matrix WO∈ R hd v× dm to project the result back to the initial dimension dm . In the result of the multi-head attention, each q i=xi·WQ∈R 1×dk k i=xi·WK∈R1×d k vi=xi·WV∈ R1×d v M σ=softmax  QKT  d k. Attσ=Mσ·V∈ R n×d v . MultiHeadAttentionσ =concat(head 1 σ ,...,head h σ )W O ∈R n×dm 2 Please note that QKT is often referred to as attention score matrix in the literature. Page 8 of 42 Käppeletal. Process Science (2025) 2:11 row represents an enriched context vector of an element of the input sequence, while each column represents a specific feature across all elements. To ensure that each head does not require the full dimension dm , the dimensions dk and dv are set to d m h . This adjustment keeps the total computational cost comparable to that of a single-head attention mechanism. The process transformer presented in Bukhsh etal. (2021) employs four attention heads, resulting in dk=dv=9 . The core idea behind the multi-head attention mechanism is that each attention head can potentially learn different relationships or types of dependencies between the elements of the input sequence. Because each head operates independently with its own weight matrices, it can focus on distinct patterns and dependencies. By calculating multiple attention heads in parallel, the model can capture more diverse and complex relationships within the sequence than it would be possible with a single attention mechanism. Interpretation of the attention score matrices: Fig.2 depicts an example of the attention score matrices for all heads obtained for a prefix of a real-world event log. Interpreting these attention score matrices is essential for understanding the explanation approaches and is therefore described in detail. All attention score matrices can be interpreted identically, regardless of the head. Each attention score matrix illustrates how the elements in the input sequence relate to one another. In these matrices, rows and columns correspond to elements of the input sequence. While the rows represent these elements as queries, the columns represent them as keys. By examining a row, we can assess the importance the transformer model assigns to each other element in the input sequence from the perspective of the element represented by that row. In contrast, inspecting a column reveals the level of attention other elements in the input sequence assign to the key element associated with that column. The entries of the matrix indicate Fig. 2 Attention scores of all heads for the prefix A_SUB, A_PAR, W_Afh, W_Afh, A_PRE, W_Com of the BPIC12 event log Page 15 of 42 Käppeletal. Process Science (2025) 2:11 Before calculating the common feature importance, we extract a directly follows graph (DFG) from the given event log. The DFG is then used to select activities that can be used to replace activities in the traces in a realistic way. Realistic means that it is likely that the modified trace represents a valid process execution. Calculation of common feature importance: For calculating the common feature importance fcom for a trace σ , we proceed as follows: We first create a collection of modified versions of the original trace σ . Therefore, we iterate over all events in σ to generate for each event a set of modified traces in which the particular event ei has either been masked out or its associated activity has been replaced by another activity (or sequence of activities). In the case of a replacement, we select the replacement activity in dependency of the position of ei with the help of the DFG: • If ei is the first event of σ , we identify all possible start activities of the process in the DFG. Then we replace ei with each possible start activity to get several modified traces. • If ei is the last event in the trace, we replace its activity with all activities that can succeed the preceding activity πa(ei−1) according to the DFG. • For any other position (i.e, if ei is neither the first nor the last event), we substitute ei with all possible paths in the DFG that connect the activity of the preceding event, πa(ei−1) , to the activity of the subsequent event, πa(ei+1) . The union LM of all modified versions of σ obtained across all variations can then be used to compute common feature importance values per activity. Therefore, we initially set the common feature importance for each activity to zero. For each modified trace σm∈LM , we first determine the replaced activity and compute then the cosine similarity of the prediction vector obtained for σm and the prediction vector for the original trace σ . This similarity value is then added to the common feature importance of the replaced activity. It is important to note that we add this score regardless whether the prediction for σ and σm differs, because we are interested in influence of the activity on the prediction. The larger the difference of the prediction vector, the higher is the impact. After evaluating all modified traces, we compute the mean of all similarity values belonging to the same replaced activity. Finally, we normalize the mean values so they sum up to 1.0 to obtain a probability distribution over the activities. Please note, that this feature importance is independent of the attention scores and only relates features and prediction. Calculation of attention score feature importance: For calculating the attention score feature importance fatt , we rely on the same collection of modified traces LM as before. If the prediction of the original trace and a modified trace are similar (i.e., their cosine similarity is lower than δsim ), we determine the attention scores per activity from the transformers’ attention score matrices. After inspecting all modified versions, we compute the mean of the cumulated attention scores per activity. It is crucial that we only consider modified traces with similar predictions to the original trace, because we want to identify, which activities are important for a particular prediction. Finally, we calculate Kendall’s tau rank correlation τ (Kendall 1938) between common feature importance and attention score feature importance. Concerning interpretability Kendall’s tau is scaled into the continuous range [−1, 1] , where −1 indicates a perfect Page 16 of 42 Käppeletal. Process Science (2025) 2:11 negative correlation, 0 indicates no correlation, and +1 indicates a perfect positive correlation. Hence, a low correlation means, that the relative importance of the features is not sufficiently reflected in the attention scores. Thus, the explanations derived from the attention scores would not use the same reasoning as the transformer to be explained. Hence, for reliable attention scores large values are preferable. The results of the correlation of common and attention score feature importance are shown in Fig.3. For the majority of the event logs the median correlation is slightly larger than 0.5. Following the guidelines proposed in Akoğlu (2018) after which values τ<0.3 , 0.3 ≤τ≤0.5 , τ>0.5 indicate a weak, moderate, and strong correlation respectively, we observe a moderate to strong association. Nevertheless, for some of the event logs, the correlation is closer to zero, indicating no correlation. The latter applies to BPIC12, BPIC12-WC, and the Sepsis event log. In summary, the correlation between the common feature importance and the attention score feature importance is often significant, giving a first indication of reliable attention scores. Experiment 2 – Attention Mechanism Parameter Manipulation The next experiment examines the necessity of the attention mechanism, adhering to the experimental setup outlined in Wiegreffe and Pinter (2019). To this end, we first train exactly one baseline model Mb without any modifications in the training process. Then we train multiple manipulated transformer models where we either randomly initialize the attention parameters (i.e., matrices WQ , WK , WV , and WO ) or freeze the attention parameter to uniform values during training. The latter means that the attention score mechanism is completely eliminated, while the first means that the training process probably ends with different attention scores. Both modified training procedures are repeated five times to mitigate the influence of change, each initiated with unique random seeds to ensure variability across the trials. Fig. 3 Results for the Kendall Tau correlation of common and attention feature importance Page 17 of 42 Käppeletal. Process Science (2025) 2:11 Subsequently, each manipulated transformer Mm (both the randomly initialized and the frozen) is compared against the baseline Mb . To this end, we send each test sample through all trained models (i.e., the baseline model and the manipulated transformers) and extract each time the corresponding attention score distribution and the prediction vector. The attention score distributions and prediction vectors derived from Mb are then compared to those obtained from the corresponding Mm using JSD and TVD, respectively. To aggregate the values, we compute the average JSD and TVD values across all test samples for each model. The rationale behind this experimental design is to assess whether varying attention score distributions (indicated by higher JSD values) lead to invariant predictions (indicated by lower TVD values) (Wiegreffe and Pinter 2019; Jain and Wallace 2019). Thus, if identical predictions can be achieved with substantially different attention scores, this would imply that the attention mechanism barely affects the decision process of the model. Hence, the attention scores would have only minimal explanatory value. To facilitate interpretation, we plot the mean JSD against the mean TVD achieved for each model (see Fig.4). Models positioned further to the right and lower in the plots represent cases with less reliable attention scores. Notably, the majority of the event logs, regardless of whether they are real-world or synthetically generated, are positioned in regions indicating reliable attention scores. In cases where the models tend to be on the right side (Complex Model, Looped And, BPIC12, and BPIC12-W) the TVD values are significantly above zero. For Sequence and LDD, however, TVD values remain close to zero, suggesting that predictions are only minimally affected by changes in attention scores. Due to the simplicity of Sequence and LDD this observation is not surprising, since for most of the test samples the next activity is obvious, so that the model takes other options barely into account. At the same time, both Sequence and LDD are also on the left side, indicating only minor variations in the attention scores. Overall, these results suggest that the attention scores can be trusted. To further investigate the stability of attention scores, we analyze the maximum attention value in a distribution, i.e., max(ασ) , in relation to JSD. The motivation for this analysis is the hypothesis that high maximum attention values indicate a model’s strong focus on a particular input element. If this focus is consistent across different model initializations, it could suggest that strong peaks in the attention distribution align with stable and meaningful model behavior. Conversely, if high maximum attention values are associated with high JSD values, it would imply that even seemingly confident attention peaks may be unreliable and vary substantially across different model instances. By examining this relationship, we aim to assess whether strong attention peaks can serve as robust explanatory indicators. Following Jain and Wallace (2019), we conduct a binning procedure in which all attention score distributions of the samples regardless of the model are categorized based on its maximum attention value. By combining results from multiple models, the analysis ensures that observed patterns are robust and not artifacts of specific model initializations. Specifically, each attention score distribution is assigned to one of four distinct bins, each defined by a numerical range for maximum attention values. Through the binning, we aim to examine how varying degrees (represented by the ranges of the bins) of selective focus correlate with shifts in attention distributions. Page 18 of 42 Käppeletal. Process Science (2025) 2:11 Figure5 visualize the distribution of JSD divergences (compared to the base model) as a function of the maximum attention value across all models in form of a violin plot. The height of each violin is scaled according to the number of instances in each bin, indicating how often JSD values lie in certain ranges. Thus, a wide area in the violin means that many JSD values are concentrated in this area. The violins can be interpreted as follows: A right-weighted violin means that the JSD values in this bin tend to be high. This indicates that the attention values for this bin are susceptible to manipulation. In contrast, a left-weighted violin (wider left, with low JSD values) means that the JSD values in this bin are predominantly low. This indicates that the attention scores for this bin are robust and less manipulable. The plots reveal that most attention score distributions fall within either the first or last bin, with minimal representation in the second and third bins. This pattern suggests Fig. 4 JSD vs. TVD plots. Rectangles = models with frozen weights, triangles = seeded base models. JSD can only take values between 0 and log(2)=0.693 . The horizontal line at 0.1 serves as reference value to compare the process-specific results, with results obtained on general machine learning datasets. The value stems from Wiegreffe and Pinter (2019), where all TVD values were lower than 0.1 Page 19 of 42 Käppeletal. Process Science (2025) 2:11 that the model tends to react to certain elements either very strongly or hardly at all. Since the middle bins (second and third bins) are only minimally represented, the model rarely shows moderate attention to an element. If attention score distributions fall into the last bin (0.00–0.25), there is often a rightward tendency. The first bin (0.75–1.00), on the other hand, shows a more heterogeneous picture, with some datasets tending to the left (BPIC13_CP, BPIC12_O, BPIC13-I, BPIC12_WC, Sequence) and two datasets (Complex, Looped And) showing a strong rightward tendency. The remaining logs are more centralized. Of particular interest are Fig. 5 All JSD vs. maximum value in the attention score distribution violin plots (scaled by count). The height is scaled by the count of instances per bin Page 20 of 42 Käppeletal. Process Science (2025) 2:11 cases where the last bin shows a rightward tendency while the first bin shows a leftward tendency (e.g., BPIC12_W, Sequence, BPIC13_CP). In these cases, instances with low attention scores are more susceptible to manipulation than those with higher attention scores. Overall, this binning approach shows that the interpretability of attention scores can vary within a dataset and that particular maximum attention thresholds may serve as indicators of model robustness. The detailed analysis also supports the initial findings from the first part of the experiment that the attention scores are predominantly trustworthy, suggesting they are suitable for explanatory purposes. Experiment 3 – Attention Score Masking In a third experiment, we investigate how particular attention values affect the prediction (see Fig.6). In detail, we want to check whether the attention scores are closely linked with the information in the input sequence. Therefore, we examine whether removing an element from the input sequence has the same effect in terms of prediction as removing the element’s attention scores. To do this, we once mask elements in the input sequence and once the corresponding attention scores in the heads’ attention score matrices and compare the model outputs via TVD. In the input sequence, we mask elements by replacing them with a padding symbol (_), indicating to the model that there is no element. In contrast, masking the attention scores is achieved by setting all values in the relevant rows and columns of the attention score matrices to zero. Figure6 illustrates this conceptual difference between masking the input prefix and the attention scores of the heads’ attention score matrices. We perform this procedure for all test samples, whereby each element in the sample is masked one time. For each of the masked samples, we obtain two prediction vectors: a prediction vector pm for the masked sample and a prediction vector pam for the unmasked sample, where we masked the attention scores. We then compare pm and pam by calculating their TVD. Low TVD values suggest that masking an element in the input sequence has a similar impact on the prediction as masking it within the attention score matrices. From this, Fig. 6 The former masks events (here activity B) directly in the prefix. The latter only masks rows and columns corresponding to the event in the attention score matrices. Predictions may be different Page 21 of 42 Käppeletal. Process Science (2025) 2:11 it can be concluded that particular attention scores have relevance and can serve as an explanation. We observe that the histograms of TVD values (see Fig.7) show a strong left skew, indicating that masking in the input prefix leads to almost identical predictions as masking in the attention score matrices. Minor deviations from this behavior appear only in the histograms for the Sequence and LDD event log. Conclusion The overall result of this series of experiments is that the attention scores possess good reliability. Only very simple event logs such as Sequence or LDD (both Fig. 7 Variation between predictions TVD between masked elements in the prefix and only masked attention scores matrix Page 22 of 42 Käppeletal. Process Science (2025) 2:11 synthetically generated) deviate slightly from this observation, although both still have sufficient trustworthiness. This finding that the attention scores appear less reliable for very simple datasets is also consistent with the results in the study of Wiegreffe and Pinter (2019), where the same observation was made for datasets from other domains. The reason for this anomaly lies in the simplicity of the underlying task so that the model does not require the powerful attention mechanism at all. Based on the results obtained, it is justified to build explanation approaches with attention scores as a pivotal component. Therefore, especially the results from Experiment 2 are beneficial, for getting deeper insights how strong the attention scores are pronounced. Most of the time the attention score distributions fall either in the first bin (maximum attention score very high) or the last bin (maximum attention score very low) but barely in the second or third bin. Hence, we have either at least a very high attention score (first bin) or a lot of small attention score (last bin). This observation helps us to define thresholds, when to consider an element in the input sequence as relevant or not. Explanation approaches In this section, we introduce two global transformer-specific explanation approaches. Each explainer receives as input a trained transformer prediction model and a set of prefixes L to be explained. Both explainers construct an interpretable directed graph that visualizes the control-flow of the process, serving as an indicator of the extent to which the prediction model captures the control-flow of the process. Because our preliminary study in Pre-study: the relevance of attention scoressection provides strong evidence that attention scores are reliable for next activity prediction, our explanation approaches leverage them as a crucial component. Explainer 1: backward explainer The first explainer (so-called Backward Explainer) generates explanations for individual prefixes and subsequently integrates these into an overarching explanation for all prefixes in L. This procedure is explained in the remainder of this section and involves creating a local graph Gσ for each prefix σ∈L , which is then merged directly into the global graph G. Creating a local graph An essential step in constructing a local graph Gσ for a prefix σ is to utilize the heads’ attention scores to identify relevant activities that are decisive for the prediction pσ . To do this we proceed as follows: An activity is considered relevant for the prediction pσ if its aggregated attention score across multiple modified versions of σ is sufficiently high. To generate these modified versions of σ , we apply randomly modification operations. Each modified prefix σm is then passed to the transformer to obtain prediction vector pσm and the attention score matrices Mi σm for a head i. To check whether the prediction for σm is still close to the prediction of σ , we compute the cosine similarity between pσm and pσ and compare it to a predefined threshold δsim . Page 23 of 42 Käppeletal. Process Science (2025) 2:11 If σm satisfies this condition, the attention scores for each element j in σm are aggregated across the different heads as follows: In this equation attention score matrices M i σm from each head are first summed component-wise, and then the jth column is summed. This yields an aggregated attention score vector ησm=(S1, ..., S|σm|) for each modification σm containing a total attention score for each event. This procedure is illustrated in Fig.8. Next, the total attention scores are further aggregated per activity to a vector ψσ by summing up the scores for all events belonging to a certain activity. The vector ψσ is then normalized to range within [0, 1]. We denote the total attention score for activity a as ψσ(a) . Finally, we filter activities based on a threshold δattr , keeping only activities with ψσ(a)>δ attr to get only those activities with particularly high total attention scores. Thus, the set of relevant activities for input sequence σ is defined as Ar={a∈A|ψσ(a)>δ attr } . As a default value we set δattr to 0.1. This filtering is essential to exclude irrelevant elements that may have low but non-zero attention scores. Without this step, nearly all activities would be classified as relevant. Analogously, we identify the most likely next activities, denoted as Pr , from the prediction vector pσ , by selecting only those activities a with prediction scores exceeding an a-priori defined threshold δpred , i.e., p σ (a)>δ pred . As a default value for δpred we used 0.1. We do not limit the selection to a fixed number of activities from pσ , because depending on the process the confidence of the model varies: for some prefixes, it may indicate that only one activity is probable (e.g., in case of a strict sequence), while in others (e.g., at an AND split), multiple activities appear viable to be executed next. Building on this we can formally define local graph as follows: Definition 8 Let σ be an input sequence, Ar the set of relevant activities identified in σ , and Pr the set of likely next activities for σ . A local graph for σ is a tuple Gσ=(V,E) with • V=Ar∪Pr being the set of nodes and • E={(s,t)|s∈Ar,t∈Pr} being the set of edges. S j= |σ m |  n=1 n  i=1 Mi σm nj . Fig. 8 Determining aggregated attention scores for each event in the prefix, assuming that M possesses two heads Page 24 of 42 Käppeletal. Process Science (2025) 2:11 Thus, for constructing the local graph Gσ , we use all elements in Ar and Pr as nodes and establish edges between all nodes from Ar to all nodes from Pr . Let us illustrate the procedure on the example depicted in Fig.9. Given the prefix σ1=�B,A,C,B,E� , the aggregated attention scores highlight activities B and C as relevant for the prediction, i.e., Ar={B,C} . Additionally, the prediction vector pσ1 suggests that either activity B or D is likely to occur next, i.e., Pr={B,D} . According to Definition 8 we get nodes V=Ar∪Pr={B,C,D} and edges V={(C,B),(B,B),(B,D),(C,D)} . This graph can then be interpreted as follows: for executing activity D, it is crucial that activities B and C have been executed before. Merging local graphs Each local graph Gσ is directly integrated into the global graph G. This integration involves adding the vertices and edges of Gσ to G if they are not yet included. To illustrate this step, suppose we have, in addition to the local graph Gσ1 from our example in Fig. 9, a second local graph Gσ2=({A,C},{(A,C)}) . The resulting graph after merging Gσ1 and Gσ2 would then be G=({A,B,C,D},{(C,B),(B,B),(B,D),(C,D),(A,C)}) . After merging a local graph into G, we apply a pruning step to remove undesired shortcuts that are unlikely to occur in the underlying process. Such shortcuts can arise if an activity in the front part of σ receives high attention for the prediction, but this activity is only indirectly required for the predicted activity, in that sense that this activity and the predicted activity are not directly sequential. To bring the global graph closer to a process model, we apply as heuristic, that we remove such shortcuts, that lead from an activity lying long back in the prefix to an activity that occurs later in the prefix. Formally, we remove edge (u,v)∈E if there exist edges (u,an)∈E and (an,v)∈E , for an activity an . Fig ure10 illustrates this step on a short example. Please note, since we generally remove shortcuts there is a risk, that we may remove a desired shortcut. Since the idea is that we analyze all prefixes in an event log, the intention was that, direct connections are sufficient. Explainer 2: attention exploration explainer The Backward Explainer has a key limitation: it directly defines the edges of both the local and global graph when analyzing a prefix. Once an edge is inserted, it remains in Fig. 9 Example for BackwardExplainer. Relevant activities and likely next activities are highlighted boldfaced in red color in the input prefix and prediction vector Page 31 of 42 Käppeletal. Process Science (2025) 2:11 Below, we provide a brief overview of the intent behind the metrics and how they can be computed. For further details we would like to refer to Nauta etal. (2023): • Correctness: This metric evaluates the truthfulness of an explanation (Nauta etal. 2023). In the literature, the correctness is measured in two different ways: (i) by comparing the explanations to ground-truth explanations, or (ii) by calculating the correlation between the explainer’s feature importance and the model’s feature importance. Because ground-truth explanations are not available in our setting, we rely on the second option. The calculation of the feature importance of explainer and model is identical to those used in the first experiment of our pre-study. Thus, this metric ranges from [−1, 1] , with higher values indicating that the explainer primarily uses the correct feature for its explanations. • Completeness: This metric measures how closely the explanations coincide with the predictions of the black box model (Nauta etal. 2023). To calculate completeness, we treat the most likely next activities Pr of the transformer as ground truth and the explanations (to be concise the right-hand sides) as predictions to be evaluated. Hence, we evaluate the explainer analogously to a machine learning model by comparing its predictions with ground truth data. Because the predictions are represented by one or more activities this results in a multi-label confusion matrix. From this matrix, we derive metrics such as precision, recall, and F1-score. Higher values indicate better completeness, meaning that the output of the explainer matches the predictions of the transformer. • Consistency: The consistency metric judges the determinism of an explainer, i.e., whether an explainer provides the same explanations for two distinct models that yield similar outputs. To compute this metric, we first train an initial model M1 and then five additional models. From these, we select the model with the greatest difference from M1 based on the mean difference between their trainable weights. In the following, we denote this model with M2 . For each test sample σ , we calculate a consistency score cσ if the predictions of M1 and M2 for σ are similar (evaluated using the cosine similarity) and if their feature importances align. The consistency score cσ is computed as follows: where R1 and R2 are the rule sets generated by the explainer for M1 and M2 , respectively, and the Jaccard coefficient measures the similarity between these rule sets. The intention behind this formula is that the difference between the similarity of the prediction vectors (i.e., |1−cosineSim(p1(σ ),p2(σ )) ) and the Jaccard coefficient should be close to zero. Or in other words: If the predictions are not similar, the Jaccard coefficient should not be either and vice versa. The consistency scores is averaged across all test samples to lie in the interval [0, 1]. A value of 1 indicates optimal consistency, i.e., that the explainer provides identical explanations for models with the same outputs. • Continuity: The continuity metric evaluates whether an explainer provides similar explanations for slightly altered inputs (Nauta etal. 2023). The intent behind this metric is, that if a model provides stable, similar explanations for slightly modified inputs, this suggests that the explainer could also provide meaningful and (1) cσ=1.0 −|1−cosineSim(p1(σ ),p2(σ )) −Jacard(R1,R2)|, Page 32 of 42 Käppeletal. Process Science (2025) 2:11 consistent explanations for similar, previously unseen data points (Nauta etal. 2023). Thus, it can be used as an indicator of the robustness of explanations. By varying the input, we examine the extent to which the new explanation deviates (Nauta etal. 2023). Therefore, we create for each test sample σ a collection of modified versions of σ by applying masking and replacing as in the pre-study. For each modified sample σm , we compare the corresponding prediction vector pσm with the prediction vector pσ obtained for the original sample σ . If these vectors are similar (i.e., cosineSim(pσm , p σ >δ pred ), we calculate a continuity score using Equation1. Here, the rule sets R1 and R2 , represent the explanations generated for σ and σm , respectively. The final continuity score is obtained by averaging the continuity scores across all modified samples. Similar to the consistency metric, this score takes values in the interval [0, 1], whereas 1 indicates perfect continuity (Nauta etal. 2023). • Contrastivity: The contrastivity metric serves as the counterpart to the continuity metric. While the continuity metric focuses on modifications that result in similar predictions, contrastivity examines modifications that produce different predictions. Therefore, we only consider modified versions σm of σ , if cosineSim(pσm,pσ)<δ pred (i.e., we invert the condition compared to the continuity metric). Apart from this modification, the metric is calculated in the same way as the continuity metric. Thus, it evaluates how distinct the explanations of an explainer are for dissimilar inputs (Nauta etal. 2023). This metric also ranges from [0, 1], with a value of 1 indicating perfect contrastivity. • Compactness: The compactness metric measures how concise an explanation is. Therefore, two different aspects can be measured: the number of rules generated by an explainer and the average length of the rules’ right-hand sides. Due to the limitations of human cognitive capacity, shorter rules are preferable, because they are easier to understand. Consequently, this metric is particularly relevant in practical applications. However, it is important to note that this metric favors brevity, which does not necessarily correlate with the quality of the explanations. An explanation with fewer or shorter rules may lack essential information and, thus, may not always be the most comprehensible or accurate. For this reason, compactness should only be evaluated in conjunction with other metrics. To apply the metrics, a fixed percentage (80%) of prefixes is randomly selected from each event log to accommodate their varying sizes. Metrics are then computed for each prefix individually, followed by averaging these values. Table3 reports these average values along with their standard deviations. In the following, we abbreviate Backward Explainer as BE and Attention Exploration Explainer as AE. Discussion of the results We consistently observe high correctness values, indicating a clear correlation between high attention scores and the activities decisive for prediction. This observation holds for both real-world and synthetic event logs. Only in the cases of BPIC12_O and the synthetic Looped And and Sequence event logs do the values drop slightly. Since the correlation values, with exception of these logs, exceeds 0.5, we observe a moderate to strong correlation overall. By comparing AE and BE, we find that Page 33 of 42 Käppeletal. Process Science (2025) 2:11 the explainers generally behave similarly. However, in the cases of Sepsis, Helpdesk, and Sequence, BE achieves significantly higher values. A more varied pattern emerges for the completeness metric when calculated using the F1-score. Depending on the event log, either relatively poor values (BPIC12, Complex Model, LDD, Sequence, Sepsis) or quite good values (BPIC13) are achieved. Further analysis shows that low F1-completeness is primarily due to low recall values (see row Completeness (Recall) in Table3). Although the explanations are often correct (as indicated by significantly higher completeness precision), not all explanations are captured. In the resulting graphs (or rules), this means most of the edges are correct (high precision), but some edges are missing (low recall). However, there are notable differences between the two explainers. The AE explainer consistently outperforms the BE in precision completeness. A similar trend is seen for recall completeness, where AE performs nearly identical to BE for BPIC12, BPIC12_0, BPIC12_WC, and Complex Model but outperforms BE in BPIC12_W, BPIC13_CP, BPIC13_I, Sepsis, and Looped And. Only for Helpdesk, Sequence, and LDD are small differences (3 to 5%) observed in favor of BE. These findings suggest that AE’s explanations align more closely with the predictions of the model. These observations will also be visible in the qualitative evaluation in Qualitative evaluationsection. For the consistency metric, we observe extreme differences both between the event logs and between the explainers. While some event logs show nearly perfect consistency for each explainer (AE on BPIC13_CP and BE on Looped And), others exhibit very poor consistency close to zero (BPIC13_I for the BE and Looped And for AE). The extreme diametral behavior of BE and AE in the case of the Looped And is astonishing. An explanation for this behavior is that the attention scores of the two models selected for the calculation of the consistency metric are very different. Nevertheless, on some event logs the models come to the same prediction, since other parts of the architecture significant contribute to the prediction. However, the relevant activities ( Ar ) derived from the attention scores differ fundamentally, which affects the structure of the global graph and thus the performance of the explainers. Since it is not a general pattern, it is a strong indicator that there are dataset specific reasons for that. Aside from these outliers, both explainers typically show mediocre consistency scores around 0.5 across most event logs. For the continuity metric, the AE explainer nearly always achieves high values, typically above 0.85. Only for the synthetic Sequence and LDD event logs do we observe lower values (0.57 and 0.78, respectively). In contrast, the BE explainer performs worse, showing mostly lower values for BPIC12 (0.71), BPIC12_W (0.81), BPIC13_CP (0.49), BPIC13_I (0.19), Sepsis (0.57), Helpdesk (0.73), Complex Model (0.72), LDD (0.57), and Sequence (0.78). Overall, we can conclude that AE is a highly robust explainer, providing meaningful explanations even for previously unseen data. Although the BE cannot compete these values most of the time, it still achieves relatively high continuity scores around 0.7. When evaluating the counterpart, i.e., the contrastivity metric, we find consistently low values across all event logs, with both explainers performing similarly. The combination of high continuity values and relative low contrastivity scores suggests that the changes in the input have hardly any effect on the prediction. Page 34 of 42 Käppeletal. Process Science (2025) 2:11 Table 3 Results for Attention Exploration Explainer (AE) and Backward Explainer (BE); cells: mean (first), std. deviation (second); boldfaced = best values; NaN = no value could be computed (in case of std there was sometimes no variance due to identical results, in case of the correctness value it occurs if rule sets for a prefix and its modifications are identical) Metric Explainer BPIC12 BPIC12_O BPIC12_W BPIC12_WC BPIC13_CP BPIC13_I Sepsis Helpdesk Complex Model LDD Looped And Sequence Correctness AE 0.65 ± 0.15 0.48±0.22 0.65±0.21 0.56±0.18 0.60±0.28 0.57 ± 0.25 0.59±0.17 0.73±0.18 0.51±0.12 0.68±0.22 0.31±0.36 0.17±0.25 BE 0.63±0.20 0.48±0.22 0.65±0.21 0.62 ± 0.18 0.75±0.18 NaN±NaN 0.86 ± 0.13 0.91 ± 0.08 0.59±0.15 0.68±0.21 NaN±NaN 0.32 ± 0.18 Completeness (Precision) AE 0.18±0.08 0.25 ± 0.07 0.49 ± 0.14 0.68 ± 0.35 0.87 ± 0.14 0.82 ± 0.06 0.54 ± 0.13 0.50±0.15 0.30 ± 0.07 0.10±0.00 0.31 ± 0.01 0.11±Nan BE 0.20 ± 0.09 0.11±0.03 0.21±0.23 0.42±0.25 0.74±0.23 0.35±0.05 0.50±0.19 0.54 ± 0.15 0.19±0.03 0.30±0.00 0.00±0.00 0.32 ± NaN Completeness (Recall) AE 0.05 ± 0.00 0.20 ± 0.01 0.22 ± 0.03 0.23±0.06 0.79 ± 0.15 0.77 ± 0.08 0.11 ± 0.01 0.25±0.03 0.10±0.00 0.14±0.00 0.25 ± 0.01 0.11± NaN BE 0.04±0.01 0.18±0.03 0.09±0.05 0.25 ± 0.06 0.46±0.11 0.26±0.04 0.04±0.02 0.30 ± 0.04 0.11 ± 0.01 0.17 ± 0.00 0.05±0.02 0.14 ± NaN Completeness (F1-score) AE 0.07 ± 0.01 0.21 ± 0.02 0.27 ± 0.04 0.28±0.09 0.79 ± 0.15 0.77 ± 0.08 0.15 ± 0.02 0.30±0.04 0.12 ± 0.01 0.12±0.00 0.27 ± 0.01 0.10± NaN BE 0.06±0.01 0.13±0.02 0.10±0.08 0.29 ± 0.10 0.50±0.14 0.29±0.04 0.04±0.02 0.36 ± 0.05 0.10±0.01 0.18 ± 0.00 0.01±0.00 0.15 ± NaN Consistency AE 0.19±0.06 0.31±0.05 0.73 ± 0.08 0.54 ± 0.07 0.99 ± 0.09 0.18 ± 0.03 0.10±0.02 0.09±0.05 0.25 ± 0.06 0.27±0.15 0.01±0.01 0.42±0.21 BE 0.46 ± 0.07 0.65 ± 0.07 0.66±0.13 0.39±0.08 0.59±0.18 0.01±0.02 0.13 ± 0.03 0.11 ± 0.09 0.02±0.03 0.45 ± 0.22 0.99 ± 0.01 0.53 ± 0.20 Continuity AE 0.93 ± 0.05 0.95±0.07 0.92 ± 0.06 0.89±0.05 0.98 ± 0.10 0.99 ± 0.02 0.92 ± 0.13 0.85 ± 0.19 0.88 ± 0.21 0.57±0.31 0.93±0.24 0.78±0.25 BE 0.71±0.05 0.95±0.07 0.81±0.13 0.90 ± 0.06 0.49±0.13 0.19±0.24 0.57±0.09 0.73±0.15 0.72±0.19 0.57±0.32 0.93±0.24 0.78±0.25 Contrastivity AE 0.26±0.25 0.07±0.18 0.16±0.26 0.02±0.09 0.05±0.19 0.03±0.12 0.13±0.27 0.20 ± 0.30 0.30±0.15 0.42±0.31 0.40±0.21 0.32±0.34 BE 0.45 ± 0.33 0.07±0.18 0.19 ± 0.30 0.03 ± 0.13 0.05±0.20 0.04 ± 0.18 0.17 ± 0.33 0.08±0.22 0.58 ± 0.17 0.42±0.31 0.40±0.21 0.33 ± 0.30 Compactness (#Rules) AE 6.32 ± 1.11 4.61±0.55 4.05±0.78 3.19 ± 0.60 1.88 ± 0.34 1.99±0.09 4.55 ± 0.85 2.45 ± 0.79 4.19 ± 0.98 3.75 ± 1.91 2.00 ± 0.00 5.08 ± 2.02 BE 13.04±2.03 4.61±0.55 3.28 ± 0.57 3.97±0.89 2.16±0.53 1.38 ± 0.49 7.72±1.42 3.32±0.94 7.47±2.03 3.94±2.17 4.00±0.00 7.08±3.34 Compactness (Mean length) AE 4.12±0.77 2.21 ± 0.09 1.86±0.25 1.89±0.15 2.09±0.35 2.00±0.09 2.76±0.42 2.55±0.62 3.60±0.64 1.89 ± 1.13 4.00±0.00 2.06±0.40 BE 1.66 ± 0.20 2.56±0.06 0.90 ± 0.14 1.48 ± 0.09 0.60 ± 0.20 0.19 ± 0.24 0.83 ± 0.16 1.99 ± 0.39 2.01 ± 0.53 2.76±0.74 1.00 ± 0.00 1.96 ± 0.65 Page 35 of 42 Käppeletal. Process Science (2025) 2:11 This is essentially due to the fact that the transformer primarily learns the most frequent execution paths, as it is limited to the event activity attribute. As a result, it tends to provide “standard” predictions even in cases of significant modifications in the input prefix. This observation is also in line with prior experiments conducted in Käppel etal. (2021), which indicate that prediction models usually focus on frequent execution variants and treat exceptional cases as standard variants. With regard to the compactness metric, we observe that the BE explainer tends to generate more rules, although they usually remain within a range that does not hinder interpretability (fewer than five rules). Exceptions can be noted for BPIC12 (13.04), Sepsis (7.72), Complex Model (7.47), and Sequence (7.08), where BE produces more rules compared to AE. When examining the mean length of the right-hand sides of the rules, we observe that the AE explainer consistently produces longer rules. However, these lengths remain below 5, which still enables human interpretability. In contrast, the Backward Explainer explainer typically generates rules with mean lengths around 1. When combined with its performance in other metrics, particularly completeness and continuity, we see this as an indicator that BE’s rules are potentially too short to adequately capture the process behavior. In summary, the performance of both explainers is nearly identical when comparing real-world and synthetic event logs. However, notable differences exist between the two explainers. Based on the most relevant metrics for explainer quality – correct and completeness – we consider the AE explainer to be the superior choice. Qualitative evaluation To provide a clearer and intuitive understanding of the explainers’ performance and weaknesses, we discuss exemplarily their output for the synthetic Complex Model event log (cf. Fig.12). Because it is the most complicated process model we used for the generation of the synthetic event logs it offers the best insights into the effectiveness of the explainers. For creating the explanation graphs, we used all prefixes from the test log and start with a minimum prefix length of one. Including all prefixes is essential to capture the full process behavior, as this allows us to trace the step-by-step development of a process instance. For better comparability, the nodes in both explanation graphs were manually aligned similarly to the activities in the process model. At first glance, the BE explanation graph shows significantly more edges than the AE graph, reflecting the results of the compactness metric. Although there are significantly more edges in the graph of the BE, the completeness recall values of both explainers are nearly identical (0.10 vs. 0.11). However, the AE outperforms the BE with regard to completeness precision (0.30 vs. 0.19). Hence, this example confirms the finding of the quantitative evaluation that the AE finds fewer incorrect edges. Nevertheless, both graphs contain erroneous transitions. For example, the AE graph contains a reflexive edge at the node for activity A and an unwanted shortcut from A to J. The latter is also contained in the BE graph. Nonetheless, similarities to the process model are evident in both explanation graphs. The initial (A-B-C) and the subsequent AND gateway after activity D are clearly recognizable. While both explainers correctly identify that activities H, E, and P can follow D, the AE graph does not cover all possible follow-up sequences, whereas the BE allows Page 36 of 42 Käppeletal. Process Science (2025) 2:11 more flexible process executions. A similar pattern occurs at the second AND gateway (after activity B), where the AE graph identifies various options but ends in dead ends, while the BE graph identifies significantly more transitions. Concluding remarks Based on the results of the quantitative and qualitative evaluation, we can answer our research question as follows: the transformer’s attention scores are effective for checking and visualizing its process understanding. Although the transformer model produces some incorrect predictions – primarily because it relies solely on the activity event attribute for its predictions – certain patterns in the control flow, such as loops or XORgateways, cannot always be predicted correctly. Nevertheless, we were able to observe these patterns in the explanation graph. The reason for that is, that the transformer considers the different options related with these patterns, paying them a certain probability, although he cannot predict the correct one due to its limited information. In the rest of this section, we conclude the work with some theoretical and practical implications. Additionally, we discuss potential limitations of our approach. Theoretical and Practical Implications Our approach offers valuable opportunities for advancing transformer-based predictions of the next activity. In the following, we outline the primary implications of our work for both theory and practice in the field of process mining and especially for the subfield of next activity prediction. Fig. 12 Resulting explanation graphs for Complex Model Event Log. Top: The underlying real process model. Left: Attention Explorer generated graph. Right: Backward Explainer generated graph Page 37 of 42 Käppeletal. Process Science (2025) 2:11 Our work has three main theoretical implications. First, our study enhances the theoretical understanding of how transformer models capture and interpret process structures, highlighting the relationship between model internals (i.e., attention scores) and recorded process behavior in form of traces. From a more general point of view, it delivers insights into how transformer models encapsulate temporal and sequential dependencies in prediction tasks. Second, the global explanatory capabilities of the proposed explainers contribute to assessing the overall process understanding of the prediction model and ensure consistency in explanations. Unlike many local explainers, which may yield contradictory explanations across different process instances, our approach provides a holistic view of the process logic, fostering coherence across explanations. Third, our approach has the potential to lay the foundation for a new class of process discovery techniques, which extract process workflows based on predictive logic. As a consequence, a prediction model would become an active component in process analysis and (re)design. From a practical standpoint, our work has two main implications. First, our approach enhances transparency and trust in prediction models. By making the decision-making process of the model more transparent and interpretable, process participants gain a clearer understanding of the predictions and have more confidence in the model’s predictions. This is particularly important for processes, where the predictions directly impact business operations or strategic decisions as well as for processes in critical contexts, where a prediction may have drastic consequences (e.g., healthcare systems). Second, the insights gained through the explainers can serve as a promising starting point for model improvement and debugging. Especially in cases of erroneous predictions, the explainers provide useful insights that can guide the model improvement, either through retraining or by providing additional training data to address situations where the model currently struggles. Moreover, by analyzing the outputted graph structure, practitioners can identify potential weaknesses or areas for refinement. This also allows to some degree to anticipate potential future behavior of the prediction model, even for situations that were not observed until now. Limitations There are some limitations in our work. First, our experiments in both pre-study and evaluation are limited to eight real-world event logs and four synthetically generated ones. Although the real-world event logs stem from diverse domains and have heterogeneous characteristics, it is possible that the obtained results would vary with different event logs. Nonetheless, our findings provide encouraging evidence that our explanation approach can handle common data quality issues in real-world logs, such as noise or inconsistencies. In our explanation approaches, we aggregate the attention scores across all four attention heads, offering a holistic perspective on the entire attention mechanism but without distinguishing the individual contributions of individual heads. Thus, it remains unexplored what particular heads learn or how their interpretations might differ. However, as the objective of our study was to explore, whether attention scores as a whole give insights into what the prediction model has learned, we considered it appropriate to first focus on the aggregated attention mechanism. While aggregating may lead to losing fine-grained information, this choice primarily affects the detail level of the explanation rather than the overall usefulness of the explainers. Page 38 of 42 Käppeletal. Process Science (2025) 2:11 Another limitation is that our explanation approach relies solely on attention scores as model internals. Although these scores play a crucial role in the architecture, as verified in the pre-study, other components and layers of the transformer architecture also influence the prediction. Currently, the impact of these components is widely unexamined. However, this flaw is mitigated by the fact that the subsequent layers primarily process the attention scores further, so that they can be still considered as the central element in the model’s decision-making process. Our explainers employ various thresholds, such as those determining the prediction similarity or the relevance of an attention score. These thresholds affect the explanation quality because they control the addition or removal of edges in the explanation graph. For the attention score thresholds, this issue is somewhat alleviated, as the pre-study reveals that attention scores are typically either very high or very low, allowing setting a threshold with minimal information loss. However, configuring the prediction thresholds is more challenging, as the ideal threshold can vary depending on the particular process and its execution state. In our study, we chose a universal threshold expected to perform reasonably well across different processes. However, the threshold would be ideally adapted to the specific process characteristics. For example, a lower threshold is suitable for flexible processes with numerous decision points and alternative execution paths, while a higher threshold is appropriate for more sequential processes. Given that in practice, processes are often a mixture of flexible and largely sequential parts, a dynamic adjustment of thresholds depending on the current execution state would provide the best result. Conclusion andfuture work This paper investigates whether a trained transformer model for next activity prediction, based on the architecture proposed in Bukhsh etal. (2021), has gained an understanding of the control-flow of the underlying process. Given that attention scores are at the heart of the transformer architecture and a key factor for its effectiveness, we thoroughly investigated their reliability for explanation purposes. Our experiments provide strong evidence that attention scores offer valuable insights into the decision-making of the prediction model, effectively supporting the assessment and visualization of its process understanding. To this end, we developed two transformer-specific global explainable artificial intelligence approaches based entirely on attention scores, which create graph structures resembling process models. These graphs illustrate how attention scores for particular activities are linked to predictions. We evaluated both explainers using established quantitative metrics, revealing that attention score based explainers hold substantial potential. However, because the prediction models only incorporate control-flow information, the process cannot be fully learned. In future research, we plan to improve the explainers by analyzing the distinct relationships learned by individual attention heads more in detail. Moreover, we aim to replace the current masking and modification operations for determining the relevance of events with more sophisticated techniques tailored to the specific needs of process data. Additionally, we plan to extend our explanation approach to transformer architectures that incorporate additional features, such as resource or time information. Page 39 of 42 Käppeletal. Process Science (2025) 2:11 Appendix1: Hyperparameter configuration The following table list the hyperparameter configuration used for the training of the transformer models. Hyperparameter Value Epochs 10 Batch size 12 Learning rate 0.001 GPU usage True Number of heads 4 Embedding Dimension 36 Dimension of the feed forward network 64 Appendix2: Process models In the following the process models used for generating the synthetic event logs are depicted: Complex Model  Fig. 13 BPMN model for the Complex Model event log LDD  Fig. 14 BPMN model for the Long Distance Dependency event log Page 40 of 42 Käppeletal. Process Science (2025) 2:11 Looped And  Fig. 15 BPMN model for the Looped And event log Sequence Fig. 16 BPMN model for the Sequence event log Authors’ contributions All authors contributed to the conception and the writing of the prior BPM paper. For this extended version M.K., L.A., and S.J. wrote the main manuscript text. M.K., L.A., and S.H. prepared the visualizations. M.K. and S.H. contributed to implementation of the approach’s prototype. M.K., L.A., and S.H. conducted experiments (including those of the prior BPM paper). M.K., L.A, and S.J. reviewed the manuscript. Funding No funding was received. Data availability No datasets were generated or analysed during the current study. Materials availability Our code and detailed results can be found in the supplementary material at https:// github. com/ mkaep/ trans formerexpla inabi lity. Declarations Ethics approval and consent to paticipate Not applicable Competing interests The authors declare no competing interests. Received: 9 November 2024 Accepted: 12 May 2025 Published: 4 June 2025