From Visualization to Decision-Making: Enhacing Software Product Line Evolution with Visual Interventions

From Visualization to Decision-Making: Enhacing Software Product Line Evolution with Visual Interventions Raul Medeiros Pérez 2024 Supervisor: Oscar Díaz Ph.D. Thesis _______________ _________ From Visualization to Decision-Making: Enhancing Software Product Line Evolution with Visual Interventions Dissertation presented to the Department of Languages and Computer Systems of the University of the Basque Country in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy (“international” mention) Raul Medeiros Pérez Supervisor: Prof. Dr. Oscar Díaz San Sebastián, Spain, 2024 (cc) 2024 Raúl Medeiros Pérez (cc by 4.0) This work was hosted by the University of the Basque Country (Faculty of Informatics). The author enjoyed a doctoral grant from the Spanish Ministry of Science, Innovation - PRE2019-087324 from 2020 to 2024. This work was co-supported by MCIN/AEI/10.13039/501100011033 and the “European Union NextGeneration EU/PRTR” under contract PID2021-125438OB-I00, Spanish Ministry of Science, Innovation and Universities grant number RTI2018-099818B-I00 and the Ministry of Education with grant number MCIU-AEI TIN201790644-REDT (TASOVA). “Only a fool worries over what he can’t control.” — Patrick Rothfuss, The Wise Man’s Fear Acknowledgements “If you want to go fast, go alone. If you want to go far, go together.” — African proverb They say that a PhD consists of the journey and skills you shape throughout the years that you spend on it. Yes, indeed, it consists of those things, but, in my opinion, the most important aspect of a PhD is the people who accompany you on the roller coaster of predoctoral research. These paragraphs try to express my gratitude to all the people who sustained me during this tough journey. Although I would like to mention all of them, it would require writing another thesis, so I hope you can forgive me for not mentioning all of you. First, I owe my total gratitude to my supervisor, Professor Oscar Díaz, who always seeks the maximum from his PhD students and pushed me to excellence. His example of resilience and hard work is inspiring; it was an honor to work alongside him these years. Second, I would like to give special mention to the person who recommended me as a good candidate to pursue a PhD, Arantza Irastorza. I suppose the results of this thesis show you did a good job. I also would like to express my gratitude to my teammates, the "Onekindarras": Xabier Garmendia, Imanol Maraña, Maider Azanza, Juanan Pereira, Haritz Medina, Jeremias Pérez, Leticia Montalvillo, Juan Miguel Lopez, Felipe Ibañez, and Iker Azpeitia. For those laughs, moments in the lab, the relaxing coffee breaks, the interesting talks during lunchtime, etc. I have spent a lot of hours with some of you in these years, and I can call you my friends. Although I leave academia, I hope we will continue in contact, and, who knows, maybe also collaborating. I am in total debt with Danilo Beuche and Mustafa Al-Hajajji from puresystems GmbH (now PTC as they did an exit during the last months), for hosting me in their headquarters in Magdeburg for three months. They not only gave me a seat in their office to work on our collaboration, but also integrated me as one more of the team. Thank you to all the team: Benjamin, Michael, Steffi, Timo, Andre, Alex, Yang, etc. I do not want to end my thanks to the people in Magdeburg without mentioning Kostaq, an employee from pure-systems who became my closest friend in Magdeburg. Thank you for sharing those beers in Strandbar while we were talking about Albania, Spain, and many other things. I hope we can see each other soon. My deepest gratitude to Jacob Krüger and Mathieu Acher for acting as external reviewers. They did a great job pinpointing the strengths and weaknesses of this manuscript. Thanks to the Ministry of Science and Innovation of Spain for financially supporting this thesis. Lastly, my friends and family deserve a special place in these words. To my friends, who are known as "zulo kuadrila", thank you for understanding my absences and being there for me when I needed it. You helped me to disconnect when my mind was near to blow up, whether through a few drinks and partying or going out to do some sport in our beautiful mountains. I would also like to thank my cousins (those non-blood cousins from Madrid included), uncles, and aunties from Medeiros. You make all my trips to our village amazing and disconnecting. Thanks also to my brother, Diego, and his girlfriend, Gera, though we only see each other once each couple of weeks, your support has been really important. Rightly, my girlfriend, Nuria, deserves a special mention, for making me speak when I needed to but was not able to, for making me laugh in difficult moments, and for being there in the good moments, but also in the very tough ones. I could not have completed the PhD without you. To my mom, Marina, who raised me and has taken care of me since I can remember, and she continues to do it. Finally, I cannot conclude these acknowledgments without expressing gratitude to the person who shaped me into who I am today. Sadly, due to life’s cruel vicissitudes, he is no longer with us. My dad, Jose Antonio, who taught me the value of hard work, discipline, and more, but also the importance of being kind to others and taking care of those you love. I know that if you were here, you would be really proud of me. I love you. Summary Software product lines (SPLs) enable the systematic reuse of a family of software products to efficiently create diverse product instances tailored to customer needs. However, SPL evolution poses complex challenges due to the presence of distinct, interdependent asset types and variability. Despite the recognized importance of SPL evolution, tool support remains limited. Software visualization has proven useful for supporting program comprehension and evolution, but few visualizations specifically target SPL evolution activities and practices. This thesis aims to address this gap. To start, we conducted a systematic mapping study that reveals underexplored areas including incorporating user feedback into SPL evolution and employing longitudinal visualizations for version control systems data. This thesis puts forth solutions including: (1) CustomDIFF, an alluvial diagram-based tool that enables customization analysis during productbased SPL evolution; (2) PortfolioScan, a dashboard enabling portfolio evolution analysis to support product line scoping decisions; and (3) a process (IF4SPLE) and tool support (FEACKER) for gathering and leveraging implicit user feedback at the SPLs platform level. Together, these contributions demonstrate the utility of tailored visualizations for providing actionable insights, integrating user data, and enhancing decision-making. The solutions proposed were rigorously evaluated and validated through close collaboration with industry partners in real-world, large-scale SPL settings. Following the Action Design Research approach, the artifacts were iteratively built, refined, and evaluated based on feedback from practitioner teams. The empirical studies within industrial contexts provided concrete evidence of the solutions’ validity, practicality, and positive impact for supporting SPL evolution needs in the industry. This thesis provides both theoretical and practical advancement in augmenting SPL evolution through dedicated visualization tools. List of Tables 2.1 Data extraction classification schema. . . . . . . . . . . . . . . . 15 3.1 Validating the GQM. Rating the importance of information needs in customization analysis. Answers are ranged in a LIKERT scale from 1 (Not Important) to 5 (Very Important) . . . . . . . . . . 22 3.2 CustomDIFF’s perceived ease of use. The six practitioners (P1, P2...) were asked the extent of their agreement for each item along a LIKERT scale that ranges from 1 (strongly disagree) to 7(stronglyagree). .......................... 27 3.3 CustomDIFF’s perceived usefulness. LIKERT scale ranges from 1 (strongly disagree) to 7 (strongly agree). . . . . . . . . . . . . . 28 4.1 How could portfolio evolution analysis help? The case of testing resource optimization, configuration assistance, and SPL scoping. 36 Chapter 1 Introduction 1.1 Overview This chapter presents the context and motivation for the research work undertaken in this thesis. Section 1.2 provides the background and situates the research within a broader context. Section 1.3 introduces the specific problems addressed by the thesis. Section 1.4 outlines the research approach followed in conducting the dissertation study. Finally, Section 1.5 summarizes the contents of the remaining chapters in the manuscript. 1.2 Context This section sets the context for the research problems and solutions explored in this thesis by providing background on Software Product Line Engineering and the challenges involved in evolving Software Product Lines over time. 1.2.1 Software Product Line Engineering In recent decades, software has become an integral part of our daily lives. From household appliances to automobiles, many of our most frequently used devices are now controlled or enhanced by software. While these types of products are often similar to some extent, customers can usually customize their particular version to suit their own preferences (e.g., selecting a 360 degree parking camera in a car). This user customization is not only made at the hardware level but is also usually facilitated by plug-in/plug-out software features. In a situation with such diversity, it is more important than ever to systematically manage variability. Software Product Line Engineering (SPLE) comes to the rescue [Sch02, ABKS13]. The main goal of a Software Product Line (SPL) is to cover an entire market segment of a family of software products. SPLE systematizes software reuse to enable the efficient creation of diverse software products (also known as variants) with shared components 1 2CHAPTER 1. INTRODUCTION and features [CN02, PBvdL05]. SPLE allows organizations to achieve shorter time-to-market, improved software quality, and increased return on investment by managing variability and leveraging commonalities. Companies like Siemens, Danfoss, Hewlett Packard, and General Motors have adopted SPLE due to these proven benefits1[CN02, AMD21]. From a process perspective, SPLE is divided into two interrelated processes: Domain Engineering (DE) and Application Engineering (AE) [ABKS13, PBvdL05] (see Fig. 1.1). DE is pivotal in SPLE, where it involves the systematic identification and encapsulation of both common and variable elements within a given domain. This process includes developing the feature model, establishing the mechanisms for handling variability, and crafting the reusable assets that form the core assets of the SPL (also known as “developing for reuse”). On the other hand, AE is centered on the generation of specific software products through the selection and configuration of features. This process is critical in tailoring software products to meet the requirements of individual customers, mapping these requirements into the feature model, and consequently generating the appropriate product configurations (also known as “developing by reuse”). Figure 1.1: Traditional SPL process model: evolution is performed at platform level (DE) and then propagated into the different variants (AE) – (Figure taken from [ABKS13]). From an asset perspective, features and feature models are the main concepts. A feature is a unique functionality or characteristic that adds value to end-users or stakeholders (e.g., an adaptive cruise control system of a car) 1A list of successful industrial software product lines is available at https://splc.net/ fame.html 1.2. CONTEXT 3 [ABKS13]. Features constitute the building blocks that enhance the functionality and uniqueness of each product variant. The feature model acts as a blueprint, outlining possible configurations and interrelationships among features, defining the scope of the SPL portfolio [GBT+19]. Finally, features are realized through code, where different strategies for variability exist: annotationbased, component-based, feature-oriented programming, or aspect-oriented programming [ABKS13]. From an implementation perspective, variant management systems (e.g., pure::variants, GEARS) facilitate managing and controlling variability [Beu19, KC18]. These variant management systems aid in creating and visualizing feature models, defining constraints and relationships among features, and assisting in the configuration and derivation of distinct product variants. In summary, SPLE is a software development approach that emphasizes systematic reuse, efficient variability management, and the production of diverse yet related software products. The various techniques and practices demonstrate SPLE’s flexibility and adaptability in meeting market and technological requirements, facilitating the creation of high-quality, cost-effective, and tailored software products. 1.2.2 SPL Evolution Like any other software, SPLs evolve in response to market changes, technological advancements, or regulatory shifts [KMB20, MD23]. However, SPLs exhibit unique characteristics that make evolution more challenging. First, SPLs contain distinct asset types like feature models, configuration models, or family models that require coordinated evolution, unlike single-product development [MSRB19]. Second, the presence of variability further complicates the evolution of these assets [BCS10]. SPLs evolve not only over time with new platform versions, but also across space when new variant configurations are created [AGK+22, MOA+22]. This multiplicity strains evolution as both the platform and individual variants change [DMM+22]. Indeed, sometimes the SPL evolves at the AE level through product-based evolution, not via the platform [KMB20] and involves a wider range of activities [SMK+19] (see see Fig. 1.2), namely: •Variant Synchronization. When bug fixes or feature implementations are made in one variant, they need to be propagated to other variants. During variant synchronization, it is important to identify the variants affected by a bug fix or a feature upgrade. To this end, criteria such as the cost of the propagation or the importance of the variant can be used [MDF18, PTS+16]. •Variant Integration. In product-based evolving SPLs, application engineers are allowed to tune the codebase to meet the specifics of a variant without waiting for this specific to be introduced in the SPL platform. Eventually, these variant modifications need to be merged back into the SPL platform [KMB20]. 4CHAPTER 1. INTRODUCTION adoption evolution evolved variant planned or existing variant(s) integrated platform derived variant Variant Synchronization Variant Integration Analysis of nonfunction properties Co-evolution of Problem Space and Solution Space Figure 1.2: Kruger et al.’s SPL process model [KMB20] enriched with Strüber et al.’s activities [SMK+19]: evolution is performed at variant level. Evolution activities involve evolved variants (e.g., variant synchronization) and also the SPL platform (e.g., analysis of non-functional properties). •Constraint Extraction. Constraint extraction refers to the process of identifying and formalizing the structural or semantic relationships between features in a SPL. It is essential during SPL evolution to facilitate understanding of feature interdependencies, validate platform configurations, optimize combinatorial interaction testing, and document features comprehensively [TGAJ16]. •Analysis of Non-Functional Properties. SPLs are common among cyber–physical systems where non-functional requirements play a crucial role, such as performance in safety-critical systems or memory usage in resource-constrained embedded systems. In domains where safety is necessary, engineers often need to determine the most suitable variant for their requirements. This process may involve evaluating how different features 1.3. RESEARCH PROBLEM OVERVIEW 5 and their interactions affect quality attributes, optimizing configurations to meet specific quality standards, and weighing the trade-offs between various non-functional properties [MGPF21, PMG+18]. •Co-evolution of Problem Space and Solution Space. It refers to the synchronized evolution of both the requirements (problem space) and the implementations (solution space) within SPLs. It is essential during SPL evolution to ensure that changes in one area (such as changes in the feature model) are accurately reflected and supported in the other (such as the implementation of the features) [PTD+16]. Addressing the various aspects of SPL evolution poses a significant challenge. Despite its recognition as an important topic within the research community, there is a notable lack of tool support [MSRB19]. Thus, the main goal of this thesis is to aid in closing this gap through the implementation of dedicated visualizations. 1.2.3 Software Visualization Software visualization facilitates engineers’ comprehension of various software aspects through visually enhanced mechanisms, serving as a critical tool in the field of software engineering [Die07]. When the focus shifts towards tracking the software’s evolution, this practice becomes Software Evolution Visualization (SEV), a process that underscores the dynamic changes within software across its lifecycle [NTM+13]. The challenge of understanding software evolution lies in the vast amounts of data originating from different data sources, such as version control systems (VCSs), bug tracking systems, or mailing and project discussion lists, in addition to the software’s own implementation assets [MD08]. By integrating visual representations of software artifacts and evolution metadata with analytical reasoning techniques [KMS+08], SEV aims to transform the abundance of software repository data into intuitive and insightful visualizations [SAA16]. This allows users to leverage both human judgment and machine intelligence to investigate software evolution, identify important trends and patterns, and gain a deeper understanding of how and why software systems change [MCDP24]. The ultimate goal is to enable data-driven and informed decision-making around critical software maintenance and evolution tasks. In this way, SEV bridges the analytical power of data science with the perceptual capabilities of the human visual system. Next section presents how we embrace SEV to approach the problems tackled in this thesis. 1.3 Research Problem Overview SPL evolution is challenging. SPL managers and engineers face critical decisions [MRA+22]. They must navigate the challenging process of synchronizing and 6CHAPTER 1. INTRODUCTION integrating the SPL with its various variants [MDF18], all while considering user preferences that influence the evolution of features [PMK+18]. Despite the proven usefulness of visualization for helping software evolution [NTM+13, GTGPT13], visualizations for SPLs evolution are scarce. In their systematic mapping study on SPL visualization, Lopez-Herrejon et al. [LIE18] found only four research papers focusing on SPL maintenance or evolution. We broaden this review four years later (see Chapter 2). Out of this endeavor, different gaps were identified. This thesis tackles three of them. 1.3.1 Research Challenge 1: Lack of tool support for conducting customization analysis Practice. Product-based SPL evolution is a practice where new features are initially introduced and developed within individual product instances to be then reintegrated into the core assets. Consequently, this approach leads to product variants becoming customized beyond mere configuration of the platform’s built-in variability mechanisms and leads to a huge landscape of customizations that must be reintegrated into the platform or discarded. A crucial activity in product-based SPL evolution is customization analysis. Customization analysis aims to quantify and analyze the adjustments made to product instances, enabling informed decision-making. As individual products are customized beyond basic configuration, it becomes challenging to evaluate the impacts of these customizations. Problem. Conducting customization analysis appropriately is crucial in SPLE. However, it becomes impractical when dealing with SPLs comprising hundreds of variants. Comparing each of the modified variants with the platform one-by-one to analyze the customization efforts would be time-consuming and error-prone. Currently, the lack of dedicated tool support for customization analysis severely limits SPL managers’ ability to accurately quantify these customization efforts across the entire SPL and estimate what will be their impact. Relevance. More effective tools for customization analysis would provide vital insights to guide variant integration during product-based SPL evolution. Customization analysis plays a pivotal role in assessing the impact of product customizations and determining which modifications should be incorporated into the core assets. Without proper tool support to understand the extent and nature of these customizations, SPL engineers will not be able to make wellinformed decisions on whether to integrate specific customizations into the core assets or allocate more resources for managing customizations, and ultimately achieve stability and consistency across the SPL. Research Questions. This thesis aims to contribute to customization analysis for product-based evolving SPLs by providing an analysis workflow and tool support. In doing so, we tackle the following research questions (see Chapter 3): •Which are the information needs for customization analysis? How much time is needed to fulfill these information needs? 1.3. RESEARCH PROBLEM OVERVIEW 7 •Which sort of visualization better fits customization analysis? 1.3.2 Research Challenge 2: Lack of tool support for conducting portfolio evolution analysis Practice. Portfolio evolution refers to the changes that occur over time in the set of product variants actively maintained and commercialized within an SPL. As market conditions evolve over time, such as changes in customer preferences, technological advancements, or new regulatory requirements, feature selections and consequently, the existing product configurations need to be adapted to align with these evolving conditions. For instance, if a customer demands a new capability or functionality, features related to that capability may need to be selected and integrated into the product configuration of the customer. Analyzing the evolution of the product portfolio (also known as portfolio evolution analysis) helps SPL managers gain insights into customer preferences by understanding trends and patterns in feature adoption. Problem. Despite the importance of studying the history of the portfolio, current variant management systems face limitations in supporting this kind of analysis. They often fail to bring together data from different configurations and versions, identify long-term trends, or provide a high-level overview necessary for analyzing extensive portfolio histories. For instance, variant management systems tend to allow only two-version comparison, making it challenging to consolidate and analyze data across multiple versions or releases of the product portfolio. Relevance. Effective support for portfolio evolution analysis would enable SPL managers to understand customer preferences, identify trends in feature usage, and plan future SPL evolution strategies by studying how the portfolio evolves over time. However, the lack of dedicated tool support, highlighted in the previous paragraph, directly impedes the ability to conduct a comprehensive analysis. Without tools to consolidate data across configurations and versions it becomes extremely challenging to gain valuable insights into changing customer preferences, and emerging trends in feature adoption. Consequently, in order to make informed decisions about future SPL evolution, such as prioritizing features for the next release, tools supporting portfolio evolution analysis are needed. Research Questions. This thesis aims to contribute to portfolio evolution analysis by providing a decision-making visualization dashboard. To this end, we tackle the following research questions (see Chapter 4): •What SPL activities can benefit from portfolio evolution analysis? •With a focus on SPL Scoping, how can portfolio evolution analysis be introduced in variant management systems? Which sort of visualization might be more suitable? 8CHAPTER 1. INTRODUCTION 1.3.3 Research Challenge 3: Lack of process and tool support for implicit feedback gathering and analysis Practice. Implicit feedback refers to the practice of tracking and analyzing how users actually interact with and use a software system, without relying on direct user input. By tracking and analyzing user interactions, such as feature usage patterns, navigation paths, and error occurrences, implicit feedback reveals how users actually engage with the software. This data offers an insightful perspective on user needs and areas for improvement. Problem. Traditionally, implicit feedback approaches involve injecting feedback tracking code directly into individual products. This approach poses significant challenges in the context of SPLs. Firstly, in SPLE, there can be delays in receiving feedback across the SPL, as domain engineers may not have direct access to the feedback collected from individual product variants. Instead, they must rely on feedback being relayed from application engineers, which can introduce latency in development decisions. Secondly, it risks overlooking implications and insights that span multiple products or the entire platform. Thirdly, when reintegrating customized variants into the core assets, merge conflicts may arise due to the injected feedback tracking code. Thus, in an SPLE setting it feels natural to move implicit feedback to the platform level. This would enable leveraging feedback data for evolution as the feedback would be directly managed and analyzed by SPL managers at the platform level. However, variant management systems offer limited capabilities for feedback gathering and analysis which makes it impractical for domain engineers to exploit implicit feedback. Relevance. The mismatch between traditional feedback practices and SPLE principles underscores the importance of incorporating implicit feedback mechanisms into the core SPL development process. A key tenet of SPLE is minimizing coding efforts in application engineering and focusing on domain engineering. This principle should extend to feedback practices, incorporating them into domain engineering and shifting from product variants to the SPL platform. This approach ensures feedback represents the entire variant portfolio, providing comprehensive insights into user interactions across products and features. For example, it acknowledges SPL managers with features or products that are performing as expected, but also highlights those where users are struggling. Research Questions. This thesis aims to contribute to implicit feedback gathering and analysis by providing a process and tool support for incorporating implicit feedback into SPLE. To this end, we tackle the following research questions (see Chapter 5): •How can platform-based feedback be incorporated into SPLE practices? •How can variant management systems support platform-based feedback? •What benefits can be derived from incorporating platform-based implicit feedback into SPLs? 1.4. RESEARCH METHODOLOGY: ACTION DESIGN RESEARCH 9 The three research challenges addressed in this thesis collectively contribute to a holistic approach for managing SPL evolution. Customization analysis informs the crucial process of reintegrating product variants into the core assets, while portfolio evolution analysis provides a broader perspective on feature adoption trends across the entire product portfolio. Critically, the incorporation of implicit feedback gathering and analysis acts as a linchpin, offering valuable user interaction data that can refine both customization analysis and portfolio evolution analysis. By tackling these challenges in a cohesive manner, this thesis tries to establish a comprehensive foundation for data-driven SPL evolution management, empowering SPL managers to navigate complex decisions surrounding variant reintegration, feature prioritization, and alignment with evolving user needs and market dynamics. 1.4 Research Methodology: Action Design Research Throughout the development of this PhD thesis, we followed the Action Design Research (ADR) approach [SHP+11]. ADR is the crosscut between Design Science Research [Hev07] and Action Research [CH07]. ADR emerges as an evolution of these two research approaches with the aim of integrating the building, intervention and evaluation of the artifacts into the real-contexts. ADR is built on the notion that artifacts are ensembles shaped by the organizational context throughout their development and use. This concept leads to a research process that blends the activities of building the artifact, intervening in the organization, and evaluating it in a concurrent and interlaced manner. Therefore, ADR conceives artifact design as a result of a researcher-practitioner collaboration within an organization. Fig. 1.3 reproduces ADR stages and principles [SHP+11]. We chose to follow the ADR methodology due to the industry-academia collaboration in this thesis. The artifacts designed to address the three research challenges were constructed and evaluated, to some extent, in conjunction with one of our industrial partners. In such a setting, where research is driven by real-world problems and involves close collaboration between researchers and practitioners, ADR emerged as the natural and appropriate research methodology to adopt. Problem Formulation This stage entails identifying the research problem, which is typically based on practical experiences or anticipated by researchers. The problem should be characterized as an instance of a broader category of problems. This approach is essential as it lays the foundation for generalizing the contributions of the project to similar problems in the field. It also serves as the conductor to define the initial scope of the project and to define roles. This is key to ensure practitioners long-term collaboration within the project beyond this problem for- 16 CHAPTER 2. VISUALIZATIONS FOR THE EVOLUTION OF SPLS ing into account the impact that overlooking non-functional requirements could have in the security or efficiency of the VRS [MNM+18] and the need of defining a proper architecture to transit from clone & own to an SPL [SBH18, FLLHE14] we believe that more research on visualizing these scenarios should be conducted. 2.4.2 RQ2: What sort of visualization are displayed? We characterized visualizations in terms of: visualized elements, visualization techniques, interactions, strategies and data sources: •Visualized elements: features, variants, and implementation assets (e.g., source code classes) are the most popular. These elements seem to satisfy the analysis needs of the most investigated scenarios. •Visualization techniques: among the 11 different techniques identified in the primary sources, the nodelink diagram and the tabular/matrix techniques are the most popular visualization techniques. This coincides with visualizations for single-product evolution [NTM+13]. •Visual Interactions: nearly half (44%) of the primary sources provided only static diagrams without interaction, while those supporting interaction most commonly used abstract/elaborate interactions to bridge the different abstraction levels from variants to features to code. Filter and select interactions were also common given the large codebases, but connect and explore were less prevalent. •Visualization Strategies: The vast majority of sources used a temporal snapshot strategy with some tentative exploration of temporal overview and differential relative strategies, exceptionally some works also supported multiple strategies based on their user needs. •Data sources: The most common data sources were source code and feature models, with traceability links and models also receiving notable attention. Three main concerns were raised from these results: research gaps in both visualization techniques and data sources, and a need for scalability awareness. First, the analyzed studies relied solely on 2D visualizations and basic filters (e.g., selecting or filtering). However, the software visualization field encompasses far more than 2D techniques. Recent works have utilized dynamic/live visualizations, augmented reality, and virtual reality [CMZ+20]. Researchers should explore how these advanced techniques could benefit VRS evolution. Second, data sources like user feedback, decision knowledge, and VCS information have been extensively leveraged in other software engineering areas [FS17, JKPB19] but deserve greater attention for VRS evolution. Finally, scalability challenges remain in visualizing large codebases, bridging problem 2.5. CONCLUSIONS & FUTURE WORK 17 and solution spaces, and analyzing extensive feature sets in variant-rich systems. Further investigation is needed into advanced interaction techniques, unified family-variant perspectives, and scaling visualization approaches to address these issues. 2.4.3 RQ3: What is the research maturity of the reported interventions? We used four proxies to assess research maturity: •Research Type: The majority of primary sources were solution proposals presenting visualization tools through proofs of concept, while nearly half conducted some form of validation or evaluation research beyond just proposing solutions. •Evaluation Method: The most common evaluation method was field experiments, with only four studies conducting rigorous field studies. •Availability of the tool: The availability of 13 tools for public access, either through publications or online searches, demonstrates confidence in their completeness and enables replicability, reproducibility, and knowledge accumulation, though wider availability remains limited. •Publication Venue: The majority of sources (64%) were published in conferences, especially in SPLC, rather than journals, likely due to the difficulty of providing rigorous empirical evidence, as over half of proposed tools lacked strong evaluation regarding realism of the VRS, analysis tasks, or experiment subjects. The fact that most tools are only partially validated, if validated at all, shows a worrying situation. This situation is shared with visualization interventions for single-product development. In their literature review about software visualization [MGAN18], Merino et al. found that 62% of the primary sources lack a strong evaluation. In the same vein, our review indicates that only 21% of the visualizations are validated in realistic industrial settings. In this respect, research methodologies such as Action Design Research might help involve practitioners [SHP+11]. 2.5 Conclusions & Future Work In this chapter, we presented a summary of a systematic mapping study about the current status of visualizations for the evolution of variant-rich systems. The complete review can be found in Appendix A.1. The work analyzed 41 visualizations to answer three research questions: •What sort of analysis is being conducted to assess VRS evolution? 18 CHAPTER 2. VISUALIZATIONS FOR THE EVOLUTION OF SPLS •What sort of visualization are displayed? •What is the research maturity of the reported interventions? Analysis wise, we found that researchers tend to overlook project managers’ role and three evolution scenarios: Feature Model Synthesis, Analysis of NonFunctional Properties, and Architecture Recovery. Visualization wise, we analyzed visual elements, visualization techniques, interactions, strategies and data sources. We found 11 different visualization techniques where the nodelink diagram was the most popular. Unfortunately, scalability seems to be an aspect with high potential for improvement as only 56% of the primary sources implement interaction mechanisms. When it comes to strategies and data sources, temporal snapshot and source code are the most used approaches, respectively. Overall, we miss studies applying visualization techniques with increasing popularity (e.g. 3D or Virtual Reality) and data sources that are used in other software engineering areas (e.g. user feedback, decision knowledge or VCS historical information). Finally, benchmarks might also help provide common ground to assess and compare visualization interventions. Maturity-wise, we found that, similar to single-product development, most visualization tools for VRS are only partially validated. Following methodologies such as Action Design Research could help improve this situation. All in all, visualizations for variant-rich systems look promising, yet in an early stage of development. As part of future work, we plan to explore some of the research opportunities highlighted in the results. In particular, this thesis examines the visualization of two overlooked data sources relevant to VRS evolution research: VCS data in Chapter 4 and user feedback in Chapter 5. Chapter 3 Visualizing Customization Efforts with Alluvial Diagrams 3.1 Overview This chapter summarizes the results of using alluvial diagrams to support customization analysis in product-based evolving SPLs, as detailed in our publication in the Empirical Software Engineering journal (see Appendix A.2) [DMM+22]. 3.2 Practice: product-based evolution Product-based evolution, also known as variant-based evolution, refers to the practice of driving the evolution of a SPL by first developing new features in individual products rather than in the core platform [FSK+16, KMB20]. This results in products being customized beyond simple configuration of the platform’s variability. Although, theoretically this would not be the ideal approach to evolve the SPL, teams might opt for customization for several reasons: •Time-to-Market Pressure: Developing a feature directly within a product can be quicker than adapting it for the entire platform, ensuring timely product releases. •Expedited Bug Fixes: Customizing the code of a specific product allows for faster resolution of bugs compared to waiting for updates to the platform. •Product Specifics: When a feature is required by only one product, customizing that particular product is more efficient than extending the functionality across the entire SPL. 19 20 CHAPTER 3. VISUALIZING CUSTOMIZATION EFFORTS However, product-based evolution poses challenges in balancing the demand for high-quality, reusable domain engineering assets with the urgency and specific needs of application engineering [FSK+16]. SPL managers must evaluate the benefits of product customization against its impact on the SPL scope. Key considerations include the effort spent on customization, deciding which customizations should be reintegrated into the platform, which core assets require frequent modification, and the degree of core asset code reuse across products. 3.3 Problem: lack of assistance in assessing customization effort As the variants of the SPL evolve, SPL managers must assess the implications of those changes, determining whether they could jeopardize the stability of the SPL or identifying which variant customizations are suitable candidates for integration into the core assets. This assessment is known as customization analysis, which is a critical activity in product-based evolving SPLs. Conducting customization analysis becomes increasingly challenging as the number of product variants grows [FSK+16]. In large-scale SPLs consisting of hundreds of variants, manually assessing the customizations made to each variant is impractical. The current lack of dedicated tool support for customization analysis hinders SPL managers from accurately quantifying and understanding the extent of customizations across the entire SPL. Without appropriate tooling, gaining a comprehensive understanding of the customization landscape and its implications is an extremely complex and time-consuming endeavor. Thus, in this part of the thesis, we aimed to increase the existing support for customization analysis in product-based evolving SPLs. 3.4 Relevance of the problem The tension between maintaining robust, reusable core assets versus accommodating urgent, specific product needs makes SPL evolution a complex balancing act. Customization analysis aims to understand the modifications made to products since the last SPL platform release. Customization analysis offers several advantages for SPL managers, including the ability to: •Identify product customizations that could be integrated into the core assets. •Detect product teams burdened with excessive customization work. •Detect uncertain or experimental features indicated by frequent code changes. •Highlight potential integration challenges when merging product developments with core assets. 3.5. RESEARCH DESIGN 21 By quantifying the extent of customization, managers can make well-informed decisions to enhance SPL stability and optimize resource allocation. Despite the importance of these decisions, support for customization analysis is currently limited or even nonexistent. 3.5 Research Design We abound into the problem of customization analysis by making a case for the use of alluvial diagrams as a suitable visualization for customization analysis. Specifically, we tackled two research questions: •RQ1 (Problem Space) Which are the information needs for customization analysis? How much time is needed to fulfill these information needs? •RQ2 (Solution Space) Which sort of visualization better fits customization analysis? We addressed the research questions through a collaborative effort between academia and industry, specifically with Danfoss Drives, a company division responsible for P400, an SPL of frequency converters [FSK+16]. First, we characterized customization analysis through a Goal-Question-Metric (GQM) model. In Danfoss Drives, decisions are made by the Change Control Board. Thus, based on the knowledge of the Change Control Board, we introduced a set of analysis questions, and rated the importance and required time to answer such questions. Regarding RQ2 we proposed alluvial diagrams as an effective visualization approach to realize the previously outlined GQM model. We implemented this proposition through CustomDIFF. To refine and validate this approach, we distilled general principles from initial empirical evidence gathered through an expert evaluation at Danfoss Drives. 3.6 Results 3.6.1 RQ1. Which are the information needs for customization analysis? How much time is needed to fulfill these information needs? We resorted to the Goal-Question-Approach to distill and validate common information needs during customization analysis [BR88]. This effort resulted in the following GQM model: Goal. Evaluate the effort trend (issue) in product customization (object) from the Change Control Board perspective (viewpoint). Our goal was to trace how much effort is being put into product customization. Question. We structured the questions around the concepts of feature and product, as SPL releases, product configurations, programming tasks, and work roles are commonly described in these terms. Using the notions of feature and 22 CHAPTER 3. VISUALIZING CUSTOMIZATION EFFORTS ID Likert scale Avg. Holistic perspective: I consider important to know ... 12345 H1 ... which features are (not) being customized by products 0 1 3 3 1 3.5 H2 ... which products are customizing (no) features 0 1 2 3 2 3.75 H3 ... how much effort (i.e., code churn) has been spent on customizing each feature, in total, no matter the product 2 1 4 0 1 2.87 H4 ... how much effort (i.e., code churn) each product is spending on customizing each feature 2 2 2 1 1 2.62 H5 ...which files are being changed on each product to customize each feature 202403 Likert scale Avg. Feature perspective: For feature F1, I consider important to know ... 12345 F1 ... which products are customizing it 0 0 1 3 3 4 F2 ... which has been the total customization diffing (i.e., code-churn aggregate) 2 1 3 1 1 2.75 F3 ... which has been the total customization diffing (i.e., code-churn aggregate), broken down by product 2 1 3 1 1 2.75 F4 ... which files have been changed 1 2 1 1 3 3.37 Likert scale Avg. Product perspective: For product P1, I consider important to know ... 12345 P1 ... which features are customized 0 1 1 2 4 4.12 P2 ... how much effort (i.e. code churn) has been spent on customization, no matter the feature 2 1 2 2 1 2.87 P3 ... how much effort (i.e. code churn) has been spent on customizing each feature 2 1 3 1 1 2.75 P4 ... which files have been changed 2 1 3 1 1 2.75 Table 3.1: Validating the GQM. Rating the importance of information needs in customization analysis. Answers are ranged in a LIKERT scale from 1 (Not Important) to 5 (Very Important) . product, we organize the questions into three distinct scopes (see Table 3.1 for the full list): •the feature-focused scope, e.g., for a given feature, which products are 3.6. RESULTS 23 HOLISTIC PERSPECTIVE (ID-H2) Time needed to know: which products are customizing (no) features. FEATURE PERSPECTIVE (ID-F1) Time needed to know: for the feature F1, which products are customizing it. PRODUCT PERSPECTIVE (ID-F2) Time needed to know: for the feature F1, which has been the total customization diffing. (ID-H1) Time needed to know: which features are (not) being customized by products. (ID-F3) Time needed to know: which has been the total customization diffing on each product to customize feature F1. (ID-P2) Time needed to know: for the product P1, how much effort has been spent on customization. (ID-H5) Time needed to know: which files are being changed on each product to customize each feature. (ID-P3) Time needed to know: for the product P1, how much effort has been spent on customizing each feature. 1 day 25% 1 day 11% 1 day 25% % 1 day 25% 1 day 13% 1 day 50% 1 day 25% 1 day 45% 1 day 38% < 1 hour 37% < 1 hour 45% < 1 hour 25% < 1 hour 50% < 1 hour 50% < 1 hour 12% < 1 hour 12% < 1 hour 22% < 1 hour 12% Few hours 38% Few hours 44% Few hours 50% Few hours 25% Few hours 37% Few hours 38% Few hours 63% Few hours 33% Few hours 50% (ID-P1) Time needed to know: for the product P1, which feature are customized. Figure 3.1: Time effort in conducting distinct customization analysis questions. IDs help link to Table 3.1. customizing it, •the product-focused scope, e.g., for a given product, which features have been customized, •the holistic scope, e.g., which is the whole customization commitments from the last platform release. 24 CHAPTER 3. VISUALIZING CUSTOMIZATION EFFORTS GQM Figure 3.2: Visual analytics main interactions (adapted from [RRB+14]). Measurement. Once the questions were developed, we proceed by associating the questions with appropriate metrics. Factors to be considered for this selection include [BR88]: the quantity and quality of the existing data, and the maturity of the measurements. On these grounds, we resorted to code churn, a well-established metric to assess evolution in single-product development [AD07, HM00]. In the realm of SPLs, code churn metrics could be tailored to features and products, not just files. While file-level diffing provides code churn between master and product branches, customization analysis requires utilities like DIFF(aFeature, aProduct) that aggregate file changes to the feature and product level. This higher abstraction aligns better with customization analysis questions compared to low-level file diffing. We validated the GQM model through a questionnaire completed by eight Danfoss’ engineers with 1-10 years of experience. Participants rated the importance of each question on a 1-5 Likert scale. The results, shown in Table 3.1, indicated that feature-focused and product-focused perspectives were rated similarly high, with overviews and specifics being prioritized over intermediate aggregations. All questions scored above moderately important, with four questions receiving high ratings. This provided evidence of strong interest in customization analysis. In addition to validating the importance of the GQM, we measured the effort required to answer the questions using traditional code churn utilities. To this end, we conducted a survey among the same Danfoss’ engineers. Figure 3.1 depicts the results, which suggest that some questions might require several hours to be answered. This is particularly true for questions related to the holistic perspective and the product perspective (see Figure 3.1), where most questions were estimated to take a few hours to answer. 3.6. RESULTS 25 Figure 3.3: Customization efforts are depicted as flows from products, acting as sources of customization, to features, which receive these customizations. Additionally, package bars located at the diagram’s extremes identify the specific core assets that have been modified. For instance, ProductLondon-v1.0 has undergone customization affecting three features: WindSpeed, AirPressure, and Temperature. Specifically, the Temperature feature experienced modifications within the “scale.js” file. . 3.6.2 RQ2. Which sort of visualization better fits customization analysis? In the previous section, we presented a GQM delineating the information needs of customization analysis. However, visualization interventions require not just identifying the right data but enabling understanding and communication to decision-makers. This motivated the design, implementation and evaluation of CustomDIFF, a visualization tool built around the concept of alluvial diagrams. Fig. 3.2 depicts the main ingredients of CustomDIFF along the Visual Analytics framework [RRB+14]. For our purposes, the Data is kept in a Git repository. This data corresponds to code files whose versions are arranged w.r.t the Git version control system model. This code is pre-processed to distill the metrics to be used to assess the customization effort (i.e., code churn). As a result, code churn is obtained and described along with a Model that serves the information needs which were identified in the GQM analysis. This model ends up being realized through a database that acts as the back-end for a web application that supports the alluvial diagram visualization. 32 CHAPTER 4. PORTFOLIO EVOLUTION ANALYSIS 4.3 Problem: lack of support for assessing portfolio evolution Understanding portfolio evolution is essential for decision-making in any software industry [ZSG15, NGL20]. What distinguishes SPLs is their scale, as they commonly have a larger number of assets, more scattered and tangled dependencies, longer lifecycles, and more involved stakeholders [MD23]. These dynamics are reflected in the actual product portfolio, which evolves in two main modes: •Incrementally, where new variants are introduced and outdated ones retired, enlarging the portfolio [MSRB19]. •Iteratively, where existing variants are modified and released in improved iterations [KMB20]. Despite the importance of the actual portfolio for assessing an SPL’s fitness, this perspective has received limited attention in research. Some exceptions analyze feature selection correlations in actual variants to improve feature model documentation through new constraints [ZB14] and simplify future configurations via recommender systems [PMK+18]. On the commercial front, pure::variants offers the so-called “matrix view” or “the two-versions configuration comparator” [Beu19]. The two-versions comparator allows users to easily identify the differences between these two versions and understand how they affect the overall configuration. The matrix view displays a set of configurations in the same portfolio version and allows to compare them in a single view. This helps users analyze the variations and similarities among different configurations within the portfolio. However, there are still some challenges that need to be addressed: •Aggregation: current tools lack mechanisms to aggregate and analyze multiple configurations and portfolio versions as a whole; •Longitudinality: current tools do not facilitate the analysis of longitudinal trends across the portfolio history, making it difficult to identify historical patterns; •Abstraction: current tools lack the necessary level of abstraction to effectively analyze feature selections at the portfolio level. This is particularly significant in the presence of large portfolios. On these bases, we can state that portfolio analysis tools face limitations in effectively supporting aggregation, longitudinal and abstraction mechanisms for portfolio evolution analysis. Thus, we can then safely conclude that current variant management systems lack support for conducting portfolio evolution analysis. 4.4. RELEVANCE 33 4.4 Relevance While current research has predominantly focused on optimizing the potential portfolio view [dAdBdALR+16, GFH+12, Mül11], the actual portfolio of actively maintained and commercialized variants may better reflect the SPL’s fitness and market relevance at a given time. Even a highly flexible SPL can gradually become stagnant if its features fail to keep pace with evolving market demands. Consequently, studying the extent to which features are utilized in current variants and analyzing the evolution of the actual portfolio is more indicative of agility than examining the potential portfolio alone. Examining how the actual portfolio evolves provides valuable insights for SPL management. First, tracking feature usage longitudinally (i.e., the extent to which variants are selecting a feature) enables forecasting of future trends grounded in historical data [RFF+08]. Second, analyzing feature usage can uncover seasonality patterns, whereby predictable recurring variations within variants reflect regular market cycles. Third, anomaly detection facilitates the identification of atypical shifts in feature adoption that may indicate pivotal changes in customer needs or the competitive environment. This analysis guides planning and equips SPL managers to respond to evolving market dynamics. Tracking the evolution of the actual portfolio thus offers an insightful window into historical customer and market behavior. 4.5 Research Design We faced the problem by adopting an ADR approach [SHP+11] in collaboration with pure-systems, an experienced SPLE consultancy company and the one behind pure::variants, one of the leading variant management systems. The main goal of this research project was to enhance existing tool support for conducting portfolio evolution analysis. We addressed this by investigating the following research questions: •RQ1 (relevance): What SPL activities can benefit from portfolio evolution analysis? •RQ2 (feasibility): With a focus on SPL Scoping, how can portfolio evolution analysis be introduced in variant management systems? Which sort of visualization can be more suitable? 4.6 Results This section delineates the obtained results for the two posed research questions. 34 CHAPTER 4. PORTFOLIO EVOLUTION ANALYSIS 4.6.1 RQ1: What SPL activities can benefit from portfolio evolution analysis? To answer the first question, we examined two aspects of portfolio evolution analysis: (1) current portfolio evolution practices at pure-systems, and (2) its relevance for SPL activities. We surveyed practitioners to capture existing portfolio evolution practices. We also demonstrated the potential value of this analysis for activities like testing optimization, configuration assistance, and SPL scoping by outlining GQM models. These were validated through questionnaires with pure::variants team members. 65% 15% 10% 10% HOW LARGE IS THE PORTFOLIO? Less than 100 VDMs Between 100 and 1,000 Between 1,000 and 10,000 More than 10,000 17% 21% 24% 17% 21% HOW OFTEN IS THE PORTFOLIO UPDATED? Weekly Monthly Quarterly Every six months Yearly 5% 17% 33% 28% 17% WHAT PERCENTAGE OF THE PORTFOLIO CHANGES DUE TO INCORPORATION OF NEW VDMS? Less than 1% of the VDMs Around 3% of the VDMs Around 5% of the VDMs Around 10% of the VDMs Above 15% of the VDMs 42% 26% 21% 11% WHAT PERCENTAGE OF THE PORTFOLIO CHANGES DUE TO MODIFICATIONS TO EXISTING VDMS? Less than 1% of the VDMs Around 3% of the VDMs Around 5% of the VDMs Around 10% of the VDMs Above 15% of the VDMs Figure 4.1: Portfolio maintenance practices. Regarding portfolio evolution practices, we designed a questionnaire to capture portfolio evolution approaches among the pure::variants team. We exam- 4.6. RESULTS 35 ined three dimensions (see Fig. 4.1): •Size. While most portfolios contained fewer than 100 variants, some organizations managed portfolios exceeding 10,000 variants. This indicates a wide range in portfolio scale across organizations. •Frequency. Quarterly portfolio updates were most common, but practices ranged from weekly to yearly reevaluations. This points to substantial variation in how often portfolios are revisited. •Update Types2. We differentiated incremental updates (adding new variants) from iterative updates (modifying existing variants). Incremental changes accounted for 5-10% of portfolio updates in over half of cases. Iterative updates impacted 1-3% of the portfolio in over two-thirds of cases. Overall, a typical update impacted 6-13% of the portfolio. In summary, our analysis characterized portfolio evolution as generally a quarterly event impacting 6-13% of variant compositions. But substantial diversity exists across organizations in the scale, frequency, and types of portfolio changes. These findings reveal portfolio evolution as a continuous process adapted to each organization’s needs. Regarding relevance, we argue that analysis of portfolio evolution is significant for the information needs of three SPL activities: configuration assistance, testing optimization, and software product line scoping. We characterized the information needs of the three SPL activities that could be satisfied by conducting portfolio evolution analysis in a GQM (see Table 4.1) and we validated it through a 5-point Likert questionnaire to assess the extent to which the aforementioned activities resonate with participants, from 1 (Not At All Important) to 5 (Extremely Important). The results in Fig. 4.2 suggest that portfolio evolution analysis is perceived as relevant for the three activities: configuration, testing, and scoping. SPL scoping received the larger resonance, with 10 respondents selecting “Very Important” or “Extremely Important”. Consequently, we focused the intervention for RQ2 on SPL scoping as a case for portfolio evolution analysis. 4.6.2 RQ2: With a focus on SPL Scoping, how can portfolio evolution analysis be introduced in variant management systems? Which sort of visualization can be more suitable? Tackling the second question naturally seeks for an artifact intervention. Our solution was driven by Simon’s theory on decision-making visualizations which is divided into three stages [Sim60]: 2VDM stands for Variant Definition Model, the pure::variants term for configuration model or variant. 36 CHAPTER 4. PORTFOLIO EVOLUTION ANALYSIS Goal Question Metric Testing Resource optimization Reduce testing redundancy [WAG13] Which features have had a strong correlation? Confidence of each correlations per version Prioritize product sampling [ATL+19] Which features have been the most/least popular? Nºof products selecting a feature per version Test case prioritization [HGPB20] Which are the most frequent values for features/attributes? Value frequency per feature/attribute and version Configuration Assistance Recommend features based on their popularity [FLP+21] Which features have been the most/least popular? Nºof products selecting a feature per version Recommend features that are usually selected together [PMK+18] Which features have had a strong correlation? Confidence of each correlations per version Hide “incompatible” features from the recommendations [FLP+21] Which features have been never selected together in any configuration? Confidence of each correlations per version SPL Scoping Detect promotion/demotion feature candidates [BP14, ALR+23] Which features have been the most/least popular? Nºof products selecting a feature per version Reallocate current human resources [DMM+22] To what extent are features being selected by existing or newly added configurations? Features selected by new products per version Prioritize features [MRA+22] Which demand patterns have occurred in which features? Nºof products selecting a feature per version Table 4.1: How could portfolio evolution analysis help? The case of testing resource optimization, configuration assistance, and SPL scoping. •Intelligence Stage: The decision-maker analyzes relevant information, focusing attention on key data sources while filtering out distractions. 2 1 1 1 1 2 4 1 8 6 7 1 4 Please indicate the RELEVANCE of portfolio evolution analysis for CONFIGURATION ASSISTANCE Please indicate the RELEVANCE of portfolio evolution analysis for TESTING RESOURCES OPTIMIZATION Please indicate the RELEVANCE of portfolio evolution analysis for SPL SCOPING Not at all important Slightly important Moderately important Very important Extremely important Figure 4.2: Assessing the relevance of portfolio evolution analysis. 4.6. RESULTS 37 •Design Stage: The decision-maker generates and evaluates options by breaking down information into manageable components. •Choice Stage: The decision-maker assesses criteria to select the best option, weighing factors like probabilities, costs, benefits, and potential outcomes. This involves considering alternatives, decision criteria, and criterion weights based on priorities. The outcome of applying Simon’s theory to a visualization tool for portfolio evolution analysis resulted in PortfolioScan, an artifact offering time-series visualizations based on the concept of feature usage derived from historical portfolio data. By incorporating time-series analysis into decision-making processes, the expectation is that SPL organizations would make more informed decisions supported by trends in quantitative data. The goal of PortfolioScan is to guide decision-makers during their deliberation about what actions (if any) should be taken in SPL scoping. PortfolioScan guides the decision-makers into the alternatives to be considered, and in doing so, prunes the problem space. Thus, our intervention is both a workflow and a companion dashboard that revolve around feature usage as the criterion. Specifically, and following Simon’s model [Sim60], the workflow unfolds along the following stages: 1. Hot-spot Detection (INTELLIGENCE). decision-makers explore overview visualizations of feature usage evolution to detect hot-spot features with noteworthy usage rates (high, low, erratic, etc.). PortfolioScan supports filters to focus attention on features that are candidates for scoping updates. 2. Usage Evolution Analysis (DESIGN). Once focused on a feature subset, decision-makers analyze usage rates in-depth, categorizing trends into manageable components. 3. Impact Analysis (CHOICE). decision-makers evaluate the impact of performing state changes on candidate features. Criteria like business value and cost models are weighed to assess the trade-offs of scoping updates. Fig. 4.3 shows how PortfolioScan supports this multi-stage workflow. The process starts by surfacing feature usage evolution trends to decision-makers (Hot-spot Detection) (Fig. 4.3 (1)). This overview can be filtered to specific features of interest. Toggling heatmap colors into underlying numbers via line charts and trend lines enables more detailed Usage Evolution Analysis (Fig. 4.3 (2)). Moving between intelligence and design views, candidate features emerge for potential scoping updates. Decision-makers can note these by adding ‘-’ and ‘+’ symbols to move features to the decision matrix (Impact Analysis) (Fig. 4.3 (3)). Here, additional criteria beyond usage can be weighed when evaluating feature change impacts. We evaluated PortfolioScan during a monthly pure-systems seminar lasting approximately 90 minutes. We began by introducing portfolio evolution analysis 38 CHAPTER 4. PORTFOLIO EVOLUTION ANALYSIS Figure 4.3: PortfolioScan’s dashboard. (1) Hot-spot Detection by filtering out irrelevant features; (2) Usage Evolution Analysis for selected features through different line charts; and (3), Impact analysis where alternative feature candidates are compared through a decision matrix. opportunities. Next, PortfolioScan’s visualization components were explained while participants asked questions. Participants also had hands-on experience testing PortfolioScan. The evaluation aimed to assess the perceived value of PortfolioScan for conducting SPL scoping from the pure::variants team’s viewpoint. To this end, we evaluated each visual component through a brief questionnaire assessing temporal heatmaps, line charts, and trend pattern charts. We also asked about the usefulness of time-series charts in general. A 5-point Likert scale from “not at all valuable” to “extremely valuable” was used. Two employees pilot tested the questionnaire to ensure clarity. Fig. 4.4 displays the results, indicating an overall positive perception. All questions received “Very Valuable” or “Extremely Valuable” ratings from at least seven participants. Surprisingly, basic time-series charts were rated higher than trend detection, perhaps due to PortfolioScan’s initial prototype having limited techniques (linear regression and Holt-Winters). 4.7 Conclusion & Future Work In this chapter we presented our work on portfolio evolution analysis visualizations. The under review paper can be found in Appendix B.1. The main goal of this research project was to address the limited tool support for portfolio evolution analysis by developing dedicated time-series dash- 4.7. CONCLUSION & FUTURE WORK 39 1 2 2 1 1 2 4 3 3 8 6 5 8 2 1 2 1 How valuable do you think could Temporal Heatmaps be for SPL scoping How valuable do you think could Usage line charts be for SPL scoping How valuable do you think could Trend Pattern Chart be for SPL scoping How valuable do you think could Simple Time-series Charts be for SPL scoping Not at all valuable Slightly valuable Moderately valuable Very valuable Extremely valuable Figure 4.4: Evaluation of PortfolioScan’s visualization mechanisms. boards. To this end we collaborated with pure-systems in an ADR project. The collaboration resulted in three contributions. First, we characterized portfolio evolution practices based on the expertise of pure-systems’ employees. Second, we identified three SPL engineering activities that could potentially benefit from analyzing portfolio evolution: testing resource optimization, configuration assistance, and software product line scoping. The information needs of each activity were characterized using the GQM framework. A questionnaire administered to the pure::variants team (n=13) evaluated the relevance of this analysis. Finally, to demonstrate feasibility, portfolio evolution analysis was built on top of pure::variants. The result is a time-series visualization prototype called PortfolioScan, designed using Simon’s model of decision-making. An initial evaluation (n=13) indicated positive perceptions of PortfolioScan’s visualization capabilities for product scoping. Moving forward, the plan is to enhance PortfolioScan by incorporating additional data sources like ticketing systems. Ticketing systems enable tracking issue reporting and resolution, providing valuable usage and quality data. By analyzing logged “tickets”, we can gain insights not just into feature usage but also robustness (bugs), maturity (lack of improvement requests), and recency (prompt issue resolution). Incorporating these factors with usage data is expected to expand PortfolioScan’s value for not only SPL scoping, but also for testing resource optimization and configuration assistance. Overall, this work demonstrates the relevance of portfolio evolution analysis and takes steps toward enhanced, data-driven decision support. 40 CHAPTER 4. PORTFOLIO EVOLUTION ANALYSIS Chapter 5 Implicit Feedback in SPLs 5.1 Overview This chapter presents the contributions of this thesis to the area of implicit feedback in SPLE. The work resulted in two papers: a platform-based approach that assess the feasibility of bringing implicit feedback into SPLE published in the Journal of Systems and Software (see Appendix A.3) [DMAH24] and a vision paper about the potential benefits of using implicit feedback in SPLE published in the GPCE2024 conference (see Appendix A.4) [MDB23]. 5.2 Practice: Implicit Feedback gathering and analysis Implicit feedback plays a crucial role in the maintenance and evolution of software applications, serving as an indirect yet valuable source of insights into user behavior and preferences. Unlike explicit feedback, which is directly provided by users through reviews, ratings, or direct communication, implicit feedback is gathered from users’ actions and interactions with the software, without them necessarily expressing their opinions or experiences in words [vOG21]. This type of feedback includes data on feature usage frequency, session lengths, navigation paths, and even error occurrences [SP12]. In short, explicit feedback is how users see the software, while implicit feedback is how they actually use it. We focus on implicit feedback. In single-product development, implicit feedback (hereafter just “feedback”) benefits developers and analysts alike. It is used for a variety of purposes, including requirement prioritization and elicitation [WDvSL19, JKBP19, LQW+15], identifying bugs and usability issues [JKBP19], motivating developers through real user value [vOG21], and generally informing software evolution based on actual usage patterns and user needs [ABPS21, KPS+21, OSF+18]. It also provides insights into how users interact with software features [GSJK16], high41 48 CHAPTER 5. IMPLICIT FEEDBACK IN SPLS Figure 5.4: Attributed Feature Models. Each attribute holds the GA API calling parameters that calculate the metric value (top side). At enactment time (i.e., selecting FeedbackPull from the drop-down menu), the API calls are generated out of these parameters and enacted. Results are stored in FeedbackPull.xfm, a copy of the feature model where feature-usage attributes will show the metric values as returned by the API call (bottom side). [BSRC10]. Implicit feedback enables the creation of new analysis tasks. For example, identifying drag features that are seldom used, helping decide whether to deprecate or upgrade them. •Feature Dependency Identification. Detecting dependencies between features is crucial to avoid deriving invalid products [SMK+19]. Implicit feedback could validate modeled dependencies if features are seldom used together, or infer escort dependencies for frequently co-used features. •SPL Testing. Product sampling techniques prioritize a subset of valid products for testing, but output can still be large [ASR18]. Implicit feedback could prioritize testing products with features impacting more users, rather than just maximizing fault detection. It could also help impose constraints to improve scalability and uniformity of random sampling. 5.6. RESULTS 49 Application Engineering While single-product developers create products from scratch, SPL engineers develop products by selecting from available features. As these features may already be in operation, implicit feedback can inform product configuration. •Optimal Product Configuration. This involves configuring products to meet an optimization function, like maximizing performance while minimizing resource consumption [MSH+22]. Implicit feedback can serve as an additional parameter, contributing to objectives like promoting highly used features (usage) or aligning with targeted user preferences (satisfaction). •Feature-based Recommender Systems. Recommender systems assist configuration by predicting relevant features based on partial configurations [PMK+18]. Incorporating implicit feedback on how users interact with similar features could enhance alignment with present user needs, aligning with recommendation approaches outside SPLs. Management •SPL Scoping The management cycle of SPLs encompasses the careful supervision of reusable assets, the tracking of their use and impact, and the periodic reassessment of the SPL’s scope. The process of SPL scoping (i.e., determining the set of features and products included in the SPL) is inherently continuous to align with the dynamic nature of the market [MRA+22]. A critical task in this context is prioritization, often done regarding features, is deciding what parts of the SPL should be evolved or included in the next SPL release. Traditionally, scoping decisions have heavily relied on explicit feedback, either collected during planning sessions or derived from strategic business objectives [KHIA21]. However, leveraging implicit feedback represents a valuable, underutilized strategy for capturing subtle shifts in feature relevance. By maintaining a feedback log, implicit feedback becomes a rich data source for generating queries that help prioritize features based on their actual usage patterns. This approach offers a nuanced method for informing scoping decisions, ensuring that the SPL remains responsive to both market demands and user needs. 5.6.4 Evaluation: Cycle 1 We evaluated the three contributions through survey studies with pure-systems’ employees. The validation took place during a monthly online seminar hosted by pure-systems. The seminar lasted 90 minutes and covered the significance of implicit feedback in SPLE. It discussed the benefits and challenges of implementing implicit feedback in SPLs and we presented IF4SPLE,FEACKER and 50 CHAPTER 5. IMPLICIT FEEDBACK IN SPLS the SPL activities that could benefit from implicit feedback. A live demonstration of FEACKER followed this. Participants were encouraged to ask questions throughout the seminar. Finally, participants were allowed to interact with FEACKER and complete the questionnaires. Eight out of the fifteen seminar attendees completed the survey. The results are displayed as follows: •Figures 5.5 and 5.6 assess the coherence (i.e., the quality of forming a unified whole with traditional SPLE practices) and the consistency (i.e., the quality of seamlessly fitting with prior roles in SPLE) of IF4SPLE respectively. •Figures 5.8 and 5.7 present the results of the TAM used to evaluate FEACKER. •Fig. 5.9 displays the perceived usefulness of implicit feedback for SPLE. 3 2 5 3 Figure 5.5: Assessing coherence for IF4SPL. IF4SPLE.The purpose of evaluating IF4SPL was to assess the coherence and consistency of the IF4SPLE principles for seamlessly introducing implicit feedback into SPLE practices. Figure 5.5 and 5.6 shows the results. Coherence-wise, the consensus among participants is that IF4SPLE seamlessly aligns with existing SPLE methodologies (question COH1). The tasks of Feedback Specification (question COH2) and Feedback Analysis (question COH5) showed unanimous agreement on their execution at the DE level, effectively shifting Implicit Feedback from a product-specific issue to a platform-wide consideration. Contrary to expectations, the process of feedback transformation (i.e., integrating feedback during product generation in a manner akin to traditional pre-compilation directive methods) received a lukewarm reception, with most participants deeming it “neutral”. 5.6. RESULTS 51 5 2 1 1 2 Figure 5.6: Assessing consistency for IF4SPL. Consistency-wise, the general view among participants is that IF4SPLE maintains the conventional distribution of roles within the SPL workforce (question CON1). That said, in line with our proposal in IF4SPLE, pure-systems’ employees seem to favor domain engineers over application engineers when conducting feedback specification and analysis (questions CON2 and CON4). 3 3 5 4 1 2 Figure 5.7: FEACKER’s perceived usefulness. FEACKER. Figure 5.7 presents the chart related to the perceived usefulness variable. It highlights how usefulness is gauged based on the support provided to the IF4SPLE process. The chart shows a positive appreciation among participants on the usefulness of FEACKER, although it also points out that there is not unanimous agreement (a point that will be further explored in the Focus Group of the next section). Regarding the perceived ease of use, as 52 CHAPTER 5. IMPLICIT FEEDBACK IN SPLS 2 1 1 2 Figure 5.8: FEACKER’s perceived ease of use. illustrated in Figure 5.8, the feedback is more definitive about how effortlessly FEACKER integrates with pure::variants. The findings indicate that employing FEACKER does not disrupt the standard GUI interactions of pure::variants. Figure 5.9: Perceived usefulness of implicit feedback for SPL practices. SPLE Activities. Regarding the potential usefulness of implicit feedback in SPLs, participants believe that it can be both suitable and beneficial for the SPLE activities described in the previous section. Figure 5.9 presents the evaluation results in a stacked bar chart, indicating that all activities are seen 5.6. RESULTS 53 as good candidates for integrating implicit feedback into their processes. Only the activities of feature dependency identification and SPL scoping received a “Disagree” from one participant. While these findings are encouraging, further comprehensive studies for each activity are necessary to validate these initial observations. Each of the six activities offers a unique avenue for research into the application of implicit feedback methods within SPLE contexts. 5.6.5 Evaluation: Cycle 2 This section describes a study that aimed to delve into divergent results from FEACKER’s TAM evaluation regarding the introduction of IF4SPLE practices into pure::variants. The goal was to understand this and gather additional insights from the perspective of SPL practitioners of WACline SPL. The focus group consisted of three engineers with at least two years of experience using pure::variants. All were part of WACline’s core development team. Participants were first introduced to the study objectives and proposed intervention. They then collaboratively created a Feedback Model to capture analysis opportunities of interest. This resulted in 18 metrics related to 11 features. Feedback was gathered for a week on WACline products running in a sandbox. Collected usage feedback was presented through an attributed feature model. The focus group discussion centered on the questions that showed significant divergence in the previous TAM evaluation. The three main questions were: •Is the Feedback Model able to capture your feedback needs effectively? This question accounts for the TAM’s US1 statement. Participants found specification cumbersome due to dependency on #ifdef blocks. Two participants missed the ability to record values of variables at execution time. This would enable recording the number of items a user interacts with simultaneously within the graphical user interface, or capturing input values that could potentially cause the system to fail. •How seamlessly were IF4SPLE’s new tasks integrated with your SPLE practices? This question accounts for the TAM’s US2 statement. The integration of feedback functionality within domain engineering was viewed positively, aligning with existing reuse principles, despite concerns about the Feedback Model’s fragility and the implications for product derivation. •How seamlessly was FEACKER integrated with pure::variants’ gestures? This relates to the ease of learning to operate FEACKER. This question accounts for the TAM’s US2 statement. Participants mostly appreciated conducting feedback analysis without leaving pure::variants. This seems to suggest that variant management systems should not overlook this concern. In general, participants appreciated the seamless integration of feedback analysis within pure::variants, highlighting the minimal disruption to existing 54 CHAPTER 5. IMPLICIT FEEDBACK IN SPLS workflows. However, challenges in specifying feedback models and the difficulties for conducting complex feature analysis through attributed feature models were noted, suggesting areas for future improvement and research within the SPL community. 5.7 Conclusion & Future Work In this chapter we presented our work on implicit feedback for SPLs, which is detailed in the papers presented in Appendixes A.3 and A.4. We explored extending feedback practices to the SPL platform by examining how these practices might impact both the SPLE processes and variant management systems. To integrate platform-based feedback into SPLE development, we introduced the IF4SPLE process, placing features at the core of the feedback analysis and resorting to a derivation approach for injecting the feedback code into the product variants’ codebase. Next, we studied how variant management systems can support platform-based feedback. To this end, we developed FEACKER, a proof-of-concept extension to pure::variants, which is publicly available for inspection and use. The evaluation involved real practitioners in a Technology Acceptance Model (TAM) evaluation (n=8) and a focus group (n=3). The results suggest that the approach aligns seamlessly with current practices but raises several issues regarding its generalizability. Participants also expressed concerns about how implicit feedback data is presented in feature models and whether they are the best representation for feedback. The next steps are driven by the limitation in generalizability and the need to enhance how implicit feedback data is visualized. Firstly, the current approach heavily relies on pre-compilation directives, which restricts the solution to annotative SPLs, potentially disregarding feature-oriented programming or component-based SPLs. Additionally, alternative visualizations to feature models should be explored, embracing decision-making workflows that would enable SPL managers to make feedback-driven decisions. Bibliography [ABKS13] Sven Apel, Don S. Batory, Christian Kästner, and Gunter Saake. Feature-Oriented Software Product Lines - Concepts and Implementation. Springer, 2013. [ABPS21] Maurizio Astegher, Paolo Busetta, Anna Perini, and Angelo Susi. Specifying requirements for data collection and analysis in data-driven RE. A research preview. In Requirements Engineering: Foundation for Software Quality - 27th International Working Conference, REFSQ 2021, Essen, Germany, April 12-15, 2021, Proceedings, volume 12685 of Lecture Notes in Computer Science, pages 182–188. Springer, 2021. [AD07] Samuel A. Ajila and Razvan T. Dumitrescu. Experimental use of code delta, code churn, and rate of change to understand software product line evolution. Journal of Systems and Software, 80(1):74–91, 2007. [AGK+22] Sofia Ananieva, Sandra Greiner, Timo Kehrer, Jacob Krüger, Thomas Kühn, Lukas Linsbauer, Sten Grüner, Anne Koziolek, Henrik Lönn, S Ramesh, and Ralf Reussner. A conceptual model for unifying variability in space and time: Rationale, validation, and illustrative applications. Empirical Software Engineering, 27(5):101, 2022. [ALR+23] Mathieu Acher, Luc Lesoil, Georges Aaron Randrianaina, Xhevahire Tërnava, and Olivier Zendra. A call for removing variability. In Proceedings of the 17th International Working Conference on Variability Modelling of Software-Intensive Systems, VaMoS 2023, Odense, Denmark, January 25-27, 2023, pages 82–84. ACM, 2023. [AMD21] Maider Azanza, Leticia Montalvillo, and Oscar Díaz. 20 years of industrial experience at splc: a systematic mapping study. In Proceedings of the 25th ACM International Systems and Software Product Line Conference - Volume A, SPLC ’21, page 172–183, New York, NY, USA, 2021. Association for Computing Machinery. 55 56 BIBLIOGRAPHY [ASR18] Mustafa Al-Hajjaji, Michael Schulze, and Uwe Ryssel. Similarity analysis of product-line variants. In Proceeedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10-14, 2018, pages 226–235. ACM, 2018. [ATL+19] Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. Effective product-line testing using similarity-based product prioritization. Software and Systems Modeling, 18(1):499–521, 2019. [BCS10] Muhammad Ali Babar, Lianping Chen, and Forrest Shull. Managing variability in software product lines. IEEE Software, 27(3):89–91, 94, 2010. [Beu19] Danilo Beuche. Industrial variant management with pure::variants. In ACM International Conference Proceeding Series, volume B, pages 1–3, New York, New York, USA, sep 2019. Association for Computing Machinery. [BP14] Goetz Botterweck and Andreas Pleuss. Evolution of software product lines. In Evolving Software Systems, pages 265–295. Springer, 2014. [BR88] Victor R Basili and H Dieter Rombach. The tame project: Towards improvement-oriented software environments. IEEE Transactions on software engineering, 14(6):758–773, 1988. [BSRC10] David Benavides, Sergio Segura, and Antonio Ruiz-Cortés. Automated analysis of feature models 20 years later: A literature review. Information Systems, 35(6):615–636, 2010. [CH07] Peter Checkland and Sue Holwell. Action research: its nature and validity. Information systems action research, pages 3–17, 2007. [CMZ+20] Noptanit Chotisarn, Leonel Merino, Xu Zheng, Supaporn Lonapalawong, Tianye Zhang, Mingliang Xu, and Wei Chen. A systematic literature review of modern software visualization. Journal of Visualization, 23(4):539–558, 2020. [CN02] Paul Clements and Linda M. Northrop. Software product lines - practices and patterns. SEI series in software engineering. Addison-Wesley, 2002. [dAdBdALR+16] Pedro de Alcântara dos Santos Neto, Ricardo Britto, Ricardo de Andrade Lira Rabêlo, Jonathas Jivago de Almeida Cruz, and Werney Ayala Luz Lira. A hybrid approach to suggest software product line portfolios. Applied Soft Computing, 49:1243–1255, 2016. BIBLIOGRAPHY 57 [Dav89] Fred D. Davis. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q., 13(3):319–340, 1989. [Die07] Stephan Diehl. Software visualization: visualizing the structure, behaviour, and evolution of software. Springer Science & Business Media, 2007. [DMAH24] Oscar Díaz, Raul Medeiros, and Mustafa Al-Hajjaji. How can feature usage be tracked across product variants? implicit feedback in software product lines. Journal of Systems and Software, page 112013, 2024. [DMM+22] Oscar Díaz, Leticia Montalvillo, Raul Medeiros, Maider Azanza, and Thomas Fogdal. Visualizing the customization endeavor in product-based-evolving software product lines: a case of action design research. Empirical Software Engineering, 27(3):75, 2022. [FLLHE14] Stefan Fischer, Lukas Linsbauer, Roberto Erick LopezHerrejon, and Alexander Egyed. Enhancing clone-and-own with systematic reuse for developing software variants. In 2014 IEEE International Conference on Software Maintenance and Evolution, pages 391–400, 2014. [FLP+21] Alexander Felfernig, Viet Man Le, Andrei Popescu, Mathias Uta, Thi Ngoc Trang Tran, and Müslüm Atas. An overview of recommender systems and machine learning in feature modeling and configuration. In VaMoS’21: 15th International Working Conference on Variability Modelling of SoftwareIntensive Systems, Virtual Event / Krems, Austria, February 9-11, 2021, pages 16:1–16:8. ACM, 2021. [FS17] Brian Fitzgerald and Klaas-Jan Stol. Continuous software engineering: A roadmap and agenda. Journal of Systems and Software, 123:176–189, 2017. [FSK+16] Thomas Fogdal, Helene Scherrebeck, Juha Kuusela, Martin Becker, and Bo Zhang. Ten years of product line engineering at danfoss: lessons learned and way ahead. In Proceedings of the 20th International Systems and Software Product Line Conference, SPLC 2016, Beijing, China, September 16-23, 2016, pages 252–261. ACM, 2016. [GBT+19] José Angel Galindo, David Benavides, Pablo Trinidad, Antonio Manuel Gutiérrez-Fernández, and Antonio Ruiz-Cortés. Automated analysis of feature models: Quo vadis? Computing, 101(5):387–433, 2019. 64 BIBLIOGRAPHY [Sch08] Mario Schmidt. The sankey diagram in energy and material flow management. Journal of Industrial Ecology, 12(2):173– 185, 2008. [SF18] Klaas-Jan Stol and Brian Fitzgerald. The abc of software engineering research. ACM Transactions on Software Engineering and Methodology, 27(3), 2018. [SHP+11] Maung K. Sein, Ola Henfridsson, Sandeep Purao, Matti Rossi, and Rikard Lindgren. Action design research. MIS Q., 35(1):37–56, 2011. [Sim60] Herbert A Simon. The new science of management decision. Harper & Brothers, 1960. [SLB14] Mojtaba Shahin, Peng Liang, and Muhammad Ali Babar. A systematic review of software architecture visualization techniques. Journal of Systems and Software, 94:161–185, 2014. [SMK+19] Daniel Strüber, Mukelabai Mukelabai, Jacob Krüger, Stefan Fischer, Lukas Linsbauer, Jabier Martinez, and Thorsten Berger. Facing the truth: benchmarking the techniques for the evolution of variant-rich systems. In Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume A, Paris, France, September 913, 2019, pages 26:1–26:12. ACM, 2019. [SP12] Rien Sach and Marian Petre. Feedback: How does it impact software engineers? In 5th International Workshop on Co-operative and Human Aspects of Software Engineering, CHASE 2012, Zurich, Switzerland, June 2, 2012, pages 129– 131. IEEE Computer Society, 2012. [SPKM15] Veena Subramanyam, Deepak Paramshivan, Amit Kumar, and Md. Alam Hossain Mondal. Using sankey diagrams to map energy flow from primary fuel to end use. Energy Conversion and Management, 91:342–352, 2015. [TGAJ16] Paul Temple, José A. Galindo, Mathieu Acher, and JeanMarc Jézéquel. Using machine learning to infer constraints for product lines. In Proceedings of the 20th International Systems and Software Product Line Conference, SPLC ’16, page 209–218, New York, NY, USA, 2016. Association for Computing Machinery. [vOG21] Simon van Oordt and Emitza Guzman. On the role of user feedback in software evolution: a practitioners’ perspective. In 29th IEEE International Requirements Engineering Conference, RE 2021, Notre Dame, IN, USA, September 20-24, 2021, pages 221–232. IEEE, 2021. BIBLIOGRAPHY 65 [WAG13] Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. Minimizing test suites in software product lines using weight-based genetic algorithms. In Genetic and Evolutionary Computation Conference, GECCO ’13, Amsterdam, The Netherlands, July 6-10, 2013, pages 1493–1500. ACM, 2013. [WDvSL19] Chong Wang, Maya Daneva, Marten van Sinderen, and Peng Liang. A systematic mapping study on crowdsourced requirements engineering using user feedback. Journal of Software: Evolution Process, 31(10), 2019. [WMMR06] Roel J. Wieringa, Neil A. M. Maiden, Nancy R. Mead, and Colette Rolland. Requirements engineering paper classification and evaluation criteria: a proposal and a discussion. Requirements Engineering, 11(1):102–107, 2006. [YKSJ07] Ji Soo Yi, Youn ah Kang, John Stasko, and J.A. Jacko. Toward a deeper understanding of the role of interaction in information visualization. IEEE Transactions on Visualization and Computer Graphics, 13(6):1224–1231, 2007. [ZB14] Bo Zhang and Martin Becker. Reverse engineering complex feature correlations for product line configuration improvement. In 40th EUROMICRO Conference on Software Engineering and Advanced Applications, EUROMICRO-SEAA 2014, Verona, Italy, August 27-29, 2014, pages 320–327. IEEE Computer Society, 2014. [ZHAK18] Qian Zhao, F. Maxwell Harper, Gediminas Adomavicius, and Joseph A. Konstan. Explicit or implicit feedback? engagement or satisfaction?: a field experiment on machine-learningbased recommender systems. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018, Pau, France, April 09-13, 2018, pages 1331–1340. ACM, 2018. [ZSG15] Jelena Zdravkovic, Eric-Oluf Svee, and Constantinos Giannoulis. Capturing consumer preferences as requirements for software product lines. Requirements Engineering, 20(1):71– 90, 2015. 66 BIBLIOGRAPHY Chapter 6 Conclusions 6.1 Overview This thesis explores how visualization tools can support SPL evolution. The introduction chapter motivated this focus by highlighting the complexities of SPL evolution and the lack of dedicated tool support. A systematic mapping study revealed specific gaps around examining portfolio evolution, and leveraging implicit feedback. Akin to ADR, the research aimed to address these gaps through developing and evaluating visualization artifacts within a real-world organizational setting. This resulted in three contributions that are deeply intertwined with the concept of enhancing the understandability of the evolution of SPLs. On one hand, customization analysis provides insights into the modifications made to individual product variants, informing decisions about reintegrating those customizations into the core assets. On the other hand, portfolio evolution analysis offers a broader perspective, analyzing trends and patterns in feature adoption across the entire product portfolio, guiding future SPL evolution strategies. Finally, implicit feedback cuts across both customization and portfolio evolution analyses. By incorporating implicit feedback data into the core SPL development process, domain engineers can leverage user interaction data to refine their understanding of feature-based customizations and feature adoption trends. This feedback data serves as a valuable input for both customization analysis and portfolio evolution analysis, ensuring that SPL managers can make well-informed decisions aligned with user preferences and evolving market conditions. Collectively, these three contributions aim to establish a comprehensive and data-driven approach to SPL management and evolution, empowering SPL managers to navigate the complexities of reintegrating customizations, prioritizing features, and aligning the SPL with user needs and market dynamics. This concluding chapter summarizes the results and limitations of each contribution and presents the publications produced over the course of the thesis. The chapter concludes by suggesting directions for future research to further 67 68 CHAPTER 6. CONCLUSIONS the field of SPL evolution visualization. 6.2 Results This thesis proposes software visualization tools to cope with the intrinsic complexity of SPL evolution. Next paragraphs detail the results obtained throughout the thesis: •Chapter 2 reviews the current state of visualizations for the evolution of SPLs by mapping approaches not only for systematically managed families of products, but also those developed through clone & own (i.e., VRSs). The work analyzed 41 studies to understand the analysis conducted, visualizations used, and research maturity. We found limitations around overlooked stakeholder roles, evolution scenarios, scalability of visualizations, and data sources leveraged. While visualizations for SPLs appear promising, they remain in an early development stage. The study identified research opportunities including exploring overlooked data sources like version control and user feedback. Accordingly, this thesis examined leveraging these data sources, with version control visualizations and user feedback. Overall, the mapping study categorized the state of research and highlighted pathways for advancing the visualization research agenda for VRSs evolution, encompassing both SPL and clone & own approaches. •Chapter 3 presents a visual approach using alluvial diagrams for customization analysis in product-based evolving SPLs. This work stemmed from product customization practices at Danfoss Drives. We identified information needs for customization analysis and proposed adapting code churn metrics for SPL constructs. However, the high number of potential customization metrics necessitated appropriate visualizations. We addressed this through alluvial diagrams in the CustomDIFF tool. Evaluations suggested alluvial diagrams effectively depict customization efforts at different levels of detail, advancing traditional SPL diff utilities. Future work includes further assessing the value of CusotmDIFF, incorporating additional data sources, exploring new metrics, and integrating CustomDIFF into DevOps workflows. •Chapter 4 presents our work on portfolio evolution analysis visualizations. The goal was to address limited tool support by developing time-series dashboards. Through an ADR project with pure-systems, we identified portfolio analysis needs using GQM and designed the PortfolioScan prototype. PortfolioScan follows decision-making theories to support SPL managers during SPL scoping through dedicated time-series dashboards. Evaluations indicated positive perceptions of PortfolioScan’s capabilities for analyzing portfolio trends to inform decisions. Moving forward, we plan to expand data sources beyond usage to include quality and robustness factors from ticketing systems. This is expected to increase value for feature-based decisions. 6.3. PUBLICATIONS 69 •Chapter 5 addresses the challenge of making SPLs implicit feedback aware. We explored extending feedback practices to the SPL platform by examining impacts on SPLE processes and variant management systems. We introduced the IF4SPLE process to integrate platform-based feedback using features for analysis and derivation for tracking code injection. We also developed FEACKER, a proof-of-concept extension to pure::variants. Evaluations suggested alignment with current practices but issues in generalizability. Participants also expressed concerns on feature models for presenting feedback. Next steps include enhancing generalizability beyond annotative SPLs and exploring alternative visualizations to feature models that support feedback-driven decisions. 6.3 Publications This work is presented as a compendium of peer-reviewed publications. It is noteworthy the fact that the majority of the papers of this thesis have been published under an Open Access agreement1. We next list the publications that endorse this thesis: •Raul Medeiros, Jabier Martinez, Oscar Díaz, Jean-Rémy Falleri, Visualizations for the evolution of Variant-Rich Systems: A systematic mapping study, Information and Software Technology, Volume 154, 2023, 107084, https://doi.org/10.1016/j.infsof.2022.107084. Related to Chapter 2. •Oscar Díaz, Leticia Montalvillo, Raul Medeiros, Maider Azanza, Thomas Fogdal, Visualizing the customization endeavor in product-based-evolving software product lines: a case of action design research, Empirical Software Engineering, Volume 27, 75 (2022), https://doi.org/10.1007/ s10664-021-10101-6. Related to Chapter 3. •Oscar Díaz, Raul Medeiros, Mustafa Al-Hajjaji, How can feature usage be tracked across product variants? Implicit Feedback in Software Product Lines, Journal of Systems and Software, 2024, 112013, https://doi.org/ 10.1016/j.jss.2024.112013. Related to Chapter 5. •Raul Medeiros, Oscar Díaz, David Benavides. Unleashing the Power of Implicit Feedback in Software Product Lines: Benefits Ahead. 2023. In Proceedings of the 22nd ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (GPCE 2023). Association for Computing Machinery, New York, NY, USA, 113–121. https://doi.org/10.1145/3624007.3624058. Related to Chapter 5. Publications under review: 1The open access fees were funded by the agreement between Elsevier and the CRUE(Conferencia de Rectores de las Universidades Españolas) 70 CHAPTER 6. CONCLUSIONS •Raul Medeiros, Oscar Díaz, Mustafa Al-Hajjaji, Michael Schulze. Portfolio evolution analysis in software product lines: Unveiling the dynamics with dedicated time-series dashboards. Under review at the Journal of Systems and Software, 2024. Related to Chapter 4. Other publications in the field of SPLs in which the student has participated include: •Maider Azanza, Arantza Irastorza, Raul Medeiros, Oscar Díaz, Onboarding in Software Product Lines: Concept Maps as Welcome Guides, 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET), Madrid, ES, 2021, pp. 122-133, https://doi.org/10.1109/ICSE-SEET52601.2021. 00022 •Raul Medeiros, Oscar Díaz,. Assisting Mentors in Selecting Newcomers’ Next Task in Software Product Lines: A Recommender System Approach. 2022. In: Franch, X., Poels, G., Gailly, F., Snoeck, M. (eds) Advanced Information Systems Engineering. CAiSE 2022. Lecture Notes in Computer Science, vol 13295. Springer, Cham. https://doi.org/10.1007/ 978-3-031-07472-1_27 •Raul Medeiros, Unburdening Onboarding in Software Product Lines, 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), Madrid, ES, 2021, pp. 260-262, https://doi.org/10.1109/ICSE-Companion52605.2021.00121 6.3.1 A Case of Open Science We are committed to Open Science in general and to software artifacts in particular. Making software artifacts openly available is of paramount importance to the advancement of science in software engineering for several reasons. Open access to these artifacts enables reproducibility, which is the cornerstone of scientific progress. When researchers publish their findings along with the actual software, datasets, and tools they used, others can validate the results and build upon them. This practice enhances the credibility of the research by allowing the community to replicate studies and confirm findings. Furthermore, open software artifacts foster collaboration and knowledge sharing, which can accelerate innovation. With access to a broad range of tools and codebases, researchers and practitioners can experiment with and extend existing work, pushing the boundaries of the field further than they could in isolation. The educational impact of open software artifacts is also significant. They serve as valuable learning resources for students and educators alike, offering real-world examples and hands-on experience with tools and techniques that are actively used in the industry. Lastly, the availability of open software artifacts can democratize access to advanced software engineering tools and methodologies. Smaller institutions or 6.4. RESEARCH VISIT 71 independent researchers who may not have the resources to develop such tools from scratch are able to participate in cutting-edge research. On these grounds, we are committed to the view of science artifacts as companions to scientific publications and the availability of replication materials. All artifacts, except for PortfolioScan, are publicly available and well-documented2 and we also published some replication packages in Zenodo: •Visualization gallery of the mapping study: https://doi.org/10.5281/ zenodo.7057654 •CustomDIFF:https://github.com/onekin/CustomDiff &https://doi. org/10.5281/zenodo.5728000 •FEACKER:https://github.com/onekin/FEACKER &https://doi.org/ 10.5281/zenodo.8187116 •Recommentor (CAISE pub.): https://github.com/onekin/RecomMentor 6.4 Research visit One vital aspect of scientific research is investigating problems relevant to society and industry. During my PhD journey, I wondered whether the problems we tackled were truly important for SPL practitioners. While we rigorously assessed relevance through literature reviews, sometimes I felt the literature did not fully align with industry needs. This concern led my supervisor and me to seek a research internship where I could encounter real-world SPL challenges. We contacted Danilo Beuche, former CEO and founder of pure-systems3, who kindly accepted me for a three-month visit at their headquarters in Magdeburg. This experience provided invaluable professional and personal growth. Professionally, I gained first-hand exposure to practitioners’ concerns, collaborated on portfolio evolution analysis, and validated ideas with them (see Appendix B.1). During my time in Germany, I had the chance to connect with various cultures. It was my initial long-term stay abroad, providing me with a valuable opportunity to explore different viewpoints. This internship helped me to anchor my research in actual industry challenges, some of which had already been studied in academia and others that were still unexplored. It underscored the significance of closing the gap between academic interests and real-world needs to achieve the greatest societal impact. 2As an example of the Open Science initiative followed in this thesis, CustomDIFF was awarded the Open Science badge in the publication in the Empirical Software Engineering journal https://link.springer.com/journal/10664/topicalCollection/ AC_deca3131734b61f9ddc593f577c02eb1. 3Note that at the time of writing this thesis pure-systems GmbH had recently been acquired by PTC Inc. 72 CHAPTER 6. CONCLUSIONS 6.5 Personal insights A PhD thesis involves a lot more than just acquiring research and technical skills. It changes the way you see society and your knowledge area, in my case Software Engineering, and teaches you how to self-organize yourself to be productive. I can assure without doubt that I am a totally different person from the guy that joined the PhD program in September 2019. The experience has broadened my perspective, enabling me to think critically, question assumptions, and approach problems from multiple angles. It has instilled in me a deep appreciation for the scientific method and the importance of rigor in research. Furthermore, the PhD journey has played a pivotal role in developing various of my soft skills, particularly public speaking. As someone who tends to be shy, the need to present my research at conferences and defend my work has helped me overcome my reservations and become more confident in articulating my ideas to diverse audiences. This valuable skill will undoubtedly serve me well in my future endeavors, whether in academia or industry. Moreover, the PhD journey has highlighted the crucial role of meticulous planning and time management, especially when collaborating with others who might have tight schedules. Achieving ambitious research goals requires careful coordination, effective communication, and the ability to adapt to changing circumstances while staying focused on the overarching goals. 6.6 Future work The final hurdle of a doctorate is the defense of their thesis. However, the journey of its contributions can continue indefinitely as there is always room for further research advancement. This thesis is not an exception and leaves opportunities and challenges to be further explored: 6.6.1 Exploring and deepening current gaps on SPL evolution visualization •Explore modern visualizations. The field of software visualization is experiencing rapid growth in novel techniques, including dynamic/live visualization, augmented reality, virtual reality, and more. As future work, incorporating modern visualization techniques into variability-rich systems could help address challenges stemming from large, multifaceted data. Modern visualizations may provide enhanced analysis support and potentially improve comprehension and decision-making in SPL evolution. •Dedicated visualizations for project managers. The mapping revealed a lack of visualizations to assist project managers during the evolution of VRSs. Since managers are typically the people responsible for making decisions, it would be valuable to design and develop tailored visualizations to support the evolution decisions that managers must make. 6.6. FUTURE WORK 73 •Visualizations for overlooked scenarios. There is a lack of visual support for non-functional property analysis, architecture recovery, and feature model synthesis evolution scenarios. Exploring tailored visualizations for these under-explored SPL evolution scenarios offers a worthy avenue for future work. Filling these visualization gaps could provide enhanced analysis and decision-making capabilities when evolving VRSs and more specifically, SPLs. 6.6.2 Supporting customization analysis •Field studies on customization analysis. The evaluation carried out at Danfoss Drives is limited to the perceived usefulness and ease of use of the participants. In order to assess the real impact of CustomDIFF in customization analysis field studies should be conducted. Field studies explore a phenomenon in a specific real-world setting. In this case the phenomenon to study would be that of customization analysis, but having CustomDIFF as a support tool. Those studies should be performed in other companies other than Danfoss, particularly in industries with varying complexity and customization demands, to validate the generalizability of the findings. •Exploring additional data sources and metrics. Currently, CustomDIFF relies solely on code churn metrics obtained from Git repositories. It would be interesting to enrich CustomDIFF with additional metrics gathered from other data sources, such as the number of bugs per feature in the latest release, the business value of the features or product variants, and so on. •Integrate CustomDIFF into a continuous development and deployment setting. CustomDIFF could be incorporated into the development workflow to provide automated, real-time insights into new code changes and their potential impact on future integrations. For example, CustomDIFF metrics could be generated on each code commit and made available to developers and product managers through dashboards or notifications. This would enable rapid assessment of customization effects from ongoing development, supporting prompt and informed decision-making. •Alternative Visualization Techniques. While alluvial diagrams effectively reveal customization efforts between products and features, other visualization methods may provide additional insights. Potential options include hierarchical edge bundling for showing inheritance structures, hive plots or Kiviat diagrams for multivariate data, chord diagrams for interrelationships, or advanced 3D and interactive visualizations. Evaluating pros and cons of different techniques could uncover improved ways to communicate and comprehend the complexities of product customization. •Automate Decision-Making Processes. The current rise of AI, with generative AI at the front, opens the doors to explore the automation of cus- Information and Software Technology 154 (2023) 107084 3 R. Medeiros et al. Fig. 1. Variant Synchronization. Using similarity analysis, this tool identifies common, shared and unique artifacts among variants. On the upper left side, the visualization shows a similarity distribution of 6 variants. Colors represent the degree of commonality of the artifacts: green (common), cyan (shared) and yellow (unique). On the upper right side, a treemap groups the shared lines of code regarding the number of products where they are present. On the bottom left side, the similarity distribution is displayed within each implementation element (i.e., package or folder) allowing a more focused structural exploration. Finally, on the bottom right corner, a phylogenetic distance diagram provides information about which variants are more similar. Figure credits [S1]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) 3.1.1. Variant Synchronization (VS) Setting. Clone & own and variant-based evolved SPLs. When bug fixes or feature implementations are made in one variant, they need to be propagated to other variants. During Variant Synchronization, it is important to identify the variants affected by a bug fix or a feature upgrade. To this end, criteria such as the cost of the propagation or the importance of the variant can be used. Example. Variant synchronization benefits from visualizing the distribution of the lines of code of each variant in terms of ‘core’, ‘shared’, or ‘unique‘. In this way, if a bug is fixed in the ‘shared’ part of a variant it becomes a candidate to be rippled across the other variants sharing the same code. This approach is followed in [S1] using set-based similarity analysis (see Fig. 1). 3.1.2. Variant Integration (VI) Setting. Variant-based evolved SPLs. Here, application engineers are allowed to tune the codebase to meet the specifics of a variant without waiting for this specific to be introduced in the SPL platform. Bug fixes or urgent customer requests might not wait for being conducted at the SPL platform, and hence, they are branched off into a separate codebase. Eventually, these branches need to be merged back into the SPL Master. Being variants from the same SPL, they are based on the very same core assets, and hence, bug fixes or functional enhancements undertaken for a variant might well serve other variants. Here, application engineers need to be aware of potential coordination problems right during coding rather than deferring it until merging time. Example. PeeringHub [S2] (see Fig. 2) advocates for making application engineers aware of potential coordination problems right during coding rather than deferring it until merge time. To this end, PeeringHub introduces the notion of ‘peering bar’ for GitHub. These visual bars reflect whether the codebase of a variant’s feature is being upgraded in other variant branches. PeeringHub helps by making developers aware of potential coordination problems right during variant coding rather than deferring it until variant specifics are merged back into the platform master. 3.1.3. Feature Identification and Location (FIL) Setting. Migration from clone & own to an SPL platform. Feature identification aims to determine which features are present, while feature location tackles the relationship between features and assets. Developers may wish to determine which features already exist in the system and which features are implemented in which assets (e.g., source code, models, requirements, or other types of artifacts) as a means of better supporting clone & own development as well as preparing for the migration to an SPL platform. Example. During the migration from clone & own to an SPL platform, SPL engineers might need to identify which parts of the variants are common and which ones are specific. Fig. 3 shows a visualization that uses clone detection for this purpose. The visualization shows the parts of the source code that are common (red), variant specific (white), and those that might be shared among some variants but not all (light red). This way, the user can interact with the results (e.g., focusing on specific source code files) and identify mandatory features and features that only appear in a given variant or a set of variants [S3]. 3.1.4. Constraints Extraction (CE) Setting. Migration from clone & own to an SPL platform. On the way towards an SPL, meaningful sets of functional requirements are named as a ‘feature’. Features hold dependencies among them: a feature might require/exclude other features. This information might not be initially known. Developers might be assisted through an automated constraint extraction approach. Constraint extraction exhibits two needs. First, it has to be easy to identify which constraints are already formalized. Secondly, constraints extraction techniques are not 100% accurate, and hence, means are needed for measuring the confidence of the extracted constraints. Example. Feature Relation Graphs (FRoGs) [S4]. Fig. 4 places the feature under analysis at the center while the rest of the features are disposed at a distance based on the frequency they appear together in the available configurations. It can be used to discover non-formalized ‘requires’ and ‘excludes’ constraints between features as well as soft-constraints such as one feature ‘encourages’ or ‘discourages’ another. Information and Software Technology 154 (2023) 107084 4 R. Medeiros et al. Fig. 2. Variant Integration. PeeringHub mimics GitHub topic bar but now colors denote code churns per feature within a branch. Branches hold temporary developments of variants (e.g., productDenmark) before being merged back into the Master. In the example, the PeeringBar hints at code efforts being conducted for the variant productDenmark, specifically, for features WindSpeed,English,AirPressure, etc. In this way, if you are upgrading namesake features, you better engage with your productDenmark colleagues to prevent later conflicts during merging. Figure credits [S2]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Fig. 3. Feature Identification and Location. The visualization consists of a heatmap inside a treemap. The treemap represents the structure of each variant’s artifacts, while the heatmap represents the degree of commonality of each variant implementation artifact. Using the colors, a developer can identify the parts of the source code that are common (red), the parts that are product specific (white), and the parts that might be shared among some product variants but not by all of them (light red). Figure credits [S3]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Information and Software Technology 154 (2023) 107084 5 R. Medeiros et al. Fig. 4. Constraint Extraction. Feature Relation Graphs use nodelink diagrams on top of a radial network. Nodes represent feature names. The node in the center is the feature under analysis while the arrangement of the others represents the relationship they have with the central feature. Concretely, the arrangement represents three types of relationships: type of constraint (proximity to the center), stakeholder perspective (radial position), and level of formalization (edge type). Using the visualization, developers identify potential feature constraints related to the feature under analysis. Figure credits [S4]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Fig. 5. Architecture Recovery. When it comes to integrating a set of existing similar variants on the way towards an SPL, it is essential to abstract a common architecture. This visualization abstracts from the variants’ component architecture to highlight the variance presence. The component diagram depicts which modules are present in several variants (list of names) and to which extent the module is similarly implemented (bar chart). Figure credits [S5]. 3.1.5. Architecture Recovery (AR) Setting. Migration from clone & own to an SPL platform. While feature models capture functionality dependencies, architectural models focus on the dependencies of implemented classes or components. Automatic extraction of architectural dependencies can be assisted through architecture recovery techniques. Architecture Recovery can be performed from a configurable platform or from a set of variants. Example. Building a platform architecture out of existing variants is error-prone and tedious. Fig. 5 introduces a visualization that abstracts and aggregates this information. Commonalities and variabilities Information and Software Technology 154 (2023) 107084 6 R. Medeiros et al. Fig. 6. Transformation. FeatureVisu uses a nodelink visualization built on top of a clustering method to depict the feature cohesion metric. For instance, on the left side, we can observe how the implementation elements of the feature BaliBuild (green) are more cohesive than those of the features Bali (blue) and CodeGen (red). On the right side, we can observe an excerpt of the previous layout but this time showing the dependency links between the implementation elements. Figure credits [S6]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) among variant architectures are captured from two perspectives: variance presence (i.e., if the existence of the module is variable or not) and variance realization (i.e., if the implementation of the module is different across variants) [S5]. 3.1.6. Transformations (TR) Setting. Evolution of an SPL platform. The developer can use transformation techniques to decrease manual effort during evolution activities. Transformation techniques include refactoring and partial refactoring of an SPL platform. For instance, high cohesion of a feature is a desired property and a relevant motivation for refactoring. For this, developers look to what extent implementation elements of a feature depend on each other, minimizing external dependencies. Example. FeatureVisu (see Fig. 6) supports feature cohesion by creating clustering among features with a high cohesion. 3.1.7. Functional Testing (FT) Setting. Clone & own and SPL platforms. Regression testing aims at ensuring that an application still functions as expected after any code changes. In view of the impossibility of testing hundreds of features and variants, deciding which features and configuration to test becomes critical. To tame this scalability issue, testers resort to distinct heuristics: hot spots (i.e., the larger the change, the more likely the impact) or variant similarity comparison analysis to increase test coverage. Example. FeatureCloud (see Fig. 7) is a tree-cloud visualization to identify hot-spot features, i.e., features undergoing substantial change in their codebase, i.e., ifdef blocks. FeatureCloud depicts feature change along three metrics: LOC (color opacity), scattering (font size), and tangling (position in the word cloud). Using FeatureCloud, developers identify which features had complex updates and thus, might require more dedicated testing. 3.1.8. Analysis of Non-Functional Properties (ANF) Setting. Clone & own and SPL platforms. VRSs are common among cyber–physical systems where non-functional requirements are key (e.g., performance in a safety–critical system, memory consumption in an embedded system with resource constraints). In safety–critical domains, engineers might need to explore which variants are optimal for their goals. This might require the analysis of the impact of features and feature interactions on quality properties, optimization of configurations towards attaining a given quality criteria, and analysis of trade-offs among non-functional properties. Example. Fig. 8 presents a bubble chart to analyze optimal variants: variants are located based on four custom non-functional metrics (battery-life, security, productivity and cost). 3.1.9. Co-evolution of Problem Space and Solution Space (CPS) Setting. Compared to the evolution of one-off applications, VRS evolution introduces additional challenges since the evolution should be synchronized across a potentially high number of variants [2]. The inter-dependency between these variants varies from highly-independent (e.g., the clone & own approach where ‘clones’ might evolve totally independently from its ancestor) to totally-dependent variants (e.g., configurable software-product lines where new variant requirements should be first considered at the core platform before impacting the variants). An automated approach may suggest co-evolution activities to maintain the two spaces in sync. This scenario includes both evolving the solution space based on the problem space, and vice-versa. Example. New feature dependencies may be reflected uniquely in the code without being mirrored in the feature model due to development urgency during SPL evolution. Fig. 9 presents a visualization that identifies dissonances between the problem and the solution space. The visualization displays inconsistencies between the current feature model and its implementation through different views: the dependency matrix shows inconsistencies between feature dependencies in red (topright); the feature model shows suggested feature dependencies with colored lines (middle-right); the code-view displays the feature implementation (middle-left); and finally, the nodelink diagram depicts the code-level interactions (bottom). 3.2. Dimension 2: Visualization approach Visualization can be regarded as the intervention to cope with increasingly complex evolution scenarios. By introducing Visual Analytics, decision makers can focus their attention on visualization-enabled analytical reasoning while benefiting from automatic data processing techniques. This dimension can be characterized along with the sort of interaction type [24] and the visualization strategy [10]. Information and Software Technology 154 (2023) 107084 7 R. Medeiros et al. Fig. 7. Functional Testing. FeatureCloud helps identify hot-spot features, i.e., features undergoing substantial changes in their codebase from the last release of the SPL platform. FeatureCloud takes the code churn, and extracts the pre-compilation directives from the associated ifdef blocks. On these grounds, the tool generates a word-cloud using feature names as words, and pre-compilation directives as phrases. Nodes stand for features. Color opacity, font size, and distribution of nodes denote the size of the change, the scattering degree and the tangling degree, respectively, for the feature at hand. Take the case of feature AUTO_BED_LEVELING_UBL (right-upper corner in the screenshot). From the last release, this feature has been subject to few changes (light-blue color). Yet, testing is guessed to be complicated. First, this feature’s changes are scattered along distinct ifdef blocks (as reflected by the medium font size). Second, these changes pertain to ifdef blocks whose pre-compilation directives involve feature AUTO_BED_LEVELING_UBL but also other five features that are depicted in the same radial branch, e.g., EEPROM_SETTING. Figure credits [S7]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Fig. 8. Analysis of Non-Functional Properties. Bubble Front Graphs display four custom quality (non-functional) metrics in a bubble chart. This way, users can select the optimal variants based on those quality metrics. The chart uses four dimensions to represent those quality metrics: horizontal axis X (bottom), the vertical axis Y (left), bubble color Z (top) and bubble size 𝑇(right) to visualize up to four quality dimensions simultaneously. In the figure, productivity is represented as a vertical coordinate (Y); batterylife is represented as a horizontal coordinate (X); security is represented as a color (Z); and cost is represented as a size (T). Figure credits [S8]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Information and Software Technology 154 (2023) 107084 8 R. Medeiros et al. Fig. 9. Co-evolution of Problem Space and Solution Space. This visualization guides developers on the evolution of the feature model by showing them inconsistencies between the current feature model and its implementation. The top-right matrix displays the inconsistencies (marked in red). The visualization also suggests feature-level dependencies which are shown in the feature model on the middle-right (colored lines). Developers can also consult the implementation of the selected feature on the upper-left part and check its code-level dependencies in the nodelink diagram on the lower part. Figure credits [S9]. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.) Visualization interaction. Yi et al. introduce the following interaction schema [24]: •Filter. When the visualization allows users to specify several conditions that the data that is shown must satisfy. •Select. When the visualization provides users with the ability to focus on a specific data item(s). •Abstract/Elaborate. When the visualization provides users with the functionality of adjusting the abstraction/detail level of the data. •Connect. When the visualization provides users with the option of checking associations and relationships between data items. •Explore. When the visualization enables users to explore a different subset of data items. •Reconfigure. When the visualization allows users to watch different perspectives by rearranging the data. Visualization strategy. Novais et al. introduce the following strategy schema [10]: •Temporal Snapshot (TS). It represents a snapshot of the software at a specific point in time. •Temporal Overview (TO). It displays all of the values of the development of a particular metric for all of the software entities being examined. This complicates the presentation since we must represent several entities (e.g. software modules) in a single visual picture, as well as the temporal evolution of one or more of their properties (e.g. size). •Temporal Accumulative Snapshot (TA). It considers the absolute value of changes to study the evolution of the program. It is commonly employed in the study of software churn [25]. •Differential Relative (DR). It expresses the increase or decrease in a software module’s property. If a software feature f1 increased its scattering by 2 points, but another feature f2 increased it by 4, then the DR representation should graphically show that f2 grew more than f1. In a similar way, decrements should be presented. •Differential Absolute (DA). When the definition of growth or reduction does not apply, the absolute strategy comes in handy. It is focused on recognizing and representing discrete events, such as entity properties that have appeared or vanished from one software version to the next. 3.3. Dimension 3: Research maturity No matter the analysis nor the visualization, an important insight is the extent of the evidence of the intervention utility. To this end, we resort to two existing classifications for Research Type [26] and Evaluation Method [27]. In addition, we consider the Publication Venue as an additional proxy of the research maturity. Research type. Wieringa et al. introduce the following research type schema [26]: •Evaluation research. Explains a method of evaluating research that involves assessing how a solution works in practice or comparing it to other solutions while highlighting positive and negative aspects. It is more comprehensive than validation and is frequently performed in an industrial context. •Validation research. Describes the validation of a research that has not yet been implemented in practice, for example, by an experiment, the performance of tests, lab experiments, and so on. It usually comes after a proposal for a solution. •Proposal of solutions. Without a thorough validation, these works suggest a solution approach and argue for its utility. The approach must be new, or at the very least a considerable improvement on one that already exists. A proof-of-concept might be presented in the form of a simple example, a compelling argument, or another method. •Philosophical papers. These publications propose a new way of thinking, a new conceptual framework, etc. Information and Software Technology 154 (2023) 107084 9 R. Medeiros et al. Fig. 10. Primary sources selection process. •Personal experience papers. Represent the authors’ experience, generally in practice, with a particular method, technology, or other device. These papers are frequently authored by industry professionals. Evaluation method. Stol and Fitzgerald introduce the following schema [27]: •Field Studies. Any research undertaken in a specific, real-world situation to explore a specific software engineering issue. •Field Experiments. Experimental studies conducted in a natural setting with a high degree of realism (similar to a field study), but this evaluation method reduces the level of realism compared to a field study because the researcher manipulates some properties or variables in the research setting in order to observe an effect of some kind. •Experimental Simulations. In this kind of evaluations, the behaviors of actors (e.g., developers, users, or software systems) that a researcher aims to observe and measure in experimental simulations are natural, but the setting in which they occur has been specifically created for the purposes of the study—that is, the setting would not exist if the study did not exist. •Laboratory Experiments. Laboratory experiments differ from field experiments in that field experiments explore phenomena in their natural setting, whereas laboratory experiments are conducted in a controlled environment. •Judgment Studies. These studies entail obtaining empirical data from a group of people who are asked to assess or rate actions, reply to a researcher’s request or ‘stimulus’, or debate a particular topic of interest. •Sample Studies. These are studies that try to attain generalizability across a certain demographic or set of factors, such as software professionals, software systems, or development process artifacts. •Formal theory. This is a way of evaluation that aims for a high level of universality so that the resultant theory or framework may be used in a wide range of situations, even though most theories have boundaries beyond which they do not apply. •Computer Simulation. Studies that replicate a real-world phenomenon or setting using a computer. These kinds of studies are like virtual laboratory experiments where hypotheses can be tested. Publication venue. We distinguish between four different venue types: journals, conferences, workshops and symposiums. Information and Software Technology 154 (2023) 107084 10 R. Medeiros et al. 4. Research design The goal of this study is to review the visualization interventions that have been proposed to assist in the maintenance and evolution of VRSs. To this end, we follow the recommendations presented in [6]. This section describes the plan for each step of the study, including the research questions, academic databases and search strategy, and inclusion/exclusion criteria. 4.1. Research questions RQ1: What sort of analysis is being conducted to assess VRS evolution?. RQ1 takes an analysis perspective. In accordance with the Goal-Question-Metric (GQM) model, an analysis is conducted by an agent with a purpose through a set of questions grounded on a collection of metrics [28]. Accordingly, we could characterize studies based on the agent (‘target role’), the purpose (‘evolution scenario’), and the support provided by the tool to conduct the analysis (‘added-value’). We can then refine RQ1 as follows: •RQ1.1: Which are the target roles involved? •RQ1.2: What are the evolution goals being pursued? •RQ1.3: What is the added-value provided by the tool? RQ2: What sort of visualization are displayed?. RQ2 takes a tool perspective. The question intends to find out what kind of visualization tools are used on VRS evolution and what are the elements used to build them. This question can be refined as follows •RQ2.1: What are the elements being visualized? •RQ2.2: Which visualization techniques are used? •RQ2.3: Which sort of visual interactions are catered for? •RQ2.4: What visualization strategies are used? •RQ2.5: What data sources are used to build which visualizations? •RQ2.6: To what extent is VRS scalability considered in each evolution scenario? RQ3: What is the research maturity of the reported interventions? No matter the analysis nor the visualization, this question sheds light on the degree of maturity of the primary sources. 4.2. Search strategy Search string. Along the lines of Kitchenham et al. [6], we broke down the question into individual facets, namely, VRS as ‘the population’, evolution as ‘the aim’, and ‘Visual Analytics’ (i.e., visualization) as ‘the intervention’. We searched for synonyms. In addition, we noticed that ‘maintenance’ and ‘evolution’ are terms that are arbitrarily interchanged. We also noticed that visualization is often expressed in other forms such as ‘visual’ or ‘visualizing’. Hence, we included visual with a wildcard *. What was more elusive was the notion of VRS as it encompasses areas such as product lines, variability management and configurable systems. This resulted on the following search string: (‘‘evol*’’ OR ‘‘maint*’’) AND (‘‘visual*’’) AND (‘‘product line’’ OR ‘‘feature model’’ OR ‘‘variability management’’ OR ‘‘configurable system’’) Academic databases. Primary sources were obtained by querying the following databases: Scopus,2ACM Digital Library,3IEEEXplore,4ScienceDirect5and Wiley Online Library6(see the top of Fig. 10). We did not query Google Scholar and SpringerLink as they lacked the advanced query functionality we needed. 2https://www.scopus.com/. 3https://dl.acm.org/. 4https://ieeexplore.ieee.org/. 5https://www.sciencedirect.com/search. 6https://onlinelibrary.wiley.com/. Fig. 11. Distribution of the primary sources per year. Search scope. Search was conducted upon papers’ title, abstract, and keywords. To check the validity of the search string, we manually double checked a few articles. 4.3. Protocol Fig. 10 outlines the process. We ran the query on the 17th of September of 2021. We obtained an initial set of 234 papers from which 24 belonged to the papers manually selected by the authors. At the onset, we removed duplicated papers (40) and secondary studies. This resulted in 189 candidate papers (‘Candidate papers’). We reserved secondary studies for a snowballing phase. Candidate papers (189) went next through a filtering process. The title and the abstract were carefully read by two researchers (the first two authors). In case of doubt, the introduction and conclusion, or even the full content of the paper was also read. Each paper was filtered based on the inclusion and exclusion criteria described in Table 1. Snowballing was conducted for the remaining 35 papers that provided an addition of 6 papers. This gives a total of 41 primary sources. To minimize potential biases on the selection process, two other researchers (the last two authors) were involved when doubts or disagreements arouse between the other two researchers. This happened for 15 papers. Finally, two researchers (the two first authors) carried out the data extraction process. Each paper classification was surveyed by at least another researcher. Fig. 11 shows the distribution of the primary sources publication per year. It suggests a sustained effort throughout with an average of 2.75 publications per year. 5. Data extraction & Paper classification Data from primary sources was extracted along the template in Table 2. The table indicates the Research Questions, the items extracted to answer them, the domain (‘type of data’) and cardinality (‘multivalued’) of these items and the section where the information item is described. Besides the literature-backed classification schemas introduced in Section 3, new ones were derived from the data itself (ad-hoc schemas). That is, if an existing scheme did not exist, thematic analysis was conducted by grounding classification themes on quotes extracted directly from the primary sources. Next, we review these ad-hoc schemas. RQ1.1: Target role. Regarding ‘target role’, a main distinction in SPLs is between domain engineers and application engineers. As the review scope is not limited to SPLs but includes VRSs in general, we conducted a fine-grained categorization of ‘target roles’ through a coding step (i.e., grouping equivalents, synonyms, and assigning ids). We identified six categories, namely: •Application Engineers: The people who develop variants out of existing code. Primary sources refer to them as either ‘application engineer’ or ‘product developer’. •Domain Engineers: The people who develop code to be reused. Primary sources refer to them as ‘SPL architects’, ‘domain engineers’ or ‘product line maintainer’. Information and Software Technology 154 (2023) 107084 11 R. Medeiros et al. Table 1 Exclusion and inclusion criteria. Type Criteria Inclusion criteria - Focus on topic – The study focuses on evolution or maintenance. –The study focuses on variant-rich systems. –The study proposes/uses some kind of graphical visualization technique to support evolution activities Inclusion criteria - Quality guarantee – The study is peer reviewed Exclusion criteria - Type of paper – The study is in a language other than English –The study is a previous version of another study in the review –The work is a Secondary study such as a literature review Exclusion criteria - Characteristics of the approach – The visualization is not tool supported. We only focus on visualizations that have an implementation behind them, e.g., we exclude manually created graphs Table 2 Data extraction template. RQ Extracted item Type of data Multi-valued Described at RQ1 Analysis scenarios Yes Yes Section 3.1 Target roles Ad-hoc schema Yes Section 5 Added-value Ad-hoc schema Yes Section 5 RQ2 Visualized elements Ad-hoc schema Yes Section 5 Visualization technique Ad-hoc schema Yes Section 5 Visualization interaction Existing schema Yes Section 3.2 Visualization strategies Existing schema Yes Section 3.2 Data sources Ad-hoc schema Yes Section 5 RQ3 Research type Existing schema No Section 3.3 Evaluation method Existing schema Yes Section 3.3 Tool availability Free text No Section 5 Metadata General meta-data Free text No Section 5 •Software Engineers: This code is used when authors do not make a clear distinction between Application Engineers and Domain Engineers. This code abstracts from quotes referring to either ‘engineers’ or ‘code developers’. •Requirement Engineers: This code is used when the primary sources explicitly mention that the tool targets people in charge of requirement engineering activity. Primary sources refer to them as ‘requirement engineer’ or ‘scope analyst’. •Testers: Here, authors mention that the tool targets testers or it is used for testing purposes. •Project Managers: The people in charge of the planning, scheduling, budgeting, execution, and delivery of a software. This code abstracts from quotes referring to either ‘managers’, ‘change control board’ or ‘product line managers’. RQ1.3: Added-value. This facet accounts for the degree of support provided by the visualization from merely displaying the raw data to engineering analysis workflows as part of the tool offerings. Codes follow: •Raw. When raw elements are used in the visual analysis (e.g., codebase), at best, tool interactions are limited to provide some sort of data aggregation, •Metric. When metrics are computed from the raw elements as proxies for the analysis (e.g., number of methods, feature cohesion), •Analysis. When some sort of analysis is being conducted out of the metric (e.g., data-flow analysis, feature-location analysis). RQ2.1: Visualized elements. Visualized elements may coincide with the data sources. Sometimes however, elements such as features are abstracted out of the code without the existence of a feature model. RQ2.3: Visualization technique. Examples of visualization techniques include nodelink diagrams, heatmaps, bubble charts, etc. RQ2.5: Data sources. This includes [10], namely: ‘source code’ (e.g., the SPL platform codebase or the variant codebase, if available), ‘feature model’,‘variant configuration model’, other model-based artifacts (e.g., UML notation such as architectural-component diagrams or statetransition diagrams), ‘traceability links’ (feature-code traces, requirementvariant traces), ‘project documentation’ (requirements or design documents), ‘VCS meta-data’ (commits, issues), and the ‘run-time context’ (logs). RQ3.3. Tool availability/replicability. We checked the availability of the visualization tool (source code or installer package). First, we searched for an URL in the paper that would take us to the tool. If broken, we contacted the authors asking for an active link. If the link was not present, we googled it using the tool name as the search string. General meta-data. We recovered the venues and publication years for the primary sources. 6. Results for RQ1: What sort of analysis is being conducted to assess VRS evolution? An analysis is conducted by an agent (‘target role’) with a purpose (‘evolution scenario’) assisted by a visualization tool (‘added-value’). We delve into each of these characteristics. 6.1. Target roles Table 3 outlines the extent each target role is being referred to in the primary sources. ‘Domain engineers’ and ‘application engineers’ get the lion’s share: 31 of the primary sources target these roles. For the rest of the primary sources, either this distinction did not apply (e.g., clone & own where a reusability platform is not available, and hence the ‘target role’ of ‘domain engineer’ does not yet exist) or they tackle marginal cases (e.g., testing, management). 6.2. Evolution scenarios Table 4 summarizes the results. We can observe how visualization support develops in tandem with the VRS field. At the onset, VRSs depart from clone & own to an extractive approach on the search for systematic reuse. Here, ‘Feature Identification and Location’ in existing variants was (and still is) a main endeavour (10 papers). As the area matures, and VRS develops towards configuration systems and SPLs, Information and Software Technology 154 (2023) 107084 12 R. Medeiros et al. Table 3 Target roles numbers. Target role Number Primary sources Application engineers 16 [S10,S9,S11,S7,S2,S12,S13,S14,S5,S15,S3,S16,S17,S6,S18,S19] Domain engineers 15 [S9,S7,S20,S4,S21,S22,S8,S23,S19,S24,S25,S26,S15,S25,S27] Software engineers 9 [S28,S29,S30,S31,S32,S33,S34,S35,S1] Testers 3 [S36,S37,S16] Requirements engineers 2 [S38,S39] Project managers 2 [S40,S41] Table 4 Evolution scenarios numbers. Evolution scenario Number Primary sources Feature Identification and Location (FIL) 10 [S28,S31,S32,S35,S3,S22,S38,S17,S39,S13] Variant Integration (VI) 8 [S26,S30,S2,S33,S21,S14,S41,S23] Variant Synchronization (VS) 6 [S10,S27,S12,S11,S37,S1] Functional Testing (FT) 6 [S30,S7,S36,S29,S16,S19] Transformations (TR) 5 [S25,S11,S24,S20,S6] Co-Evolution of Problem and Solution Spaces (CPS) 4 [S9,S7,S40,S18] Constraints Extraction (CE) 4 [S9,S4,S34,S15] Architecture Recovery (AR) 2 [S5,S19] Analysis of Non-Functional Properties (ANF) 1 [S8] ‘Variant Synchronization’ ‘Variant Integration’, and ‘Transformations’ are moved to the spot (19 papers). Visualization wise, efforts are made to understand and support evolution rather than on testing the preservation of certain (non)functional properties. Interestingly, it is not uncommon for primary sources to investigate multiple scenarios: [S30] (VI and FT), [S11] (VS and TR) or [S19] (FT and AR). 6.3. Added-value Table 5 displays the results: 8 papers contribute through elaborated mining approaches to provide the raw data; 14 papers place a visualization on top of the raw data to highlight different metrics; finally, 18 papers contribute by providing some sort of analysis on top of the visualization. This distribution 8-14-18 is akin to the role played by each added-value: one sort of raw data might be aggregated along different metrics which, in turn, might allow for different analysis perspectives. We complement this report by listing the metrics and analysis types at play: code churn [S23,S2,S7], feature scattering [S26,S7,S31], number of feature dependencies [S9,S18], code delta metric [S23], feature cohesion metric [S6], feature size [S26], feature tangling [S7], feature model metrics [S20], number of method overload [S35], runtime non-functional metrics [S8], significance criteria [S10], t-wise coverage [S36] and weighted hamming distance [S16]. On the other hand, analysis strategies include, formal concept analysis [S21,S38], nway analysis [S5,S17,S3,S14], behavior analysis and comparison [S12], bug prevalence analysis [S37], clone detection analysis [S39], dataflow analysis [S34], feature location analysis [S32,S13], MOEA Analysis [S25], NLP analysis [S22], pair-wise feature relations [S4], PLC project analysis [S28], scope analysis [S41], and variant similarity analysis [S33,S1,S39]. With the exception of [S23,S39,S7,S26], papers focus on either a single metric or a single analysis strategy. This pattern is common in the early stages of an area subject to analysis (as it is the case of VRS evolution). Rationales are many-fold: mining techniques are not yet mature; lack of evidence about the appropriate metrics; and the scarcity of real scenarios where to validate the analysis strategies. As the area matures, we expect more hybrid approaches where different metrics and strategies are combined to tackle more realistic settings. 7. Results for RQ2: What sort of visualization are displayed? 7.1. Visualized elements (RQ2.1) Table 6 displays the results: ‘features’,‘variants’, and ‘implementation assets’ (e.g., source code classes) are the most popular. These elements seem to satisfy the analysis needs of the most investigated scenarios (see Section 6.2), i.e., ‘Variant Synchronization’ and ‘Feature Identification and Location’. 7.2. Visualization techniques (RQ2.2) Table 7 summarizes the results: ‘nodelink diagram’ (e.g. Figs. 6 and 4), and ‘tabular/matrix’ (e.g., Fig. 8) are by far the most popular visualization techniques. This coincides with visualizations for oneoff development [10]. In total, we found 11 different visualization techniques. We note that 16 primary sources implemented more than one: ‘heatmap’ and ‘enriched software diagram’ (see Fig. 9) [S9]; ‘tabular/matrix’ and ‘enriched software diagram’ [S26]; ‘tabular/matrix’ and ‘line chart’ [S11]; ‘alluvial diagram’ and ‘bar chart’ (see Fig. 2) [S2]; ‘tabular/matrix’ and ‘bar chart’ [S31,S14]; ‘tabular/matrix’,‘bubble chart’, ‘treemap’ and ‘heatmap’ [S36]; ‘tabular/matrix’ and ‘bubble chart’ (see Fig. 8) [S8]; ‘nodelink diagram’ and ‘tabular/matrix’ [S34,S40]; ‘bar chart’,‘heatmap’ and ‘treemap’ (see Fig. 3) [S3]; ‘tabular/matrix’ and ‘heatmap’ [S16]; ‘bar chart’ and ‘nodelink diagram’ (see Fig. 1) [S1]; ‘nodelink diagram’ and ‘word cloud’ (see Fig. 7) [S7]; ‘tabular/matrix’, ‘bar chart’ and ‘enriched software diagram’ (see Fig. 5) [S5]; or ‘alluvial diagram’ and ‘treemap’ [S23]. 7.3. Visualization interactions (RQ2.3 & RQ2.6) Table 8 outlines the results. Nearly half of the primary sources (44%) do not support any kind of interaction but just provide static diagrams. For those supporting some interaction, ‘abstract/elaborate’ is the most common. This may be due to the different levels of abstraction present in VRSs (e.g. variants, features, source code). Some of the primary sources start the analysis at the variant level and next, drilldown to either the features or the source code [S30,S2,S1,S39,S23,S12, S5,S3]. Both ‘filter’ and ‘select’ mechanisms are also widely used, akin to the large codebase of VRSs. Finally, ‘connect’ and ‘explore’ are the least popular which could be due to the yet poorly elaborated analysis frameworks for managing VRS evolution. 7.4. Visualization strategies (RQ2.4) Table 9 summarizes the results: the vast majority of primary sources use a ‘temporal snapshot’ strategy. Tentatively, researchers are also considering other scenarios, e.g., ‘temporal overview’ and ‘differential relative’. Exceptionally, three primary sources support various strategies: [S9,S11] both support ‘temporal snapshot’ and ‘differential relative’, while [S1] supports ‘temporal snapshot’ and ‘temporal overview’. The ability to change the strategy based on user-demand seems to be a desirable property for future visualization tools. Information and Software Technology 154 (2023) 107084 19 R. Medeiros et al. [22] R. Capilla, J. Bosch, P. Trinidad, A.R. Cortés, M. Hinchey, An overview of dynamic software product line architectures and techniques: Observations from research and industry, J. Syst. Softw. 91 (2014) 3–23, http://dx.doi.org/10.1016/ j.jss.2013.12.038. [23] D. Strüber, M. Mukelabai, J. Krüger, S. Fischer, L. Linsbauer, J. Martinez, T. Berger, Facing the truth: benchmarking the techniques for the evolution of variant-rich systems, in: T. Berger, P. Collet, L. Duchien, T. Fogdal, P. Heymans, T. Kehrer, J. Martinez, R. Mazo, L. Montalvillo, C. Salinesi, X. Tërnava, T. Thüm, T. Ziadi (Eds.), Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume a, Paris, France, September 9-13, 2019, ACM, 2019, pp. 26:1–26:12, http://dx.doi.org/10.1145/3336294.3336302. [24] J.S. Yi, Y. ah Kang, J.T. Stasko, J.A. Jacko, Toward a deeper understanding of the role of interaction in information visualization, IEEE Trans. Vis. Comput. Graph. 13 (6) (2007) 1224–1231, http://dx.doi.org/10.1109/TVCG.2007.70515. [25] T.L. Graves, A.F. Karr, J.S. Marron, H.P. Siy, Predicting fault incidence using software change history, IEEE Trans. Softw. Eng. 26 (7) (2000) 653–661, http: //dx.doi.org/10.1109/32.859533. [26] R.J. Wieringa, N.A.M. Maiden, N.R. Mead, C. Rolland, Requirements engineering paper classification and evaluation criteria: a proposal and a discussion, Requir. Eng. 11 (1) (2006) 102–107, http://dx.doi.org/10.1007/s00766-005-0021-6. [27] K. Stol, B. Fitzgerald, The ABC of software engineering research, ACM Trans. Softw. Eng. Methodol. 27 (3) (2018) 11:1–11:51, http://dx.doi.org/10.1145/ 3241743. [28] V.R. Basili, G. Caldiera, H.D. Rombach, The goal question metric approach, in: Encyclopedia of Software Engineering, Vol. 2, 1994, pp. 528–532, URL: http://www.csri.utoronto.ca/~sme/CSC444F/handouts/GQM-paper.pdf. [29] A. Lex, N. Gehlenborg, H. Strobelt, R. Vuillemot, H. Pfister, UpSet: Visualization of intersecting sets, IEEE Trans. Vis. Comput. Graphics (InfoVis) 20 (12) (2014) 1983–1992, http://dx.doi.org/10.1109/TVCG.2014.2346248. [30] D.A. Keim, F. Mansmann, J. Schneidewind, J. Thomas, H. Ziegler, Visual analytics: Scope and challenges, in: S.J. Simoff, M.H. Böhlen, A. Mazeika (Eds.), Visual Data Mining - Theory, Techniques and Tools for Visual Analytics, in: Lecture Notes in Computer Science, vol. 4404, Springer, 2008, pp. 76–90, http://dx.doi.org/10.1007/978-3-540-71080-6_6. [31] M. Mukelabai, D. Nesic, S. Maro, T. Berger, J. Steghöfer, Tackling combinatorial explosion: a study of industrial needs and practices for analyzing highly configurable systems, in: M. Huchard, C. Kästner, G. Fraser (Eds.), Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Montpellier, France, September 3-7, 2018, ACM, 2018, pp. 155–166, http://dx.doi.org/10.1145/3238147.3238201. [32] Z.T. Sinkala, M. Blom, S. Herold, A mapping study of software architecture recovery for software product lines, in: J. Pérez, R. Mirandola, H. Chen (Eds.), Proceedings of the 12th European Conference on Software Architecture: Companion Proceedings, ECSA 2018, Madrid, Spain, September 24-28, 2018, ACM, 2018, pp. 49:1–49:7, http://dx.doi.org/10.1145/3241403.3241454. [33] A. Krause, M. Hansen, W. Hasselbring, Live visualization of dynamic software cities with heat map overlays, in: Working Conference on Software Visualization, VISSOFT 2021, Luxembourg, September 27-28, 2021, IEEE, 2021, pp. 125–129, http://dx.doi.org/10.1109/VISSOFT52517.2021.00024. [34] A. Schreiber, L. Nafeie, A. Baranowski, P. Seipel, M. Misiak, Visualization of software architectures in virtual reality and augmented reality, in: 2019 IEEE Aerospace Conference, 2019, pp. 1–12, http://dx.doi.org/10.1109/AERO.2019. 8742198. [35] S. Romano, N. Capece, U. Erra, G. Scanniello, M. Lanza, On the use of virtual reality in software visualization: The case of the city metaphor, Inf. Softw. Technol. 114 (2019) 92–106, http://dx.doi.org/10.1016/j.infsof.2019.06.007. [36] B. Fitzgerald, K. Stol, Continuous software engineering: A roadmap and agenda, J. Syst. Softw. 123 (2017) 176–189, http://dx.doi.org/10.1016/j.jss.2015.06.063. [37] A. Kleebaum, B. Paech, J.O. Johanssen, B. Bruegge, Continuous rationale visualization, in: Working Conference on Software Visualization, VISSOFT 2021, Luxembourg, September 27-28, 2021, IEEE, 2021, pp. 33–43, http://dx.doi.org/ 10.1109/VISSOFT52517.2021.00013. [38] J.O. Johanssen, A. Kleebaum, B. Paech, B. Bruegge, Continuous software engineering and its support by usage and decision knowledge: An interview study with practitioners, J. Softw. Evol. Process. 31 (5) (2019) http://dx.doi.org/10. 1002/smr.2169. [39] N.M. Tiwari, G. Upadhyaya, H.A. Nguyen, H. Rajan, Candoia: a platform for building and sharing mining software repositories tools as apps, in: J.M. González-Barahona, A. Hindle, L. Tan (Eds.), Proceedings of the 14th International Conference on Mining Software Repositories, MSR 2017, Buenos Aires, Argentina, May 20-28, 2017, IEEE Computer Society, 2017, pp. 53–63, http: //dx.doi.org/10.1109/MSR.2017.56. [40] Y. Ma, T. Dey, C. Bogart, S. Amreen, M. Valiev, A. Tutko, D. Kennard, R. Zaretzki, A. Mockus, World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data, Empir. Softw. Eng. 26 (2) (2021) 22, http://dx.doi.org/10.1007/s10664-020-09905-9. [41] L. Merino, M. Ghafari, C. Anslow, O. Nierstrasz, A systematic literature review of software visualization evaluation, J. Syst. Softw. 144 (2018) 165–180, http: //dx.doi.org/10.1016/j.jss.2018.06.027. [42] O. Díaz, L. Montalvillo, R. Medeiros, M. Azanza, T. Fogdal, Visualizing the customization endeavor in product-based-evolving software product lines: a case of action design research, Empir. Softw. Eng. 27 (3) (2022) 75, http://dx.doi. org/10.1007/s10664-021-10101-6. [43] X. Zhou, Y. Jin, H. Zhang, S. Li, X. Huang, A map of threats to validity of systematic literature reviews in software engineering, in: 2016 23rd Asia-Pacific Software Engineering Conference, APSEC, 2016, pp. 153–160, http://dx.doi.org/ 10.1109/APSEC.2016.031. References (Primary sources) [S1] V.L. Tenev, S. Duszynski, M. Becker, Variant analysis: Set-based similarity visualization for cloned software systems, in: M.H. ter Beek, W. Cazzola, O. Díaz, M.L. Rosa, R.E. Lopez-Herrejon, T. Thüm, J. Troya, A.R. Cortés, D. Benavides (Eds.), Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume B, Sevilla, Spain, September 25-29, 2017, ACM, 2017, pp. 22–27, http://dx.doi.org/10.1145/3109729. 3109753. [S2] L. Montalvillo, O. Díaz, T. Fogdal, Reducing coordination overhead in spls: peering in on peers, in: T. Berger, P. Borba, G. Botterweck, T. Männistö, D. Benavides, S. Nadi, T. Kehrer, R. Rabiser, C. Elsner, M. Mukelabai (Eds.), Proceedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10-14, 2018, ACM, 2018, pp. 110–120, http://dx.doi.org/10.1145/3233027.3233041. [S3] J. Martinez, A.K. Thurimella, Collaboration and source code driven bottomup product line engineering, in: E.S. de Almeida, C. Schwanninger, D. Benavides (Eds.), 16th International Software Product Line Conference, SPLC ’12, Salvador, Brazil - September 2-7, 2012, Vol. 2, ACM, 2012, pp. 196–200, http://dx.doi.org/10.1145/2364412.2364445. [S4] J. Martinez, T. Ziadi, R. Mazo, T.F. Bissyandé, J. Klein, Y.L. Traon, Feature relations graphs: A visualisation paradigm for feature constraints in software product lines, in: H.A. Sahraoui, A. Zaidman, B. Sharif (Eds.), Second IEEE Working Conference on Software Visualization, VISSOFT 2014, Victoria, BC, Canada, September 29-30, 2014, IEEE Computer Society, 2014, pp. 50–59, http://dx.doi.org/10.1109/VISSOFT.2014.18. [S5] S. Duszynski, J. Knodel, M. Naab, D. Hein, C. Schitter, Variant comparison - a technique for visualizing software variants, in: A.E. Hassan, A. Zaidman, M.D. Penta (Eds.), WCRE 2008, Proceedings of the 15th Working Conference on Reverse Engineering, Antwerp, Belgium, October 15-18, 2008, IEEE Computer Society, 2008, pp. 229–233, http://dx.doi.org/10.1109/WCRE.2008.22. [S6] S. Apel, D. Beyer, Feature cohesion in software product lines: an exploratory study, in: R.N. Taylor, H.C. Gall, N. Medvidovic (Eds.), Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu, HI, USA, May 21-28, 2011, ACM, 2011, pp. 421–430, http://dx.doi. org/10.1145/1985793.1985851. [S7] O. Díaz, R. Medeiros, L. Montalvillo, Change analysis of #if-def blocks with featurecloud, in: C. Cetina, O. Díaz, L. Duchien, M. Huchard, R. Rabiser, C. Salinesi, C. Seidl, X. Tërnava, L. Teixeira, T. Thüm, T. Ziadi (Eds.), Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume B, Paris, France, September 9-13, 2019, ACM, 2019, pp. 59:1–59:4, http://dx.doi.org/10.1145/3307630.3342386. [S8] A. Murashkin, M. Antkiewicz, D. Rayside, K. Czarnecki, Visualization and exploration of optimal variants in product line engineering, in: T. Kishi, S. Jarzabek, S. Gnesi (Eds.), 17th International Software Product Line Conference, SPLC 2013, Tokyo, Japan - August 26-30, 2013, ACM, 2013, pp. 111–115, http://dx.doi.org/10.1145/2491627.2491647. [S9] K. Feichtinger, D. Hinterreiter, L. Linsbauer, H. Prähofer, P. Grünbacher, Guiding feature model evolution by lifting code-level dependencies, J. Comput. Lang. 63 (2021) 101034, http://dx.doi.org/10.1016/j.cola.2021.101034. [S10] Y. Kim, H. Jeon, Y. Kim, Y. Ki, H. Song, J. Seo, Visualization support for multi-criteria decision making in software issue propagation, in: 14th IEEE Pacific Visualization Symposium, PacificVis 2021, Tianjin, China, April 1921, 2021, IEEE, 2021, pp. 81–85, http://dx.doi.org/10.1109/PacificVis52677. 2021.00018. [S11] S. Entekhabi, A. Solback, J. Steghöfer, T. Berger, Visualization of feature locations with the tool featuredashboard, in: C. Cetina, O. Díaz, L. Duchien, M. Huchard, R. Rabiser, C. Salinesi, C. Seidl, X. Tërnava, L. Teixeira, T. Thüm, T. Ziadi (Eds.), Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume B, Paris, France, September 9-13, 2019, ACM, 2019, pp. 55:1–55:4, http://dx.doi.org/10.1145/3307630. 3342392. [S12] A. Zamansky, I. Reinhartz-Berger, Visualizing code variabilities for supporting reuse decisions, in: X. Franch, M. Snoeck, R.S.S. Guizzardi, I. Jureta (Eds.), Proceedings of the 5th Symposium on Conceptual Modeling Education and the 2nd International iStar Teaching Workshop Co-Located with the 36th International Conference on Conceptual Modeling (ER 2017), Valencia, Spain, November 6-9, 2017, in: CEUR Workshop Proceedings, vol. 1954, CEUR-WS.org, 2017, pp. 25–34, URL http://ceur-ws.org/Vol-1954/SCME_2017_paper_5.pdf. Information and Software Technology 154 (2023) 107084 20 R. Medeiros et al. [S13] S. Illescas, R.E. Lopez-Herrejon, A. Egyed, Towards visualization of feature interactions in software product lines, in: B. Sharif, C. Parnin, J. Fabry (Eds.), 2016 IEEE Working Conference on Software Visualization, VISSOFT 2016, Raleigh, NC, USA, October 3-4, 2016, IEEE Computer Society, 2016, pp. 46–50, http://dx.doi.org/10.1109/VISSOFT.2016.16. [S14] S. Duszynski, J. Knodel, M. Becker, Analyzing the source code of multiple software variants for reuse potential, in: M. Pinzger, D. Poshyvanyk, J. Buckley (Eds.), 18th Working Conference on Reverse Engineering, WCRE 2011, Limerick, Ireland, October 17-20, 2011, IEEE Computer Society, 2011, pp. 303–307, http://dx.doi.org/10.1109/WCRE.2011.44. [S15] M. Sinnema, S. Deelstra, P. Hoekstra, The COVAMOF derivation process, in: M. Morisio (Ed.), Reuse of Off-the-Shelf Components, 9th International Conference on Software Reuse, ICSR 2006, Turin, Italy, June 12-15, 2006 Proceedings, in: Lecture Notes in Computer Science, vol. 4039, Springer, 2006, pp. 101–114, http://dx.doi.org/10.1007/11763864_8. [S16] M. Al-Hajjaji, M. Schulze, U. Ryssel, Similarity analysis of product-line variants, in: T. Berger, P. Borba, G. Botterweck, T. Männistö, D. Benavides, S. Nadi, T. Kehrer, R. Rabiser, C. Elsner, M. Mukelabai (Eds.), Proceedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10-14, 2018, ACM, 2018, pp. 226–235, http://dx.doi.org/10.1145/3233027.3233044. [S17] J. Martinez, T. Ziadi, J. Klein, Y.L. Traon, Identifying and visualising commonality and variability in model variants, in: J. Cabot, J. Rubin (Eds.), Modelling Foundations and Applications - 10th European Conference, ECMFA@STAF 2014, York, UK, July 21-25, 2014. Proceedings, in: Lecture Notes in Computer Science, vol. 8569, Springer, 2014, pp. 117–131, http://dx.doi.org/10.1007/ 978-3-319-09195-2_8. [S18] D. Sellier, M. Mannion, Visualizing product line requirement selection decisions, in: Software Product Lines, 11th International Conference, SPLC 2007, Kyoto, Japan, September 10-14, 2007, Proceedings. Second Volume (Workshops), Kindai Kagaku Sha Co. Ltd., Tokyo, Japan, 2007, pp. 109–118. [S19] N. Anquetil, U. Kulesza, R. Mitschke, A. Moreira, J. Royer, A. Rummler, A. Sousa, A model-driven traceability framework for software product lines, Softw. Syst. Model. 9 (4) (2010) 427–451, http://dx.doi.org/10.1007/s10270009-0120-9. [S20] S. Urli, A. Bergel, M. Blay-Fornarino, P. Collet, S. Mosser, A visual support for decomposing complex feature models, in: 3rd IEEE Working Conference on Software Visualization, VISSOFT 2015, Bremen, Germany, September 27-28, 2015, IEEE Computer Society, 2015, pp. 76–85, http://dx.doi.org/10.1109/ VISSOFT.2015.7332417. [S21] T. Huysegoms, M. Snoeck, G. Dedene, A. Goderis, F. Stumpe, Visualizing variability management in requirements engineering through formal concept analysis, Proc. Technol. 9 (2013) 189–199, http://dx.doi.org/10.1016/j.protcy. 2013.12.021, cENTERIS 2013 - Conference on ENTERprise Information Systems / ProjMAN 2013 - International Conference on Project MANagement/ HCIST 2013 - International Conference on Health and Social Care Information Systems and Technologies. [S22] J. Martinez, T. Ziadi, T.F. Bissyandé, J. Klein, Y.L. Traon, Name suggestions during feature identification: the variclouds approach, in: H. Mei (Ed.), Proceedings of the 20th International Systems and Software Product Line Conference, SPLC 2016, Beijing, China, September 16-23, 2016, ACM, 2016, pp. 119–123, http://dx.doi.org/10.1145/2934466.2934480. [S23] L. Montalvillo, O. Díaz, M. Azanza, Visualizing product customization efforts for spotting SPL reuse opportunities, in: M.H. ter Beek, W. Cazzola, O. Díaz, M.L. Rosa, R.E. Lopez-Herrejon, T. Thüm, J. Troya, A.R. Cortés, D. Benavides (Eds.), Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume B, Sevilla, Spain, September 25-29, 2017, ACM, 2017, pp. 73–80, http://dx.doi.org/10.1145/3109729.3109737. [S24] G. Gharibi, Y. Zheng, Archfeature: integrating features into product line architecture, in: S. Ossowski (Ed.), Proceedings of the 31st Annual ACM Symposium on Applied Computing, Pisa, Italy, April 4-8, 2016, ACM, 2016, pp. 1302–1308, http://dx.doi.org/10.1145/2851613.2851764. [S25] W.M. Freire, M. Massago, A.C. Zavadski, A.M.M.M. Amaral, T.E. Colanzi, Opla-tool v2.0: a tool for product line architecture design optimization, in: E. Cavalcante, F. Dantas, T. Batista (Eds.), SBES ’20: 34th Brazilian Symposium on Software Engineering, Natal, Brazil, October 19-23, 2020, ACM, 2020, pp. 818–823, http://dx.doi.org/10.1145/3422392.3422498. [S26] D. Hinterreiter, P. Grünbacher, H. Prähofer, Visualizing feature-level evolution in product lines: A research preview, in: N.H. Madhavji, L. Pasquale, A. Ferrari, S. Gnesi (Eds.), Requirements Engineering: Foundation for Software Quality - 26th International Working Conference, REFSQ 2020, Pisa, Italy, March 24-27, 2020, Proceedings [REFSQ 2020 was Postponed], in: Lecture Notes in Computer Science, vol. 12045, Springer, 2020, pp. 300–306, http: //dx.doi.org/10.1007/978-3-030-44429-7_21. [S27] J. Fischer, B. Vogel-Heuser, E. Estévez-Estévez, M. Male, Varapp: Variant management app for IEC 61131-3 compliant legacy software, in: IEEE Conference on Industrial Cyberphysical Systems, ICPS 2020, Tampere, Finland, June 1012, 2020, IEEE, 2020, pp. 269–276, http://dx.doi.org/10.1109/ICPS48405. 2020.9274774. [S28] A. Schlie, K. Rosiak, O. Urbaniak, I. Schaefer, B. Vogel-Heuser, Analyzing variability in automation software with the variability analysis toolkit, in: C. Cetina, O. Díaz, L. Duchien, M. Huchard, R. Rabiser, C. Salinesi, C. Seidl, X. Tërnava, L. Teixeira, T. Thüm, T. Ziadi (Eds.), Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume B, Paris, France, September 9-13, 2019, ACM, 2019, pp. 89:1–89:8, http://dx.doi.org/10.1145/3307630.3342408. [S29] D.H. Carmo, S.T. Carvalho, L.G.P. Murta, O. Loques, Runtime monitoring and auditing of self-adaptive systems (S), in: The 25th International Conference on Software Engineering and Knowledge Engineering, Boston, MA, USA, June 27-29, 2013, Knowledge Systems Institute Graduate School, 2013, pp. 731–736. [S30] B. Duhoux, K. Mens, B. Dumas, H.S. Leung, A context and feature visualisation tool for a feature-based context-oriented programming language, in: A. Etien (Ed.), Proceedings of the Seminar Series on Advanced Techniques & Tools for Software Evolution (SATTOSE 2019), Bolzano, Italy, July 8-10 Day, 2019, in: CEUR Workshop Proceedings, vol. 2510, CEUR-WS.org, 2019, URL http: //ceur-ws.org/Vol-2510/sattose2019_paper_7.pdf. [S31] J. Krüger, L. Nell, W. Fenske, G. Saake, T. Leich, Finding lost features in cloned systems, in: M.H. ter Beek, W. Cazzola, O. Díaz, M.L. Rosa, R.E. LopezHerrejon, T. Thüm, J. Troya, A.R. Cortés, D. Benavides (Eds.), Proceedings of the 21st International Systems and Software Product Line Conference, SPLC 2017, Volume B, Sevilla, Spain, September 25-29, 2017, ACM, 2017, pp. 65–72, http://dx.doi.org/10.1145/3109729.3109736. [S32] B. Andam, A. Burger, T. Berger, M.R.V. Chaudron, Florida: Feature location dashboard for extracting and visualizing feature traces, in: M.H. ter Beek, N. Siegmund, I. Schaefer (Eds.), Proceedings of the Eleventh International Workshop on Variability Modelling of Software-Intensive Systems, VaMoS 2017, Eindhoven, NetherlandS, February 1-3, 2017, ACM, 2017, pp. 100–107, http://dx.doi.org/10.1145/3023956.3023967. [S33] T. Kanda, T. Ishio, K. Inoue, Approximating the evolution history of software from source code, IEICE Trans. Inf. Syst. 98-D (6) (2015) 1185–1193, http: //dx.doi.org/10.1587/transinf.2014EDP7286. [S34] M. Ribeiro, T. Tolêdo, J. Winther, C. Brabrand, P. Borba, Emergo: a tool for improving maintainability of preprocessor-based product lines, in: R. Hirschfeld, E. Tanter, K.J. Sullivan, R.P. Gabriel (Eds.), Companion Volume of the 11th International Conference on Aspect-Oriented Software Development, AOSD 2012, Potsdam, Germany, March 25-30, 2012, ACM, 2012, pp. 23–26, http://dx.doi.org/10.1145/2162110.2162128. [S35] X. Tërnava, J. Mortara, P. Collet, Identifying and visualizing variability in object-oriented variability-rich systems, in: T. Berger, P. Collet, L. Duchien, T. Fogdal, P. Heymans, T. Kehrer, J. Martinez, R. Mazo, L. Montalvillo, C. Salinesi, X. Tërnava, T. Thüm, T. Ziadi (Eds.), Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume a, Paris, France, September 9-13, 2019, ACM, 2019, pp. 32:1–32:13, http://dx.doi.org/10.1145/3336294.3336311. [S36] R.E. Lopez-Herrejon, A. Egyed, Towards interactive visualization support for pairwise testing software product lines, in: A.C. Telea, A. Kerren, A. Marcus (Eds.), 2013 First IEEE Working Conference on Software Visualization (VISSOFT), Eindhoven, the Netherlands, September 27-28, 2013, IEEE Computer Society, 2013, pp. 1–4, http://dx.doi.org/10.1109/VISSOFT.2013.6650543. [S37] T.H.B. de Oliveira, M. Becker, E.Y. Nakagawa, Supporting the analysis of bug prevalence in software product lines with product genealogy, in: E.S. de Almeida, C. Schwanninger, D. Benavides (Eds.), 16th International Software Product Line Conference, SPLC ’12, Salvador, Brazil - September 2-7, 2012 Volume 1, ACM, 2012, pp. 181–185, http://dx.doi.org/10.1145/2362536. 2362561. [S38] N. Niu, S.M. Easterbrook, Concept analysis for product line requirements, in: K.J. Sullivan, A. Moreira, C. Schwanninger, J. Gray (Eds.), Proceedings of the 8th International Conference on Aspect-Oriented Software Development, AOSD 2009, Charlottesville, Virginia, USA, March 2-6, 2009, ACM, 2009, pp. 137–148, http://dx.doi.org/10.1145/1509239.1509259. [S39] T.F.L. de Medeiros, E.S. de Almeida, S.R. de Lemos Meira, Codescoping: A source code based tool to software product lines scoping, in: V. Cortellessa, H. Muccini, O. Demirörs (Eds.), 38th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2012, Cesme, Izmir, Turkey, September 5-8, 2012, IEEE Computer Society, 2012, pp. 101–104, http://dx. doi.org/10.1109/SEAA.2012.70. [S40] R. Rabiser, D. Dhungana, W. Heider, P. Grünbacher, Flexibility and end-user support in model-based product line tools, in: 35th Euromicro Conference on Software Engineering and Advanced Applications, SEAA 2009, Patras, Greece, August 27-29, 2009, Proceedings, IEEE Computer Society, 2009, pp. 508–511, http://dx.doi.org/10.1109/SEAA.2009.13. [S41] K. Wnuk, B. Regnell, L. Karlsson, What happened to our features? visualization and understanding of scope change dynamics in a large-scale industrial setting, in: RE 2009, 17th IEEE International Requirements Engineering Conference, Atlanta, Georgia, USA, August 31 - September 4, 2009, IEEE Computer Society, 2009, pp. 89–98, http://dx.doi.org/10.1109/RE.2009.32. 98 APPENDIX A. PUBLISHED OR ACCEPTED PUBLICATIONS A.2 Visualizing the customization endeavor in productbased-evolving software product lines: a case of action design research Title: Visualizing the customization endeavor in productbased-evolving software product lines: a case of action design research Authors: O. Díaz, L. Montalvillo, R. Medeiros, M. Azanza, T. Fogdal Journal: Empirical Software Engineering Publisher: Springer DOI: https://doi.org/10.1007/s10664-021-10101-6 Year: 2022 Source of impact: JCR Category: COMPUTER SCIENCE, SOFTWARE ENGINEERING Impact factor: 4.1 (Q2) Position: 29/108 Vol.:(0123456789) Empirical Software Engineering (2022) 27: 75 https://doi.org/10.1007/s10664-021-10101-6 1 3 Visualizing thecustomization endeavor inproduct‑based‑evolving software product lines: acase ofaction design research OscarDíaz1· LeticiaMontalvillo1· RaulMedeiros1· MaiderAzanza1 · ThomasFogdal2 Accepted: 6 December 2021 / Published online: 18 March 2022 © The Author(s) 2022, corrected publication 2022 Abstract Software Product Lines (SPLs) aim at systematically reusing software assets, and deriving products (a.k.a., variants) out of those assets. However, it is not always possible to handle SPL evolution directly through these reusable assets. Time-to-market pressure, expedited bug fixes, or product specifics lead to the evolution to first happen at the product level, and to be later merged back into the SPL platform where the core assets reside. This is referred to as product-based evolution. In this scenario, deciding when and what should go into the next SPL release is far from trivial. Distinct questions arise. How much effort are developers spending on product customization? Which are the most customized core assets? To which extent is the core asset code being reused for a given product? We refer to this endeavor as Customization Analysis, i.e., understanding the functional increments in adjusting products from the last SPL platform release. The scale of the SPLs’ code-base calls for customization analysis to be conducted through Visual Analytics tools. This work addresses the design principles for such tools through a joint effort between academia and industry, specifically, Danfoss Drives, a company division in charge of the P400 SPL. Accordingly, we adopt an Action Design Research approach where answers are sought by interacting with the practitioners in the studied situations. We contribute by providing informed goals for customization analysis as well as an intervention in terms of a visual analytics tool. We conclude by discussing to what extent this experience can be generalized to product-based evolving SPL organizations other than Danfoss Drives. Keywords SPL evolution· Visual analytics· Code diffing· Action design research Communicated by: Philippe Collet, Sarah Nadi, Christoph Seidl, and Leopoldo Motta Teixeira This article belongs to the Topical Collection: Open Science This article belongs to the Topical Collection: Software Product Lines and Variability-rich Systems (SPLC) This paper has been awarded the Empirical Software Engineering (EMSE) open science badge * Maider Azanza maider[email protected] Extended author information available on the last page of the article Empirical Software Engineering (2022) 27: 75 1 3 75 Page 2 of 44 1 Introduction A Software Product Line (SPL) is a set of software-intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way (Clements and Northrop 2002). Prescription is given in terms of both developing for reuse and developing with reuse. This results in the interplay of two distinct activities: (1) Domain Engineering (DE), where the scope and variability of the system is defined and reusable assets are developed to conform the SPL platform; and (2) Application Engineering (AE), where products (a.k.a. variants) are derived by selecting and resolving variability, i.e., by variability configuration (Pohl etal. 2005). To obtain the full benefits, AE should be limited to configuration without further modifying the derived products (Krueger 2006). However, this is not always possible. Time-to-market pressure, expedited bug fixes, or product specifics lead to development to first happen at the product level and to be later merged back into the SPL platform. Indeed, product-based evolution (a.k.a. variant-based evolution) is being acknowledged as a major strategy to drive the evolution of a product line (Krüger etal. 2020). Kruger etal. observe that this way of working switches the typical order of DE and AE, i.e., developers first customize core assets to product requirements (AE) to next consolidate the product upgrades into the platform (DE). The bottom line is that AE is no longer limited to mere configuration but it might also involve development. We refer to this practice as ‘product customization’ as opposed to ‘product configuration’, where AE is limited to selecting the features to be exhibited by the product with no development involved. Product-based SPL evolution might result in tensions between the quality and reuse effectiveness required by DE, and the time-to-market and customer pressure faced by AE. SPL managers need to analyze whether efforts invested in product customization pay off in terms of better SPL scoping (i.e., deciding on the products, technical areas, and functionalities that a product line should support). Questions might arise about how much effort is dedicated to product customization; which are the most customized core assets; or to which extent is core-asset code being reused in a given product. We refer to this endeavor as Customization Analysis, i.e., understanding the functional increments in adjusting products from the last SPL platform release. Customization analysis can help SPL managers in different ways: (1) identify which product developments should be promoted to the core-asset base, (2) spot overloaded product teams with a heavy customization duty, (3) uncover indecisive or exploratory design where developers’ hesitation is reflected in the volatility of the code, or (4), spot eventual merging issues when product developments are merged back into the core-asset base. Being able to quantify the customization activity might help managers take informed decisions about both the SPL’s stability and the rearrangement of the SPL task force. Despite the importance of these decisions, support for customization analysis is rather limited. We abound into this practice by making a case for the use of alluvial diagrams (a.k.a. Sankey diagrams) as a suitable visualization for customization analysis. Specifically, two research questions are tackled: – RQ1 (Problem Space): Which are the information needs for customization analysis? How much time is needed to fulfill these information needs? – RQ2 (Solution Space): Might alluvial diagrams be useful for supporting customization analysis visualization? Empirical Software Engineering (2022) 27: 75 1 3 Page 3 of 44 75 We address these questions through a joint effort between academia and industry, specifically, Danfoss Drives, a company division in charge of P400, a product line for the software embedded in frequency converters (Fogdal etal. 2016). On these grounds, we adopt an Action Design Research (ADR) approach (Sein et al. 2011). The objective of action research is to solve or at least explain, the problems of an analyzed situation (i.e., product customization) by the researchers interacting with the participants (i.e., P400 engineers) in the studied situations (i.e., customization analysis). Action research becomes action design research if the problem is tackled through an artifact (i.e., a new method or a new tool) whose design and evaluation is conducted within the organization (Sein etal. 2011). Accordingly, we develop a Visual Analytics tool for customization analysis. Akin to the ADR principles, we distill this experience in some general outcomes. By doing so, we aim at contributing to the two previous RQs: – RQ1. We characterize customization analysis through a Goal-Question-Metric (GQM) model. Based on Danfoss Drives’ practices, we introduce a set of analysis questions, and rate the importance and required time to answer such questions (Section4). – RQ2. We make a case for alluvial diagrams as an effective way to realize the previous GQM model. We flesh out this case through CustomDIFF, a visual analytics tool that uses Git as the SPL code repository, and pure::variants as the variability manager (Section6). – RQ2. We distill general principles from first empirical evidence through an expert evaluation at Danfoss Drives (Section8). This article is an extension of a REVE’17 workshop paper (Montalvillo etal. 2017), which is extended in three major aspects. First, we considerably expand the description about the phenomenon at hand: product customization. Second, it is methodologically sounder since it sticks to the Design Activity Framework proposed for visual analytics, and provides empirical evaluation. Third, this paper focuses on annotation-based SPLs using pure::variants, whereas the REVE paper tackled component-based SPLs using FeatureHouse. By conducting this research in close collaboration with an industrial partner, we hope to facilitate a valuable transfer into practice. We start by introducing the phenomenon under study: product customization. 2 The Phenomenon: Product Customization Code development during AE (i.e., product customization) has been documented in distinct scenarios: to meet products’ deadline and budget (Deelstra etal. 2005; Jensen 2007; Schackmann and Lichter 2006), to expedite bug fixes (Fogdal etal. 2016), to speed up unexpected functional changes in customer needs (Nagamine et al. 2016; Carbon et al. 2008; Iida et al. 2016), to decrease reusable asset complexity for single-product needs (Deelstra etal. 2005; Kircher and Hofman 2012; Bartholdt and Becker 2011), and, finally, in the transition to a fully-configured SPL, product specifics might remain in product teams (Kodama etal. 2014; Takebe etal. 2009). Even fully-configurable SPLs might reach the scale and complexity that make maintenance in short time spans infeasible. Hence, when organizations are faced with urgent customer or market requests, product-specific adjustments are realized first in the product to be later propagated to the SPL platform (Deelstra etal. 2005). Empirical Software Engineering (2022) 27: 75 1 3 75 Page 4 of 44 Implementation wise, this is normally realized through a version control system. Here, a master branch keeps the core assets, i.e., those assets which are in the SPL baseline. During product customization, it is allowed for this master to be branched off to support (urgent) product specifics. This leads to a grow-and-prune branching model (Faust and Verhoef 2003). This model states that, during a time-lapse, quick reaction to changes often requires copying and specializing in the branches (grow) to be later cleaned up by refactoring and merged back into the master (prune). The notion of time-lapse is important since product branches are not intended to live for a long time but are to be periodically merged back into the master. In this way, product needs drive the evolution of the SPL. As an example, consider the WeatherStationSPL, an SPL for building web-based applications for weather stations. We borrow this example from the instructional material provided by pure::variants.1 Let us suppose that this SPL holds a baseline release Baselinev1.0, that accounts for seven features, clustered around three parent features,2 namely: Sensors that encompasses AirPressure, Temperature, and WindSpeed; Warnings that comprises Gale and Heat; and Languages that is the parent feature for English and German. Let us imagine that some urgent customization needs arise that prevent developers from waiting until the next platform release. This causes Baseline-1.0 to be branched off into three product branches: PR-NewYork, PR-Paris, and PR-Berlin (see Fig.1). This unleashes the grow-and-prune process: – Grow-Customization. AE adjusts core assets to product specifics, potentially evolving the product through different versions (e.g., PR-Paris-v1.0). – Prune-Consolidation. Eventually, DE gets integration requests from AE. Missing to reintegrate these product variants back to the master risks product-line engineering becoming clone&own development (Krüger and Berger 2020). It might happen that the customization might be of interest but not yet mature enough to be offered to products other than the product that hosts it. Here, DE developers might integrate the adjustment into the platform using spurious features, i.e., transient features that might be exclusively used for the driving product. This allows AE developers to generate again Fig. 1 WeatherStationSPL branching model: the master branch holds the core assets from where SPL products are branched off 1 http://www.pure-systems.com/products/pure-variants-9.html 2 In pure::variants terminology, a parent feature serves to aggregate semantically related features Empirical Software Engineering (2022) 27: 75 1 3 Page 5 of 44 75 this product using traditional configuration mechanisms.3 This guarantees that the platform is under control, and application projects may maintain their independence by providing product-specific artifacts as new features. After a modification has been evaluated, a judgment will be made as to whether the change should be applied to additional products and so incorporated into the core assets (Fogdal etal. 2016). For Danfoss Drives, decision is taken by the Change Control Board that includes domain experts but also application engineers who were involved in the customization in the first place. The choice is not always easy, as evidenced by the so-called configuration oscillation phenomenon (Faust and Verhoef 2003). Here, engineers can be hesitant about which new assets should be promoted to the platform. The bottom line is that the Change Control Board regulates the tempos of this growand-prune process. In this setting, customization analysis helps the Change Control Board identify if the customization endeavor matches the plan by correlating where the customization is occurring with where the customization is planned to occur. A lot of customization happening in unplanned areas may be an indicator of problematic code, that is, a code that continuously requires patching. In general, customization analysis might help foresee distinct problematic scenarios, namely: – product units that are under heavy customer pressure in terms of specifics that need to be accounted for. This might lead to reinforcing these units or to rescheduling customer petitions, – stable features that are passed (almost) untouched when deployed in distinct products. This showcases mature features. This scenario might point to needing to consider deviating programming resources to other units, – unstable features which might require additional refactoring, and testing efforts to cope with emerging scenarios coming from customer petitions. This might lead to reinforcing the domain engineers in charge. Tracking these scenarios directly from code is time consuming and error prone. Tools are needed that abstract from the code-base. We tackle this challenge as a joint effort between academia and industry. On these premises, we decided to follow Action Design Research as our research methodology (Sein etal. 2011). 3 A Brief onAction Design Research Sein etal. define Action Design Research (ADR) as a research method for generating prescriptive design knowledge through building and evaluating IT ensemble artifacts in an organizational setting (Sein etal. 2011,p. 40). A key insight is the role played by the organization (i.e., Danfoss) in driving and shaping the design knowledge that ends up being instantiated in the IT artifact (i.e., the CustomDIFF visualization tool). Hence, the term ensemble artifact denotes the artifact taking its full meaning in conjunction with the context where it displays its utility (i.e., CustomDIFF reflects the practice and brings utility to Danfoss). Therefore, ADR conceives artifact design as a result of a 3 Notice that spurious features are transient. They are in trial for a period of time until they are finally integrated (and hence available for other products of the SPL) or discarded. Empirical Software Engineering (2022) 27: 75 1 3 75 Page 6 of 44 researcher-practitioner collaboration within an organization. Figure2 reproduces ADR stages and principles (Sein etal. 2011). Problem Formulation. The first stage is triggered by a problem encountered in practice or predicted by researchers. It serves as a catalyst for developing a research strategy. This stage draws on two principles: Practice-Inspired Research and Theory-Ingrained Artifact (Sein etal. 2011). The former emphasizes viewing organization problems as knowledge-creation opportunities. The second principle highlights that the intervention (e.g., the IT artifact) is to be informed by theories, existing knowledge that grounds design decisions. Building, Intervention, and Evaluation. This stage builds upon the problem framing and theoretical premises adopted in stage one. These premises provide a platform for generating the initial design of the IT artifact. From here on, the IT artifact is further shaped by organizational use and subsequent design cycles (Sein etal. 2011). Or using Sein etal’s principles: reciprocal shaping (i.e., the IT artifact and the organization feedback each other: prototypes serve to profile the interpretation of the organizational environment that help a better fit in subsequent versions), mutually influential roles (i.e., researchers and practitioners bring complementary insights), and authentic and concurrent evaluation (i.e., authenticity is a more crucial element for ADR than controlled conditions, thus assessment should take place within the company and throughout the research). Fig. 2 The ADR Method: the first three stages conform an iterative cycle where tasks are intermingled till distilled into the final learnings at the end of the project (taken from Sein etal. 2011). Arrows stand for influential flows Empirical Software Engineering (2022) 27: 75 1 3 Page 7 of 44 75 Reflection and Learning. ADR involves more than merely solving a problem to an organization. To guarantee that contributions to knowledge are made, conscious reflection on the problem framing, theories adopted, and the emerging IT artifact are critical. The principle is termed as guided emergence where ‘emergence’ captures this notion of unanticipated consequences that arise during the intervention in the organization and to which researchers should be sensitive to (Sein etal. 2011). Formulation of Learning. At this point, we reach an artifact that brings with it some premises about the problem framing and the organization setting (i.e., an ensemble artifact). It represents a solution to a problem. Both can be generalized. Sein etal. suggest three levels for this effort: (1) generalization of the problem instance, (2) generalization of the solution instance, and (3) generalization of mechanisms through design principles. Design principles abstract away from the specific IT implementation into the abstract mechanisms that brought the utility, and underlie the solution. The rest of the paper is structured along these stages. 4 Problem Formulation Problem formulation draws on two principles: practice-inspired research and theoryingrained artifact. The former reflects the premise that IT artifacts are ensembles shaped by the organizational context (Sein etal. 2011). Therefore, it is most important to describe the organization whose practices and characteristics will inform the artifact design. The second principle highlights that ADR does not stop at identifying a problem, but provides an intervention to alleviate the problem. This intervention should be informed by existing theories. This section sets this research’s problem along with these two principles. 4.1 Practice‑Inspired Research This research builds upon the fifteen-year experience of Danfoss Drives, a company division in charge of P400, a product line for the software embedded in frequency converters (Fogdal etal. 2016). P400 is a member of the SPLC’s Product Line Hall of Fame.4 Table1 characterizes P400 in terms three main contextual dimensions: the stakeholders, the complexity of the task (i.e., customization analysis), and the setting that frames the problem (i.e., the technical infrastructure and features of the SPL that might be relevant for the problem at hand). Danfoss follows an Annotation-Based Approach to P400 definition. This implies that variations are supported through pre-compilation directives. A directive states when a block code is to be included in the final product based on the presence or absence of a feature selection at configuration time. In pure::variants, these directives start with an opening directive //PV:IFCOND and end with a closing directive //PV:ENDCOND. Figure3 shows an example. The snippet illustrates two variation points, i.e., VP-1 and VP-2 that correspond to two ifdef blocks. In the example, VP-1 comprises lines 24 to 49, whereas VP-2 expands along lines 30 to 46. Danfoss follows a Product-Based Approach to P400 evolution This implies that DE and AE co-exist not only at the onset but throughout the SPL life-cycle. We aim at 4 https://splc.net/fame.html Empirical Software Engineering (2022) 27: 75 1 3 75 Page 14 of 44 Table 2 Validating the GQM. Rating the importance of information needs using a LIKERT scale from 1 (Not Important) to 5 (Very Important) Question ID Likert scale Avg. 12345 ---------- Holistic perspective: I consider important to know ... H1 ... which features are (not) being customized by products 0 1 3 3 1 3.5 H2 ... which products are customizing (no) features 0 1 2 3 2 3.75 H3 ... how much effort (i.e., code churn) has been spent on customizing each feature, in total, no matter the product 214012.87 H4 ... how much effort (i.e., code churn) each product is spending on customizing each feature 2 2 2 1 1 2.62 H5 ...which files are being changed on each product to customize each feature 2 0 2 4 0 3 ---------- Feature perspective: for feature F1, I consider important to know ... F1 ... which products are customizing it 0 0 1 3 3 4 F2 ... which has been the total customization diffing (i.e., code-churn aggregate) 2 1 3 1 1 2.75 F3 ... which has been the total customization diffing (i.e., code-churn aggregate), broken down by product 2 1 3 1 1 2.75 F4 ... which files have been changed 1 2 1 1 3 3.37 ---------- Product perspective: for product P1, I consider important to know ... P1 ... which features are customized 0 1 1 2 4 4.12 P2 ... how much effort (i.e. code churn) has been spent on customization, no matter the feature 2 1 2 2 1 2.87 P3 ... how much effort (i.e. code churn) has been spent on customizing each feature 2 1 3 1 1 2.75 P4 ... which files have been changed 2 1 3 1 1 2.75 Empirical Software Engineering (2022) 27: 75 1 3 Page 15 of 44 75 the Change Control Board perspective (viewpoint). Our goal is to trace how much effort is being put into product customization. Question. Questions are posed to characterize the way the assessment/achievement of a specific goal is going to be performed based on some characterized model (Basili etal. 1994). For this model, feature and product emerge as natural constructs. Indeed, SPL releases, product configuration, programming assignments, or work positions are commonly described in terms of features and products. Using the notions of ‘feature’ and ‘product’, we can arrange the questions along with three distinct scopes (refer to Table2 for the full list of questions): – the feature-focused scope, e.g., for a given feature, which products are customizing it, – the product-focused scope, e.g., for a given product, which features have been customized; if a product has made no change, it is not considered. – the holistic scope, e.g., which is the whole customizationeffort from the last platform release, Measurement. Once the questions have been developed, we proceed by associating the questions with appropriate metrics. Factors to be considered for this selection include (Basili etal. 1994): the quantity and quality of the existing data, and the maturity of the measurements. On these grounds, we resort to code churn, a well-established metric to assess evolution in one-off development (Ajila and Dumitrescu 2007; Hall and Munson 2000). Code churn is a popular measure to inform about the rate at which the code evolves (Khoshgoftaar and Szabo 1994). The churn for a file over a specified period is computed as: [LinesAdded]+[LinesDeleted]+[LinesModified] (Faragó et al. 2015). Traditionally, this is achieved by diffing files (Schulze etal. 2016): DIFF(aFilet1,aFilet0) displays the code churn for file aFile in the interval [t0,t1]. In single-off development, code churn is being extensively used for defect prediction (Nagappan and Ball 2005), assess code erosion (Ohlsson etal. 1999), or detect code volatility (Faragó etal. 2015). Moving back to SPLs, code churn could also be a valuable metric for customization analysis: DIFF(aFilet1,aFilet0) where t0 stands from the time where the product branch is generated out of the master’s, and t1 corresponds to a time before the product branch is merged back to the master’s. Though the notion of churn might be appropriate, the object of the churn is not. Rather than files, the GQM’s questions are posed in terms of ‘features’ and ‘products’. Needed are mechanisms that move from file-based diffing to higher abstraction terms. Specifically, consider DIFF, a function that returns the code churn for aFile as it is kept in the master branch (i.e., aFile.core) vs. how it has been customized in a product branch (aFile.aProduct). Rather than DIFF(aFile.core, aFile.aProduct, we long for DIFF(aFeature.core, aFeature.aProduct) utilities that abstract out dozens of DIFF(aFile, aFile) for those aFiles that realize aFeature as it is being customized for aProduct. That is, DIFF(aFeature.core, aFeature.aProduct) outputs the code-churn aggregate of customizing aFeature for aProduct, no matter the files the aFeature is spread over. On these grounds, we could define three sort of aggregates to measure the customization diffing (see Fig.8), namely: (1) feature-focused DIFF (i.e., DIFF(aFeature, *)) where aFeature stands for the set of aFile realizing this feature, and ‘*’ indicates no matter the product branch; (2) product-focused DIFF (i.e., DIFF(*, aProduct)) where aProduct stands for the set of aFiles realizing this product, and ‘*’ indicates no matter the feature; and (3), holistic DIFF (i.e., DIFF(*, *)) where ‘*’ corresponds for all aFile no matter the feature Empirical Software Engineering (2022) 27: 75 1 3 75 Page 16 of 44 nor the product. These aggregates compute the sum of code churn for the underlying involved files. 5.2 Evaluation ADR promotes continuous assessment of the intervention (principle 5: authentic and concurrent evaluation) (Sein etal. 2011). The intervention is not only the IT artifact but also the design principles this artifact realizes. This includes the information needs the IT artifact aims to fulfill. This section provides a first validation of the GQM model. Participants. Participants were selected who had at least one-year experience on Danfoss Drives. Among the eight participants that took part in this evaluation, three had 10 years of experience while the other five accounted for 9, 7, 6, 3, and 1 year of experience, respectively. Process. A questionnaire was prepared to assess the relevance of the above mentioned GQM model in terms of the importance given to the different questions raised by it. Face validity was conducted, that is, we checked whether the questionnaire seemed to correspond to the GQM model. A first draft of the questionnaire was prepared by the authors and next, the questionnaire was delivered to this paper’s Danfoss Drives’ author. Based on his comments, some amendments were added to clarify the purpose and adapt the terminology to that of the practitioners. Next, practitioners were requested to indicate the importance given to each question using a LIKERT scale from 1 (Not Important) to 5 (Very Important). Results. Table2 shows the results. Some conclusions can be drawn from them: – Perspective wise, both feature-focused and product-focused are similarly rated. The highest rated questions are “for the feature F1, which products are customizing it” (avg. 4) and its sibling, i.e., “for the product P1, which are the features being customized” (avg. 4.12). The lowest rated questions correspond to the fine-grained holistic perspective. This might be due to this information being better captured at either the feature perspective or the product perspective. – Aggregation-level wise, quite an unexpected result: intermediary aggregates were not prioritized. When pondering analysis needs, participants seem to favor either a general overview of the customization diffing or, instead, being able to dive into the specifics. The bottom line is that all questions rate above 2.5, with four questions going beyond 3.5 (i.e., above “Moderately Important”). This provides first evidence about the interest in customization analysis. Yet, Visual Analytics does not stop at identifying the right data. Making better decisions also depends on the ability to understand and communicate adequately the measurement to the decision-makers. This moves us to the next cycle. 6 Building, Intervention, andEvaluation: Cycle 2 The basic idea is to visually represent the information, allowing humans to interact directly with such information, to gain insight, to draw conclusions, and to ultimately make better decisions. Figure9 depicts the main ingredients of Visual Analytics tools (Reddivari etal. 2014). For our purposes, the Data is kept in a Git repository. This data corresponds to code files whose versions are arranged w.r.t the Git version control system model. This code is Empirical Software Engineering (2022) 27: 75 1 3 Page 17 of 44 75 pre-processed to distill the metrics to be used to assess the customization effort (i.e., code churn). As a result, code churn is obtained and described along with a Model that serves the information needs which were identified in the GQM analysis. This Model ends up being realized through a database that acts as the back-end for a Web application that supports the Visualization strategy. Main architectural components include: a mining component that extracts data from Git repositories; a database that holds mined data along with the data Model; and a front-end component that queries the database and display results using appropriate visualization means. This section instantiates this architecture for CustomDIFF, a web-based tool for customization analysis. CustomDIFF uses pure::variants (Pure-Systems 2018) and Git as the variability management tool and version control system, respectively. Three additional resources are made available: – an interactive online version of CustomDIFF which the reader is encouraged to access: http://customdiff.onekin.org/, – a video describing CustomDIFF (6’): https://vimeo.com/577936099, – a Zenodo replication package for the CustomDIFF implementation: https://doi. org/10.5281/zenodo.5728000 6.1 Building & Intervention 6.1.1 The Model Dimensional Modeling is a data structure technique that is specifically designed for data storage when used for decision taking. This implies a sharp distinction between two sort of tables: “fact” and “dimensions” (see Fig.10). The “fact” table collects the events of the phenomenon under study. For our purposes, the phenomenon under study is “product customization”. We consider an event of this phenomenon to occur when it happens the consecutive deletion/addition of code churn for a file. Each code churn gives rise Fig. 9 Visual analytics main interactions (adapted from Reddivari etal. (2014)) Empirical Software Engineering (2022) 27: 75 1 3 75 Page 18 of 44 to a fact tuple. Fact properties include: the number of lines added, the number of lines deleted or the actual code being changed (custom_diff) in a customization episode. Facts are the finest grain of the customization endeavor. Obtaining a higher perspective of the customization endeavor requires these facts to be aggregated along different dimensions: “the what” (i.e., the variation point being affected by the customization), “the where” (i.e., the product in which the customization occurred), “the when” (i.e., the time of the product release), and “the who” (i.e., developers who conducted the customization). Figure10 displays the database schema. This schema will be instantiated from the codebase of the SPL at hand. The SPL codebase is held in a Git repository where the master branch contains the core-asset baseline while products branch off the master. This moves us to the next sub-section. 6.1.2 The Data In data warehousing, ETL, which stands for extract, transform and load, is a data integration process whereby data is extracted from data sources (that are not optimized for analytics), and moved to a central host (which is). In our setting, ETL mines the SPL’s Git repository, runs the corresponding diffing, and populates the tables. Specifically, facts (i.e., customization_fact tuples) are obtained by working out a DIFF between the namesake artifacts of the master branch and the product branches. For the DIFF depicted in Fig.4, two facts would be obtained. Fact #1 would stand for the changes introduced in line 29, whereas Fact #2 would correspond to those changes introduced in lines 34–38. Details about the mining algorithm are provided in the Appendix. A related approach is described by Zhang etal. (2013). Fig. 10 CustomDIFF’s data model Empirical Software Engineering (2022) 27: 75 1 3 Page 19 of 44 75 6.1.3 The Visualization This section makes a case for the use of alluvial diagrams (a.k.a. Sankey diagrams) as a suitable visualization for customization analysis, measured in terms code churn from diffing. Broadly, we advocate to ideate the notion of customization diffing as an energy flow. The metaphor of energy flow is used in domains familiar with the dynamics of mass flow (e.g., energy, capital, transportation). Examples can be found for resource use (Lupton and Allwood 2017), energy flow (Schmidt 2008; Subramanyam etal. 2015) or material flows (Schmidt 2008), to name a few. For our purposes, the flow stands for the customization energy spent on attending products’ specifics. This flow moves from products to features, and from there, it percolates down to the files where these features end up being realized. For flow display, alluvial diagrams are commonly used. We resort to alluvial diagrams to factor out the customization effort. Figure11 shows the case for the WeatherStationSPL. On the left, flow emitters: the products where AE spends energy to account for their specifics. On the right, flow sink: the features where the energy produced during AE ends up. Arrows connect products with those features being the subject of a customization intervention. Nodes are the issuers of the flow. The larger the node area, the larger the flow potential. If the node stands for a product, then the node width corresponds to the effort in customizing this product. If the node stands for a feature, then the node width corresponds to the effort in adjusting this feature no matter the product. On the other hand, Fig. 11 Customization diffing are visualized as flows from products (as customization emitters) to features (as customization sinks). For example: ProductLondon-v1.0 requires customization for three features: WindSpeed, AirPressure & Temperature Empirical Software Engineering (2022) 27: 75 1 3 75 Page 20 of 44 arrows stand for the flow between nodes. If the arrow connects product P to feature F, then the arrow’s width captures the effort involved in adjusting F to Pdemands. Alluvials pursue a prompt answer to the GQM’s questions. Take Fig.11 as an example. Which features are being customized? AirPressure, Temperature, WindSpeed, Gale and German. Which products are customizing features? productBerlin, productLondon, etc. How much effort (i.e., code churn) has been spent on customizing the feature AirPressure? This is reflected in the node width; the tool also shows the customization diffing (i.e., 36 LOC) on a mouse over. How much effort has spent productParis in customizing AirPressure? This is reflected in the arrow width; mouse over to get 31 LOC. This anecdotal evaluation looks promising. Yet, there exists a first stumbling block: scalability. SPLs can hold hundreds of features that can be combined into thousands of products. Although not every feature/product is involved in each customization cycle, visualization might become cluttered for a larger number of features or products. Filters and grouping are the most common techniques to reduce the number of nodes. Filters limit the flow to either the products or the features that meet the filtering criteria. Grouping permits products/features to be grouped into clusters. Products could be grouped based on product units. A product unit might be in charge of one or several products. Grouping permits this effort to be visualized for the whole unit. Likewise, features might be grouped based on their parent features so that the effort is displayed for the whole set of child features. The grouping of nodes also implies a grouping of the adjacent edges. Figure12 shows groupings for the case of features. Now the flow concentrates along WeatherStationSPL’s parent features, i.e., Sensors, Languages, and Warnings. Fig. 12 Grouping. The Feature bar aggregates values by parent features. Likewise, the Product bar could also aggregate values by product units Empirical Software Engineering (2022) 27: 75 1 3 Page 21 of 44 75 6.2 Evaluation To assess the suitability of alluvial diagrams, informal demo sessions were conducted. Interestingly enough, during the demo sessions two sort of clusters, similar to those that emerged during the understand evaluation phase, started to surface: coarse-grained (preferred by the Change Control Board) and fine-grained (preferred by developers). At the onset, researchers were mainly concerned about an intervention for correlating where the customization was occurring with where the customization was planned to occur. Here, the stakeholders are SPL managers interested in tracing the balance between DE and AE. Yet, as the interaction with practitioners advanced, a new role started to emerge, i.e., developers, who were concerned not only about the high-level customization representation but also about the code behind it. Rationales rest on the grow-andprune model. Single-off development is more about growing than about pruning. That is, single-off development does not face later consolidation of upgrades into a common platform (i.e. the master). Applications evolve at their own pace. By contrast, product developers at SPLs are well aware that sooner or later their upgrades need to be merged back into the SPL platform. The bottom line is that developers’ concerns include not only customizing but also consolidating the upgrade. The alluvial diagram so far seems to be appropriate to capture the customization diffing but falls short to assess the consolidation effort. This requires moving down to code, to how features are fleshed out. 7 Building, Intervention, andEvaluation: Cycle 3 This section moves the developer perspective to the forefront. The alluvial diagram so far might be sufficient for managers to evaluate the effort trend in product customization (i.e., the GQM model in Fig.8). Yet, developers do not stop at the big picture. When it comes to calibrating the consolidation effort, developers might need to go down to how features are both spread along distinct classes (i.e., scattering) and mixed up with other features (i.e., tangling). 7.1 Building & Intervention CustomDIFF needs to cater for scattering and tangling. Nevertheless, the introduction of additional details might lead to cluttered interfaces. Hence, we should care not only for the visualization as such, but also for the interaction workflows that allow users to smoothly transit between the distinct perspectives. Accordingly, this subsection is structured along these two concerns: fine-grained visualization and workflows. 7.1.1 Extending The Visualization So far, alluvial diagrams stopped at products and features. This alluvial is now extended to account for scattering, tangling and code peering. Scattering. We resort to extending the scope of the alluvial flow by incorporating the package bar at both ends (see Fig.13). Details follow: Empirical Software Engineering (2022) 27: 75 1 3 75 Page 22 of 44 Fig. 13 Scattering visualization. Two additional Package bars are added at each extreme to expand the flow down to files. For example: customization on AirPressure (no matter the product) goes across files scale.js, sensors.js & settings.js Fig. 14 Tangling visualization for feature WindSpeed. Customizations involving #ifdef blocks that include WindSpeed in their directives are broken down into two FS sets: ‘FeatureSibling: WindSpeed’ and ‘FeatureSibling: AirPressure-WindSpeed’ Empirical Software Engineering (2022) 27: 75 1 3 Page 23 of 44 75 – On the right side, the package bar extends the flow from feature to file so that DE developers can have a piece-meal perspective on how their core assets are being adjusted. This permits an estimation of the eventual consolidation effort. For instance, the code churn for file scale.js is three times larger than the one for sensors.js. Accordingly, engineers might now foresee that merging scale.js might most likely demand more time than merging sensors.js. – On the left side, the package bar extends the flow from product to file so that product units can assess what other units are working on the same files. Thispermits promptly spotting overlapping risks. For instance, developers at productBerlinv1.0 and productSeville-v1.0 might probably arrange a meeting together to prevent redundant efforts. This in turn, might alleviate the burden at consolidation time by smoothing out their differences in advance. Tangling. We resort to grouping (see Fig.14). So far, the Feature bar captures the customization diffing along with the features being updated. Nodes might account for parent features that can be next broken down into their child features. We can further break down feature nodes into Feature Sibling (FS) Sets, a set of features that appear together in at least one ifdef directive, no matter the boolean expression that links them together. Figure14 shows the case of WindSpeed. This feature is broken down into two FS sets. First, the WindSpeed set which accounts for modified ifdef blocks with WindSpeed as the only feature in their pre-compilation directive. Second, the AirPressure - WindSpeed set that agglutinates ifdef blocks where these two features are referred to in their directives, no matter the boolean operator. The flow from productBerlin-v1.0 to AirPressure - WindSpeed depicts the tangled effort made to evolve the AirPressure and WindSpeed features. By zooming into FS sets, developers can have a first insight into the extent of tangling along with the customization diffing. Code Peering. CustomDIFF sticks to the traditional DIFF view for code peering. Yet, some subtle changes are needed. Traditionally, the DIFF context refers to the three nearest unchanged lines that precede and follow the change (see Fig.4). The context serves as a reference to locate the places of the changed lines. However, this might not be enough for variability-intensive code. Here, the code holds variability points where a pre-compilation directive regulates whether the block code is to be included in the final product. This pre-compilation directive is a main contextual clue to know which features are affected. Yet, these pre-compilation directives are right at the start of the block, potentially away from where the change has occurred, and hence, these directives might not show up in a traditional DIFF context. Figure 4 illustrates this situation. The change is located at line 29. However, the context (i.e., lines 26, 27, 28) does not include the pre-compilation directive. This deprives engineers from promptly knowing which features are affected. Therefore, feature-minded DIFF utilities should include pre-compilation directives as part of the DIFF context. Figure15(c) mimics the case of Fig.4, but now information about the pre-compilation directive is included into the hunk headings. The hunk corresponds to the changes in lines 43 and 50 under the scope of VP-1 (WindSpeed or AirPressure). In the case that VP-1 is nested within another variation point (e.g., VP-2), this is reflected in the hunk’s heading along the pattern : <enclosingVP > −− > nestedinto −− >< enclosedVP > . In short, pre-compilation directives should become a main ingredient of the DIFF context for variabilityintensive code. Empirical Software Engineering (2022) 27: 75 1 3 75 Page 30 of 44 Table 5 CustomDIFF’s perceived usefulness. LIKERT scale ranges from 1 (strongly disagree) to 7 (strongly agree) ID Item: P1 P2 P3 P4 P5 P6 Avg. St. Dev. U1 CustomDIFF was useful to determine which parent-features are not customized by the products 7 5 6 6 6 6 6 0.63 U2 CustomDIFF was useful to determine which parent-features is productBerlin customizing 7 4 6 6 6 6 5.83 0.98 U3 CustomDIFF was useful to determine how productParis is changing the implementation of the Sensors parent-feature 6 5 6 6 7 5 5.83 0.75 U4 CustomDIFF was useful to determine which products are customizing the AirPressure child-feature 6 4 6 6 6 6 5.67 0.82 U5 CustomDIFF was useful to determine how is each product customizing the code that realizes the AirPressure feature 6 4 6 6 7 4 5.5 1.22 U6 CustomDIFF was useful to determine how the implementation of productBerlin has evolved 6 4 6 6 6 4 5.33 1.03 Empirical Software Engineering (2022) 27: 75 1 3 Page 31 of 44 75 understandability. To reduce its influence, we carefully designed a running example that aimed at helping participants contextualize the different questions and ensuring a common understanding. External Validity tackles the representativeness of the study and the ability to generalize the conclusions beyond the scope of the study itself. Representativeness can be challenged by the participants or the evaluation tasks. As for the former, we resorted to Danfoss engineers. To account for different perspectives, we aimed at involving practitioners with experience in both DE and AE. As for the tasks, the WeatherStationSPL is a rather small SPL compared to industrial SPLs. It is explicitly designed to help new users understand the concepts of pure::variants and, thus, it is expected to be cleanly engineered (which is, quite safe to say, not the case for industrial SPLs). Yet, for an incipient tool such as CustomDIFF to be put to the test using industrial code is difficult. Moving to variability-intensive open applications could have been an option. This would have brought more realistic tasks, yet at the expense of reducing the realism of participants and environment. The latter are however pivotal in a decision-taking scenario such as customization analysis. 8 Formalization ofLearning In accordance with ADR, the situated learning from the project should be further developed into general solution concepts for a class of field problems (Sein etal. 2011). Sein etal. suggest three levels for this conceptual move: – generalization of the problem instance, i.e., to what extent is customization analysis a problem for organizations other than Danfoss Drives; – generalization of the solution instance, i.e., to what extent is CustomDIFF a solution to customization analysis; and – derivation of design principles, i.e., what sort of design knowledge can be distilled from the CustomDIFF experience that might inform other tool builders. The rest of this section tackles these questions. 8.1 Generalization oftheProblem Instance We tackle product customization at Danfoss Drives. This section elaborates on generalizing this experience to product-based evolving SPLs. Product customization challenges the traditional vision whereby new requirements are transmitted to domain engineering, features are built on the platform, and the product is then finally created. This conventional vision is increasingly called into question. Indeed, Krueger etal. report that an increasing number of companies and open-source projects add new variations or platform features using feature forks through version control systems (e.g., Git) (Krüger etal. 2020). Here, the platform, kept in the master branch, evolves through re-integrating these forks. If the master branch holds the SPL core assets, and the forks account for transient product enhancements, then we are talking about product-based evolving SPLs. Provided this way of SPL evolution, the question remains whether customization analysis is also a problem in these organizations. It could be argued that SPLs in an earlier stage of their life-cycle might be the ones subject to a larger customization endeavor while more mature SPLs can obtain most of their products out of the core assets with minimal Empirical Software Engineering (2022) 27: 75 1 3 75 Page 32 of 44 customization. That said, Danfoss Drives can be considered a mature SPL, and yet product customization is still very relevant. As usual in SPLs, the rationales might rest on scalability. SPLs handle a large number of products. Hence, perfective maintenance needs to scale up to the petitions of not one product’s stakeholders but a myriad of products. Timeliness might require the products’ perfective maintenance to develop in parallel. If this is your organization’s case, chances are you need to track how customization is conducted, i.e., customization analysis. 8.2 Generalization oftheSolution Instance This project develops CustomDIFF as an intervention for tackling customization analysis. We resort to visual analytics, using Git repositories as the data mines. This is not new. Distinct authors tackle both mining code repositories (we claim no contribution in this area) and software visualization for SPLs (our main contribution). The question arises about how alluvial diagrams compare w.r.t. other visualizations proposed in an SPL setting. LopezHerrejon et al. (2018) conduct a systematic survey where interventions are categorized based on the SPL stage to which the visualization technique applies. We enlarge this study for the stage maintenance and evolution along three dimensions (Novais etal. 2013): (1) the point of view, i.e., who is the user group that will use the visualization, (2) the object of study, i.e., what is being analyzed, and (3) the purpose and the focus, i.e., why is the analysis being done. Table6 shows the results. Specifically, CustomDIFF can be pigeonholed as helping the Change Control Board (who) conduct customization analysis (what) for assessing SPL scoping (why). We use the visualization means as a way to structure the rest of the paragraphs. Trees. Its intuitiveness and the large support of graphical libraries make trees the most popular intervention. Trees naturally convey the notion of hierarchy as well as setting node clusters in terms of family dependencies (e.g., ancestors, siblings, descendants, etc). These affordabilities are put into play for different purposes. Kanda etal. aim at helping engineers in migrating a set of products, created through clone&own, to an SPL (Kanda etal. 2013). For the sake of better identifying commonalities and variability, engineers require to understand how products are derived from each one. This can be challenging if no tracing records exist. In these instances, Kanda etal. introduce the Product Evolution Tree visualization whereby derivation relationships among products are displayed in a way similar to VCS branching (Kanda etal. 2013). If the aim is not migration but testing, De Oliveira etal. introduce the Product Genealogy Tree with the aim of identifying products to be retested when a bug is found in a product (de Oliveira etal. 2012). This tree captures three traces: (1) which products were derived from which core assets, (2) which products are created from already derived products, and (3) which products have propagated changes to which products. When a bug is detected in a product, this visualization can help testers identify which other related products also need to be tested. Tree-Map. This approach arranges data in a hierarchical, tree-structured diagram where the size of the rectangles is organized from the largest to the smallest. Main benefits include showing the ratio of each part to the whole. Tenev etal. resort to tree-maps for helping domain engineers identify the reuse potential of a number of similar software variants created through clone&own (Tenev etal. 2017). They compute the similarities of the source code of multiple software systems, and visualizes the commonalities and variabilities by means of multiple visualization means, such as, bar diagrams, tree-maps and phylogenetic diagrams. These diagrams provide domain engineers information about code Empirical Software Engineering (2022) 27: 75 1 3 Page 33 of 44 75 Table 6 Related work on SPL visualization, along facets:who, what, why and visualization means Reference Who What Why How Kanda etal. (2013) Domain engineers Product-to-product relationships SPL migration Product evolution tree FROM product code de Oliveira etal. (2012) SPL testers Co-product testing Product testing Testing tree FROM product code Tenev etal. (2017) Domain engineers Code similarity Spotting reuse opportunities for SPL adoption Treemaps FROM product code Wnuk etal. (2009) Change control board Feature survivability SPL scoping Feature survival tree FROM requirement documents Hinterreiter etal. (2020) Change control board Customization diffing SPL scoping Feature evolution plot FROM platform code CustomDIFF Change control board Customization diffing SPL scoping Alluvial diagrams FROM product code Empirical Software Engineering (2022) 27: 75 1 3 75 Page 34 of 44 similarity across a group of cloned software systems at different abstraction levels, i.e., from a single code line through files, folders and subsystems up to the whole system, which can then identify reuse potential and schedule an SPL migration plan. Charts. Unlike trees, charts are more convenient when a longitudinal analysis is required. Wnuk etal. introduce the Feature Survival Chart for the visualization of scoping change dynamics: the X-axis stands for time while the Y-axis holds the features (Wnuk etal. 2009). In this way, the complete life-cycle of a single feature can be followed by looking at the same Y-axis position over time. This work does not tackle code but documents that formulate features for an upcoming platform project. A feature in this case is a concept of grouping requirements that constitute a new functional enhancement to the platform. At this stage, the features usually contain a description, their market values, and effort estimates. Features are refined to requirements which are specified, reviewed, and approved. The requirements are written in domain-specific natural language. The final scope is decided and agreed with the development resources. Wnuk etal. tap into these requirement documents to build up the Feature Survival Chart where the introduction/deletion of features are monitored. In this way, the visualization shows the decision process of including or excluding features that are candidates for the next SPL release. The aim is to prevent two sorts of problems: (1) setting too large scope compared to available resources and (2), setting a limited scope early, missing market opportunities. More recently, Hinterreiter etal. (2020) propose an IDE tool for developing and evolving a clone-and-own SPL. Their approach rests on an ad-hoc variability-aware VCS. Their tool supports a codediff view that allows engineers to view how a given feature evolved for a given product (i.e., the product perspective). Specifically, Hinterreiter etal. (2020) propose two metrics to reflect how the code evolves: the relative change of the realizing artifact size (a sort of code churn), and the feature’s scattering change. In contrast, CustomDIFF is based on Git and pure::variants (i.e., annotation-based SPLs), reflecting a more industrial setting. In addition, visualization aims to account not only for the product perspective, but also the holistic perspective and the feature perspective. Both Hinterreiter etal. and Wnuk etal. track SPL evolution based on requirement documents and the codebase, respectively (Hinterreiter etal. 2020; Wnuk etal. 2009). By contrast, CustomDIFF does not tackle the visualization of the evolution of the SPL. Rather, we focus on how to inform the follow-on SPL release. Alluvial diagrams depict the customization endeavor in the interim between SPL releases. 8.3 Derivation ofDesign Principles So far, we have looked at CustomDIFF as a whole. Now, we disentangle the distinct mechanisms that on balance are responsible for the usefulness and perceived ease of use as detailed in Section7.2. Table7 outlines the main design principles. Design principles reflect knowledge of both IT and human behavior (Gregor et al. 2020). Accordingly, a design principle should provide cues about the effect (i.e., Change Control Board activity made possible), the cause (affordability brought about by CustomDIFF), and the context where this cause can be expected to yield the effect for the target audience (i.e., the Change Control Board and the application engineers). Table 7 outlines the four principles we consider more relevant. Principles flow-ness and grouping collect the benefits brought about by alluvials. On the other hand, principles zoom-ness and filtering are common principles in Visual Analytics. At this point, we also consider it significant to collect stakeholders’ opinions about each mechanism in isolation. Empirical Software Engineering (2022) 27: 75 1 3 Page 35 of 44 75 Table 7 Design principles from the CustomDIFF experience Provide visual analytics tool with ... in order for the Change Control Board to ... CustomDIFF realization Flow diagrams Track customization efforts along SPL-relevant concerns Alluvial diagrams Grouping facilities Obtain aggregates along with SPL-relevant concerns Parent-feature grouping Expansion/contraction utilities Handle distinct data-granularity levels Package bar Filtering and tracing of flows throughout the alluvial diagram Not feel lost in the tangle of flow branches Feature-product filtering Empirical Software Engineering (2022) 27: 75 1 3 75 Page 36 of 44 Table 8 Questionnaire on CustomDIFF’s mechanism Statement agreement is arranged along with a LIKERT scale from 1 (“Strongly disagree”) to 7 (“Strongly agree”) for the six participants Mechanism: Item: P1 P2 P3 P4 P5 P6 Avg. St. Dev. Alluvial diagrams I would find alluvial diagrams useful for grasping the customization diffing 6 5 6 6 6 6 5.84 0.41 Feature-product filtering I would find feature-based filtering utility useful to easy focus 7 4 6 6 6 6 5.83 0.98 Parent-feature grouping I would find the parent-feature grouping was useful to abstract away from individual features 6 5 5 5 6 6 5.5 0.58 Package bar I would find the “Package” bar was useful to further refine the flow 5 5 6 5 6 5 5.33 0.52 n.a. I would find the VP-enriched context DIFF useful to easy locate change placement into the code 6 5 7 6 5 4 5.5 1.05 Empirical Software Engineering (2022) 27: 75 1 3 Page 37 of 44 75 Participants were the same throughout the research. We limited ourselves to one question for each mechanism. Table8 collects the results. Mechanisms are in general appreciated with an average above 5 out of 7. Dispersion wise, the VP-enriched context is the mechanism with the larger dispersion. Rationales might rest on this mechanism being of interest only for application engineers while the Change Control Board might not need to delve into the code. 9 Conclusions This research started with a phenomenon perceived in practice, i.e., product customization at Danfoss Drives. In this setting, we raised two questions: (1) which are the information needs for customization analysis? and (2), might alluvial diagrams be useful for supporting customization analysis visualization? As for the former, we provided some estimates on the cost of answering some questions, and developed a GQM model. We proposed the adjustment of the popular metric of code churn for SPL constructs: feature and product. However, the potentially high number of feature churns and product churns require appropriate visualization means. This moved us to the second question. Here, we made the case of alluvial diagrams. Proof-of-concept was provided through CustomDIFF, a publiclyavailable Web application using pure::variants as the variability manager, and Git as the code repository. Proof-of-value was conducted for ease of use and usefulness using Danfoss practitioners as the subjects. Results seem to suggest that alluvial diagrams facilitate a natural way to describe the flow dispersion at different levels of detail. On one hand, the Change Control Board gets the big picture along with products and features. On the other hand, developers can expand the flow to packages and files, down to the raw code. In this way, CustomDIFF advances traditional DIFF utilities for SPL specifics. We ended by generalizing this experience to SPLs other than Danfoss Drives. In short, we contribute to the existing literature by identifying information needs and advocating for alluvial diagrams as a feasible and effective way to reflect product customization in SPLs. Additional empirical studies are needed to assess the value of customization analysis for actionable interventions. Our first evaluations indicate that customization diffing is a factor, but it is certainly not the only one that intervenes during decision making. In this respect, we plan to supplement Git data with data about products, customers, and developers, and to see what other kinds of analysis these additional sources would permit. In the same vein, metrics other than code churn might be of use. Besides tangling and scattering, more traditional metrics such as cyclomatic complexity and Halstead’s, might also be useful. Another interesting follow-on would be integrating CustomDIFF into a DevOps framework. Here, customization endeavors can be continuously tracked, so that actions can be attached to some customization thresholds being surpassed. Other scenarios include the use of CustomDIFF by application engineers to gaze at what other colleagues are customizing. For instance, a feature enhancement introduced in a given product might be promptly and directly incorporated into other products, without waiting for this enhancement to be promoted as a core asset. This opens up new scenarios for SPL evolution. Here, longitudinal evolution (between core assets and products) might coexist with traversal evolution where products sharing the same features might decide to incorporate enhancements from other products, and later on, be jointly consolidated. The aim is to find ways to alleviate the Empirical Software Engineering (2022) 27: 75 1 3 75 Page 38 of 44 tension between the quality and reuse effectiveness required by domain engineering, and the time-to-market and customer pressure faced by application engineering. Appendix: Mining atCustomDIFF CustomDIFF supports the grow-and-prune model (Faust and Verhoef 2003). Here, product branches are derived off the master branch (see Fig.1). This appendix outlines how CustomDIFF derives customization facts from so-organized Git repositories. More details can be found at the CustomDIFF repository itself: https://github.com/onekin/customdiff The process starts with the main function Mine_Customizations:6 “List <Customization_Facts> Mine_Customizations(GitRepository gitRepo, String baseline_tag)” This function takes a GitRepository as input, and returns the set of customization_facts that have been performed to a given baseline by all the products derived from it.7 In addition, baseline_tag stands for the name of the git tag that identifies the baseline for which the customization facts will be computed. A running example, take the content of Fig.1 as the GitRepo; “Baseline-v1.0” as the value for baseline_tag, “PR-” as the value for pr_pattern, 6 This algorithm was implemented in Java, using the JGit library http:// www. eclip se. org/ jgit/ 7 For automated processing, the following parameters need to be configured beforehand: (1) pr_pattern, i.e the pattern that product release tags should match (e.g., “PR-*”); (2) baseline_pattern, i.e., the pattern that baseline release tags should match (e.g., “Baseline-*”), (3) vp_init_clause, i.e. the pattern that variation point opening clauses should match (e.g., “PV:IFCOND*”), and (4) vp_end_clause, i.e. the pattern that product release tag should match (e.g., “ENDCOND*”). Empirical Software Engineering (2022) 27: 75 1 3 Page 39 of 44 75 “Baseline-” as the value for baseline_pattern, and “PV:INFOND” and “PV:ENDCOND” as the values for vp_init_clause and vp_end_clause, respectively. Algorithm1 provides the details: 1. Identify which is the baselineCommit to analyze (line 4). The function getCommitByTagName returns the commit to which the baseline_tag points to. For our running example, baselineCommit holds the commit c5. 2. Identify the product releases that were derived from the baselineCommit (lines 6–11). This implies to: – For all the tags in gitRepo, identify those that are product releases (lines 6–9). First, collect all the existing tags in the repository (line 6). For our running example, the variable all_tags holds now: London-v1.0, NewYork-v1.0, Paris-v1.0, Berlin-v4.0, Baseline-v0.5 and Baseline-v1.0. Second, filter out those tags that are not product releases. i.e., those that do not match the pr_pattern (line 7–9). For our running example tags Baseline-v0.5, and Baseline-v1.0 are filtered out. – Filter out the product releases that were not actually derived from the baselineCommit (lines 10-11). This is achieved in two steps. First, we identify the baseline commit from which each product release was derived. This is calculated by getBaselineForRelease (line 10). This method takes a product release tag (e.g., Berlin-v4.0), traverses the git history (e.g backwards from c17) until it finds a commit tagged with a label that matches the pattern baseline_pattern (e.g., Baseline-v1.0), and finally, returns the commit it points to (e.g. c5). Second, we filter out those product releases whose baseline is not equal to baselineCommit (line 11). For our running example, the product release London-v1.0 would be filtered out, as the baseline it was derived from is c3 instead of c5. 3. Finally, compute the customization facts for each product release that was indeed derived from baselineCommit (lines 12–16). This implies for each product release to: – Perform a DIFF operation between the baselineCommit and the commit to which the product release tag is pointing to (line 12). For instance, the DIFF operation for the product release tag Berlin-v4.0 would be as follows: diff(c5, c17). The result of the operation, i.e., diffs, is the list of diff-outputs (a.k.a patches), one per file that the product has changed from the baseline. For instance, if the product release Berlinv4.0 changes five files from the baseline, then diffs would contain five diff-output files, each per file changed (see Listing 1 as an example of a diff-output). – For each diff-output, extract the customization facts by calling the method extractCustomizationFacts (line 14). This method, parses the diff-output, identifies the set of consecutive changes performed to the same variation point, and returns the corresponding customization facts. – Finally, add the extracted customization facts to the global container customizations (line 15). Return this container when all product releases are mined (line 20).