Large Language Models to generate meaningful feature model instances

Large Language Models to generate meaningful feature model instances José A. Galindo [email protected] University of Seville Seville, Spain Antonio J. Dominguez [email protected] University of Seville Seville, Spain Jules White [email protected] Vanderbilt university Tenesse, USA David Benavides [email protected] University of Seville Seville, Spain ABSTRACT Feature models are the “de facto” standard for representing variability in software-intensive systems. Automated analysis of feature models is the computer–aided extraction of information of feature models and is used in testing, maintenance, configuration, and derivation, among other tasks. Testing the analyses of feature models often requires relying on a large number of models that are as realistic as possible. There exist different proposals to generate synthetic feature models using random techniques or metamorphic relations; however, the existing methods do not take into account the semantics of the concepts of the domain that are being represented and the interrelations between them, leading to less realistic feature models. In this paper, we propose a novel approach that uses Large Language Models (LLMs), such as Codex or GPT-3, to generate realistic feature models that preserve semantic coherence while maintaining syntactic validity. The approach automatically generates instances of feature models from a given domain. Concretely, two language models were used, first OpenAI’s Codex to generate new instances of feature models using the Universal Variability Language (UVL) syntax and then Cohere’s semantic analysis to verify if the newly introduced concepts are from the same domain. This approach enabled the generation of 90% of valid instances according to the UVL syntax. In addition, the valid models score well on model complexity metrics, and the generated features mirror the domain of the original UVL instance used as prompts. With this work, we envision a new thread of research where variability is generated and analyzed using LLMs. This opens the door for a new generation of techniques and tools for variability management. CCS CONCEPTS •Software and its engineering → Software product lines;• Computing methodologies →Natural language generation. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. SPLC ’23, August 28-September 1, 2023, Tokyo, Japan https://doi.org/10.1145/3579027.3608973 KEYWORDS large language models, deep learning, universal variability language, synthetic models ACM Reference Format: José A. Galindo, Antonio J. Dominguez, Jules White, and David Benavides. 2023. Large Language Models to generate meaningful feature model instances. In 27th ACM International Systems and Software Product Line Conference - Volume A (SPLC ’23), August 28-September 1, 2023, Tokyo, Japan. ACM, New York, NY, USA, 12 pages. https://doi.org/10.1145/3579027.3608973 1 INTRODUCTION Feature models[ 17 ] are widely used to represent the common and variable elements of software product lines and variability-intensive systems. Current models of such systems, such as the Linux kernel [ 24 ], Debian distributions [ 10 ], and web-oriented frameworks [ 32 , 36 ], can contain more than 7,000 features. The large size of these models made manual analysis a laborious and error-prone task. Automated analysis of feature models [ 4 , 12 ] was introduced 30 years ago to allow computer-aided mechanisms to extract useful information from feature models. Since then, several applications and techniques have been developed for automated analysis and have allowed testing [ 13 ], maintenance and improvement of software product lines [24], and variability-intensive systems. A large amount of tooling built around feature modeling analysis tools such as product analysis, testing, or reverse engineering often requires realistic feature models. For example, in [ 33 ], authors propose a method to create synthetic feature models based on the Thum et al. algorithm[ 37 ]. Furthermore, more complex solutions have been proposed, ranging from feature model repositories [ 25 ] to reverse engineering algorithms based on metamorphic relationships [ 23 ]. Also, some other approaches seek to embed domain knowledge within the generation of feature models [ 3 ]. However, none of the aforementioned solutions takes into account the domain of the features involved in the generation of the feature models. For example, when increasing the features of an IoT device feature model, it makes sense to add features such as Zigbee communication protocol. Also, it is well known that feature models in the industry are strategic artifacts that are rarely shared with the scientific community, which makes it difficult to obtain realistic feature models from a given domain. Large Language Models (LLMs) are artificial intelligence models that use deep learning algorithms to process and analyze large SPLC ’23, August 28-September 1, 2023, Tokyo, Japan José A. Galindo, et al. amounts of natural language data [ 7 ]. These models are trained on massive text data sets, which allows them to understand and generate human-like language. They can be used for a variety of natural language processing tasks, such as language translation, text summarizing, and language generation. Some popular large language models include GPT-3[ 6 ], BERT[ 8 ], and T5 [ 30 ] as well as their conversational counterparts such as ChatGPT [ 27 ]. Furthermore, patterns for structuring prompts for these models allow them to learn and work with domain-specific languages [ 40 , 41 ]. The usage of LLMs is a trending topic in research and application for generating music, text, conversations, or even artificial speeches and interviews, also in software engineering for code generation [42]. In recent years, an effort has been made to develop a new language to serialize feature models called Universal Variability Language (UVL), which focuses on sharing models, teaching, and enabling cross-tool support [ 35 ]. However, currently, there are not that many UVL models generated, and most of the generation relies on the translation from other languages such as the FeatureIDE [ 19 ] format. In this paper, we present a pioneering approach based on the use of LLMs to generate realistic variants of feature models that preserve the semantics of the concepts used within the model. Concretely, we present the following contributions: • the usage of OpenAI Codex LLM to generate realistic feature models using UVL as the syntax, • the usage of Cohere 1 embedding models for creating semantic representations of the features. In this way, feature models mimic realism in terms of structure and w.r.t. domain elements. • an empirical evaluation showing the proposal’s feasibility with more than 90 % of valid instances with a good score of semantic coherence in the used domain. This paper lays the foundation for a new avenue of research at the intersection of LLMs and variability. Our findings suggest that LLMs offer promising opportunities for advancing software product line engineering and variability management tasks, and we anticipate a new generation of tools that will facilitate this progress. This work represents an initial contribution toward realizing the full potential of LLMs in this domain, and we hope that our results will inspire further research and development in this area. The rest of the paper is structured as follows. Section 2 introduces the basic knowledge required to understand this research. Then, the proposed solution for generating UVL instances is presented in Section 3. Next, the evaluation is presented in Section 4. Next, threats to the validity of our study are presented in Section 4.6. Finally, the conclusions and future work are presented in Section 5. 2 BACKGROUND In this section, we present the main concepts necessary to understand the scope of this contribution. 2.1 Universal Variability Language The Universal Variability Language (UVL) is designed to manage variability in software product lines. Designed to simplify representing and managing feature models, allowing developers to more 1https://cohere.ai/ features Sandwich mandatory Bread optional Sauce alternative Ketchup Mustard Cheese constraints Ketchup => Cheese Figure 1: UVL model example [35] effectively customize their products to meet diverse requirements [ 35 ]. UVL provides a robust set of constructs and abstractions to model the relationships and dependencies among features. Figure 1 presents a basic example feature model for a sandwich product line. Note that the constraint is not graphically represented. UVL uses tabs to mimic the tree structure of the feature model and the basic feature model relationship names to depict the type of relationship between features. It is important to note that tabs are also common in other programming languages, which is relevant to the LLM selection performed in the experimentation. Concretely, this model represents a sandwich in which the cheese and sauce (either ketchup or mustard) are optional, while the bread is a mandatory feature. A cross-tree constraint indicates that if ketchup is selected, the sandwich will have cheese on it. 2.2 Large Language Models A Large Language Model is an AI model trained with an extremely large amount of text that can predict some output of text accurately when you feed the model with some other text. One of the most well-known applications of this kind of model is ChatGPT which allows humans to prompt the model with any question or order, and the model can reply with an answer that is useful in many cases (although in others is not). It is easy to imagine asking for software engineering tasks, such as giving the code for connecting to a database using a given programming language or an algorithm for a specific task. Recent advances in LLMs with pre-trained task-agnostic language have shown significant evidence of their capabilities to downstream tasks while showing remarkable accuracy [ 30 ]. Scaling laws [ 18 ] have further proven this general agnostic training paradigm success, which can reduce loss of information from the model by increasing training data, the number of parameters of the model, or the computation time. Scaling improves the model’s overall accuracy and has also shown the emergence of novel prompting properties [ 6 ] not observed in smaller language models. In-Context Learning has emerged as one of the most remarkable unexpected properties of this class of language models. In-context learning allows for controlling the model’s expected output by effectively and precisely selecting the input, thus crafting the optimal prompt for a given task and domain. The shape of an optimal prompt is task-dependent, and some Large Language Models to generate meaningful feature model instances SPLC ’23, August 28-September 1, 2023, Tokyo, Japan current research approaches document the patterns of the input prompts, similar to software patterns, to determine how to achieve a given output or behavior [40, 41]. For example, the LLMs explored in this paper were not explicitly trained to perform tasks with UVL, but if we provide them with examples, they can generate and perform operations on them. Note that the examples can be provided in different ways, either directly in the prompt (e.g., GPT) or with an LLM trained with code (e.g., Codex). LLMs can also be surprisingly capable of code generation tasks, although there are far fewer lines of code in the training data sets than in general natural language statements. Therefore, it is sensible to think that making code samples the majority of training data would gradually lead LLMs to approach human-level code generation proficiency. As examples of this approach, we can see OpenAI Codex [ 7 ], DeepMind AlphaCode [ 20 ], MetaAI InCoder [ 9 ], and Salesforce CodeGen [ 26 ] to mention some of the most recognized large language models. 2.3 In-Context Learning & Prompting Specializing a model by fine-tuning it for a particular task (e.g., summarizing, sentiment classification) requires enough training data and gradient weight updates [ 22 ]. Data collection and preparation are expensive and sometimes impossible because of the difficulties in obtaining quality samples for the given task. Fine-tuning the whole model for introducing new training steps in task-specific training data can degrade the model generalization without losing previous knowledge. When this occurs, the model diverges from its original homogenization version [ 14 ] and becomes a new unimodal model for the given task. This paradigm leads to a disjoint set of model weights per task, even if they are all descendants of the same root foundation model. Note that this approach is not efficient on large model scales [16]. In-Context Learning is a non-observable property in small language models. It emerges as the model scales, allowing control of their responses to specific tasks by leveraging the provided context as a prompt [ 1 ]. That is, by modifying the prompt, the language model can maintain the same accuracy while being able to tackle tasks they were not explicitly trained for. This property sits on top of the general language model information and enables the generation of high-quality outputs by effectively understanding the intent and domain of the task in the prompt [ 1 ]. The prompt can be composed of simple queries or more complex domain-specific problems, such as arithmetic, reasoning, response to facts, or classification. Note that modifying the appropriate prompt only increases the model’s accuracy for a specific task. Therefore, we freeze the LLM’s weights (values stored within it) when prompt tuning and avoid weight gradient updates [ 1 ]. Although the model is the same, we can craft task-specific prompts that achieve nearly the same accuracy as model fine-tuning. In the paradigm of in-context learning, we can distinguish three main alternatives depending on the number of examples in the prompt. Each set of examples depends on the task to be performed, in our case, generating a variant of a UVL model. Concretely: (1) Zero-shot (0 𝑆 ): The model receives a textual description of the desired task, but there are no demonstrations in the form of examples to infer. The language model is meant to generalize and avoid memorization of instances occurring very broadly in the pre-training data for the given prompt. This configuration can be considered the hardest for the model, requiring robust learning and low task context. (2) One-shot (1 𝑆 ): Similar to zero-shot, but providing a single demonstration. The context ends with a demonstration in which the output is blank. The model then generates this output at inference time. (3) Few-shot (FS): We continue the 1S configuration but increase the number of demonstrations inhabiting the context. K examples of context are given. This value usually ranges from 10 to 100 examples. For a given task, the K-value depends on the model context window. In this case, we provide a set of examples. Since this emerging capability was discovered in language models, extensive research has been done to improve further how we build prompts [ 2 ]. Introducing domain keywords, identifying the optimal order and quantity of examples, or consolidating general templates are examples of these efforts [ 2 , 21 , 40 ]. From the extensive set of prompting approaches, we highlight several techniques. Chain-of-Thought: Deconstructing the initial task into smaller and longer subtasks [ 39 ], augmenting the overall problem context in the prompt and increasing the model’s reasoning. Therefore, the model can better understand the correct solutions by adding intermediate reasoning steps in the prompt task description. Instruction prompting: The main underlying idea is tasks expressed as instructions [ 38 ]. We create a prompt template with a list of instructions in order to accomplish the task and provide several potential outputs or intermediate considerations. Meta Language Pattern: “From now on, whenever I type two identifiers separated by a “ → ”, I am describing a graph. For example, “a → b” describes a graph with nodes “a” and “b” and an edge between them. If I separate identifiers by “-[w:2, z:3] → ”, I am adding properties of the edge, such as a weight or label.” Template Pattern: “I am going to provide a template for your output. Everything in all caps is a placeholder. Anytime you generate text, try to fit it into one of the placeholders that I list. Please preserve the formatting and overall template that I provide at https://myapi.com/NAME/profile/JOB” Persona Pattern: “You are going to pretend to be a Linux terminal for a computer compromised by an attacker. When I type a command, you will output the corresponding text that the Linux terminal would produce. Your output should include evidence of a cyberattack.” A larger catalog of prompt tuning patterns was presented in [ 40 ] and [21]. 2.4 Analyzing Semantic Similarity with LLMs Sentence embedding is a mathematical representation of human language constructions like words and phrases in a high-dimensional vector space [ 31 ]. The main target is to learn meaningful representations from textual data that consider not only the structure of the sentences but also the context that derives from the semantic relationships present in the sentences. Therefore, we expect to learn embedding that creates semantic regions in this high-dimensional SPLC ’23, August 28-September 1, 2023, Tokyo, Japan José A. Galindo, et al. vector space. If this statement holds, we can compute closeness embedding using distance metrics such as Euclidean distance or cosine similarity. Given a sequence 𝑥={𝑥1, ..., 𝑥|𝑥|} , the representation of the token 𝑥𝑖 produced by the language model is ℎ𝑥𝑖 . Then, we can effectively compute embedding similarity using the function 𝑠 , which receives two embedding as input arguments. We can evaluate this function using the following formal expression: 𝑠(ℎ𝑥𝑖,ℎ𝑥 𝑗 )= ℎ⊤ 𝑥𝑖ℎ𝑥 𝑗 ∥ℎ𝑥𝑖 ∥·∥ℎ𝑥 𝑗 ∥(1) Where: •ℎ𝑥𝑖 and ℎ𝑥 𝑗 are the embedding representations of the tokens 𝑥𝑖and 𝑥𝑗respectively. •ℎ⊤ 𝑥𝑖ℎ𝑥 𝑗 is the dot product. • ∥ℎ𝑥𝑖 ∥·∥ℎ𝑥 𝑗 ∥ is the scalar resulting from multiplying each embedding norm. Cosine similarity is a useful metric because it is scale-invariant and takes into account the orientation of the vectors rather than their magnitudes. This makes it more appropriate for comparing documents with different lengths or items with different scales. 3 GENERATING UVL WITH LLMs In this section, we show the process we followed to generate new UVL instances that preserve the domain of the new features included. This is, to obtain a set of instances that are correct with respect to the previous definition, we follow the process from Figure 2. First, we feed the process with an existing UVL model. Second, we could apply certain techniques of prompt engineering to obtain better results. Note that these are dependent on both the original UVL model and the underlying LLM. Third, we obtain a set of proposals for completion by the LLM. Then, we use the UVL Flama [ 35 ] parser to check the syntactic validity of the models. Later, we extracted the feature names of the models that passed the syntax check. Finally, we feed those sets of words to the Cohere semantic analysis and perform the cosine similarity to verify to what degree it differs from the original set of feature names within the original model. UVL instance LLM generation 2 UVL instance Promnt preparation and model selection 1 Set of UVL models Syntactic checker Semantic checker UVL instance UVL instance 3 4 Figure 2: Process followed to generate UVL instances 3.1 Prompting The first step would be to prepare the model and prompt that will lead to the resulting variations of the UVL models. For that, two main alternatives are available, first, to perform instruction prompting, second, to rely on a more specialized LLM. Instruction prompting plays a crucial role in optimizing LLM performance. Prompt tuning refers to the process of fine-tuning an AI model by adjusting its input prompts to generate more accurate, relevant, and contextually appropriate responses. By using carefully crafted instruction prompts, developers can guide the AI model to understand the intended query or task better, thus improving its output quality. Instruction prompting in prompt tuning involves providing the AI model with additional context, clarifying the desired response format, or specifying particular constraints that the generated answer should follow. The goal is to create prompts that are unambiguous and explicit, enabling the AI model to generate more precise, useful, and reliable responses. This process often requires experimentation, iterative refinement, and an understanding of the AI model’s strengths and limitations, ultimately leading to enhanced performance and more satisfying interactions for end users. For example, if we apply instruction prompting[ 38 ], we will follow these steps. First, we define the context in which the inference will occur, "UVL specifies variability models with a tree-like structure to represent the hierarchical structure of variability models. Below, you can see a simple example of a variability model in UVL." Then, we explicitly specify the instruction, "Augment the following UVL example while preserving formalism and semantic similarity". Finally, we obtain the result shown in Figure 3. features Sandwich mandatory Bread optional Sauce alternative Ketchup Mustard Mayonnaise optional Cheese optional Toppings alternative Onion Lettuce Figure 3: Variant resulting from applying pseudo-tuning to non-code-oriented LLMs. Prompt engineering is as important as model selection when using statistical models such as LLMs. Relying on prompt engineering techniques can help to improve the efficiency of models on tasks on which they were not explicitly trained. On the other hand, there are models that are suited for a custom task, such as code generation, that require almost no prompt engineering to achieve the same result. In our experimentation (see Section 4), after analyzing the characteristics of UVL and the available LLMs we got access to, we chose Large Language Models to generate meaningful feature model instances SPLC ’23, August 28-September 1, 2023, Tokyo, Japan to keep the prompt as clean as possible, only providing an example of the UVL model and asking the LLM to augment it. This was seeking the reproducibility of the experimentation and reducing threats due to the prompting techniques selected. 3.2 Inference The second step is the inference of new feature models using LLMs. LLMs are stochastic models that predict output based on a given set of inputs. In this step, we present how the inference is performed. Sampling methods in LLMs are techniques used to generate text by selecting words or tokens from the model’s output probability distribution (a.k.a. inference). These methods help to control the diversity and quality of the generated text. There are two major categories of sampling methods, deterministic and stochastic sampling. Each one offers particular parameters to regularize and control the model generation, thus offering a path to perturbing the overall model’s accuracy for a given task, like code generation or other natural understanding tasks. Deterministic sampling methods, such as greedy search, can yield suboptimal generations, while their results might be more predictable and interpretable. This low performance can be observed by creating undesirable and semantically incorrect repetitions at different levels (e.g., token, sentence) while losing similarity with respect to the previous context[34]. Stochastic sampling methods offer higher variance in the results at the cost of introducing higher chances of generating semantically incorrect results or even pure noise results, such as empty strings. These methods have a set of hyperparameters not present in deterministic sampling strategies [ 29 ]. We relied on this type of sampling method and, concretely, on the following parameters: (1) Temperature: Controls the variance and degree of randomness of the generated text by adjusting the probability distribution of the model’s output tokens. Therefore, we interpret this hyperparameter as the model’s creativity. A higher temperature value (e.g., 1.0) results in more diverse and creative output, while a lower temperature value (e.g., 0) produces more focused and conservative output, closely adhering to the training data. Proper tuning of the temperature can help balance the trade-off between diversity and coherence in the generated text. (2) Top-p (Nucleus Sampling): Top-p sampling, also known as nucleus sampling, selects a subset of tokens whose cumulative probability exceeds a predefined threshold 𝑝. (3) Repetition penalty: Mitigates the issue of repetitive or redundant output. By applying a repetition penalty, the probability of selecting a token that already appears in the generated text is reduced, which encourages the model to generate more diverse and coherent content. (4) Bad Words Tokens: We explicitly instruct the model to stop the generation early if the results contain any of the listed sequences, like words, that are categorized as stop sequences. We primarily employ this procedure as a means of stopping generations that diverge from the original task. There is an extensive set of sampling hyper-parameters [ 29 ], but in our research, we limit our attention to the previous ones. Note that, in our experimentation, we relied on OpenAI alternatives which are based on stochastic sampling methods and fixed all the aforementioned parameters. For example, Figure 4 presents two possible results of the inference result. features Sandwich f o r c e d Bread optional Sauce alternative Ketchup Mustard Cheese constraints Ketchup => Cheese Variant 1 features Sandwich mandatory Bread optional Sauce or Ketchup Mustard Mayonaisse Cheese constraints Ketchup => Cheese Variant 2 Figure 4: Variants resulting from the inference of LLMs. 3.3 Syntactic validation The third step is syntactic validation. For the syntactic validation of UVL instances, we have relied on the ANTLR [ 28 ] Python implementation of the FLAMA framework [ 11 , 35 ] 2 . This grammar checks if the file of the UVL instance expresses a valid model in terms of the syntaxis of the UVL language. This parser uses a custom lexer to handle indentation-based scoping, making it similar to Python in terms of syntax. The grammar contains the following main sections: (1) Tokens: The grammar defines INDENT and DEDENT tokens, which are used to represent the indentation levels in the input text. (2) Lexer customization: The lexer is customized with a Python class called UVLDenter, which extends the DenterHelper class. This class is used to handle indentation-based scoping for a language similar to Python. (3) Parser rules: The parser rules define the structure of the language, including the feature model, namespace, features, constraints, imports, and their corresponding subrules. (4) Lexer rules: The lexer rules define the individual tokens that make up the language, such as WORD, BOOLEAN, NUMBER, VECTOR, and various keywords (e.g., ’true’, ’false’, ’requires’, ’excludes’). This syntactic validator makes sure that the models adhere to the basic models specified in the UVL language. This is, without taking into account attributes and complex cross-tree constraints. For example, it would detect that the forced (which is not a valid UVL construct) relationship does not exist in the first variant of the models presented in Figure 4, and thus it discards it. 3.4 Semantic validation Finally, the fourth step of the process is where we check if the elements of the features that have been added were from the same domain as the ones present in the original UVL instance that we 2https://flamapy.github.io/ SPLC ’23, August 28-September 1, 2023, Tokyo, Japan José A. Galindo, et al. feed in the first step. For this validation, we also propose the use of an LLM to have a quantification of how much the elements are close or far from the domain of the original UVL instance. To verify the distance between two feature sets, we propose the use of the cosine similarity technique. Cosine similarity is a measure that is used to determine the similarity between two nonzero vectors by calculating the cosine of the angle between them. Below we enumerate the steps required to do this calculus from the two sets of features. This metric is calculated as follows: (1) Represent both sets of features as vectors, this can be done either using term frequency-inverse document frequency (TF-IDF) vectors or using other vectorization techniques such as word embedding. (2) Calculate the dot product of the two embeddings: Multiply the corresponding elements of the vectors and sum the results. (3) Calculate the magnitudes of A and B: For each vector, find the square root of the sum of the squares of its elements. (4) Divide the dot product by the product of the magnitudes: This gives the cosine similarity, which ranges from -1 (completely dissimilar) to 1 (identical). The cosine similarity score ranges from -1 to 1, where a score of 1 indicates that the two sets of features have identical meanings and thus domain, and a score of -1 indicates that they have completely opposite meanings. Scores closer to zero indicate that the two phrases are less similar in meaning. This is, in the case of obtaining a 1 that could mean that no new concepts have been added to the UVL semantics. For example, assume the following three vectors that represent the features of three models that were inferred using the model presented in Figure 4 as prompt. (1) Vector A: Sandwich, Bread, Sauce, Ketchup, Mustard, Cheese. (2) Vector B: Sandwich, Bread, Sauce, Ketchup, Mustard, Mayonnaise, Cheese. (3) Vector C: Sandwich, Bread, Sauce, Ketchup, House, Oil, Cheese. First, we would need to calculate the embedding; let us assume that the vectors A, B, and C have the following embedding. (1) Vector A: [1, 2, 3, 0, 0, 0] (2) Vector B: [1, 2, 3, 4, 0, 0] (3) Vector C: [1, 2, 3, 22, 53, 4] Now we can determine the cosine similarity between each pair of vectors. This is, for the pair of vectors A, B 𝑠(ℎ𝐴,ℎ𝐵)= 0 . 9746; for the pair A, C 𝑠(ℎ𝐴,ℎ𝐶)= 0 . 9344, and; the pair B, C 𝑠(ℎ𝐵,ℎ𝐶)= 0 . 5202. As we can see, the cosine similarity between A and B is the highest, indicating that they are most similar. The cosine similarity between A and C is still high but lower than the similarity between A and B. The cosine similarity between B and C is the lowest, indicating that they are the most dissimilar pairs. 4 EMPIRICAL EVALUATION In this section, we present the three experiments we carried out to generate semantically meaningful valid UVL instances. First, we analyze the syntactic validity of the models being generated. Second, we analyze how the generated models differ from the UVL model we used as a prompt. This is shown in terms of the number of features, number of constraints, and number of relationships. Finally, in the third experiment, we analyze whether the generated models share the same domain as the model used in the prompt by performing a cosine similarity using the Cohere embedding system. 4.1 Experimental Platform and Process Initially, we downloaded 1215 SPLOT website feature models in splx format and converted them to FaMa XML format[ 5 ]. Then, we manually filtered out unrealistic feature models and those without meaningful feature names, resulting in 30 feature models that we translated into UVL. Figure 5 shows the process we followed. 1215 models UVL instances Transform Transform FaMaXML instances UVL meaningful instances 30 models Inference UVL meaningful instances Experiment 1 150 variants UVL meaningful instances FM analysis Domain analysis Manually filtering Figure 5: Experimentation process Then we used each of those 30 realistic SPLOT models as prompts without adding any extra sentences or prompt engineering techniques in the OpenAI platform (relying on its API). Then, we ask the API to generate five input augmentations for each prompt. Note that this number is to find a trade-off between 1S experimentation and it is based on the AlphaCode results, which ranged from 0 to 10. This resulted in a total of 150 models. Then, we analyzed these models syntactically and semantically, comparing them with the model used as a prompt. For syntactic analysis and format transformations, we relied on Flama [ 11 ] running on an Intel i7 processor equipped computer with 8 Gb of ram and Ubuntu 20.04. For semantic analysis, we used the Cohere platform to calculate the embedding and compared the domains of the original models with the generated syntactically valid variants. All this process has relied on the Python implementation of FLAMA and the API calls to the different services. 4.1.1 Experiment parameter selection. Currently, we find a variety of techniques for tuning the experimentation using LLM. The first experimental parameter to select is the LLM we will use. First, there are foundation LLMs trained to be as generalist as possible and then adapt it to many applications (this is the case of GTP3 and Chat-GPT). This adaptation can be made in multiple ways, such as increasing the amount of code in the LLM training set (data distribution) [ 6 ] or adapting an LLM trained in text by prompting [ 21 ]. Large Language Models to generate meaningful feature model instances SPLC ’23, August 28-September 1, 2023, Tokyo, Japan However, there is no consensus on what the best mechanisms are to adapt LLM to concrete tasks such as code completion. On the other hand, there has been an effort to produce LLMs that are better suited for code generation. This is achieved by training the models with more data out of code rather than plain text (or literature), among other optimizations. This is the case for the Codex LLM, which was trained with large amounts of data from GitHub and other open-source repositories. Model Selection: We selected the Codex LLM, which was trained on large amounts of code. Codex is the LLM behind the Github Copilot technology (see https://github.com/features/copilot). This would remove the need to perform a heavy prompt tuning in which we need structures to perform our queries to the model, thus, removing the bias of the prompt from this experiment. This is, we will feed just the input UVL serialization as a prompt and tell the LLM to augment it. This is a one-shot experiment (see Section 2). This selection was made not only because of how Codex was trained but also because of how it manages the input tokens. For example, Codex is capable of grouping the tabular data, while other GPT3 alike options consider those tokens as a set of spaces making it more difficult to produce syntactically valid code. This process is called tokenization 3 and, in Codex, is optimized due to the need for some general-purpose languages, such as Python, to rely on tabs to organize the code. We selected Codex because UVL shares some language constructs with Python, making Codex a more suitable LLM for generating code. Prompts. Our prompts consist of feature model serializations, those models were extracted from the SPLOT repository, and we are aiming at the task of text augmentation. Concretely, we are seeking the generation of different models that share the domain of the original ones. This is so because the results are both valid syntactically and semantically. Therefore, our prompts are 1-shot prompts in which we provide the original model from SPLOT and ask the LLM to extend them. LLM Hyperparameters. These are the hyperparameters that we set in our experiments. Note that this set of parameters is suited when stochastic sampling methods are used, like in the case of the Codex LLM in the OpenAI API. Temperature: The correct temperature value is tied to the number of generations we will sample from the model [ 7 ]. Ideally, we should have unlimited inference resources for the model of interest. However, in our case, the chosen model, Codex, was delivered under a restricted and limited API call policy. This led us to reduce the number of generations per prompt to five. Then, we fixed the temperature of 0 . 7, which increases our overall confidence about the likelihood of the model generating too many duplicate results, thus wasting the inference call to Codex’s API. Max tokens: In this parameter, we specified that we would reach a maximum of 500 tokens in the output. Top-p: In this parameter we selected the value of 1. This is because it offers a good trade-off between generating irrelevant tokens while not being too repetitive. 3https://platform.openai.com/tokenizer Repetition penalty: We left this parameter at 0. This is, we will not penalize if a model reuses a tab token, for example. Bad word tokens The set of bad word tokens are set to those that are very common in general-purpose languages but do not exist in UVL. The ones we selected were “def”,“#”,“public”, and “$”. 4.2 Experiment 1: Evaluating the syntactic validity of the models generated In this experiment, we try to see if the LLM is capable of generating syntactically valid instances of UVL models. For that purpose, we start by selecting the 30 models from the SPLOT repository that contain meaningful feature names. This is those models in which the names represent domains, types, or other things and labeled the models with obfuscated names of features. Hypothesis: we hypothesized that for most generations, the new UVL instances would be syntactically valid. Initially, we expected that at least half of the generations were valid. This is so because, for a generation of five models for a prompt, we should obtain three or more valid generations. Experiment 1 Results. After following the process of generating new UVL models, 14 variants were not syntactically valid out of 150 variants which led to a success of 90%. aircraft_fm REAL-FM-6 REAL-FM-17 REAL-FM-12 REAL-FM-2 0 1 2 3 4 5 optional Engine mandatory or <−missing feature alternative Jet Piston Figure 6: Wrong variants and causes. Moreover, bad generation seems to affect only a small subset of the 30 models used as prompts. Figure 6 shows the number of generations that failed for the models with at least one failure. After looking closely at the models with problems w.r.t. syntactic analysis, we observe that all the cases were caused by either a wrong use of tabs or adding a relation between two features but not introducing the type of relationship. Figure 6 shows some examples of the problems mentioned earlier. This increases our confidence in the importance of an adequate tokenizer for UVL generation. 4.3 Experiment 2: Evaluating the increase in complexity when generating In this experiment, we start analyzing the 150 models (e.g., UVL instances) we generated using the Codex LLM. Then, we see if there are actual differences between the model used in the prompt and the returned model from the LLM. Additionally, in this experiment, we only report on syntactically valid models. Concretely, we relied on the number of features, the number of cross-tree constraints, and the number of relationships. To perform this analysis, we have relied on the FLAMA framework and the FM fact label tool[ 15 ] to SPLC ’23, August 28-September 1, 2023, Tokyo, Japan José A. Galindo, et al. categorize the different instances of feature models generated using LLM. Hypothesis: The generated variants differ from the original model. After obtaining the UVL variants, we hypothesized that we would obtain a new feature model that differs from the original model. We expect a variation in the number of features, constraints, and relationships. Experiment 2 Results. Figure 7 shows the results with respect to the number of features, the number of relationships, and the number of cross-tree constraints. The different generations are represented in the X-Axis of the different plots being the one represented by a 0, the initial prompt we used. We observe that for most models, the instances obtained differ from the original prompt; however, the difference is not remarkable in cross-tree constraints. This is caused because the LLMs tended to increase the models’ size by adding content to the middle of the files instead of the end. However, this could be improved by concatenating LLM generations and specifying it to include new constraints if needed. Also, it is noticeable that there were no changes in the case of REAL-FM5 and cfpd_library models. After taking a closer look, the reason is that there we do not have actual model differences between variants and the original model, as shown later in Section 4.4. 4.4 Experiment 3: Evaluating domain relativeness in generation In this experiment, we focus on determining whether the new concepts introduced in the model are from the domain of the already existing features of the model we used as a prompt. To verify if this is correct, we extracted the feature names of all the generated models, as well as the original prompt (0 in the X and Y Axis, and then compared it using another LLM. Concretely, we have relied on the Cohere API, which has embedding calculation as one of its main features. Hypothesis: We hypothesized that the domain of the new concepts introduced would be close to the domain of existing characteristics. Concretely, we expect that the divergence in most cases does not go below 0.5 in terms of cosine similarity. Experiment 3 Results. Figure 8 shows the results we obtained for cosine similarity of all pairs of vectors extracted from the features of the models used as input and the generated ones. As we can see, in the case of the generation that failed (see Section 4.2) the variants introduced no new concepts, and thus the cosine similarity is equal to one. Also, this happens in the diagonals of each model because we are comparing the same variants between them. For example, in the points (0,0), we are comparing the original prompt with itself. On the other hand, we observe that in most cases, the similarity is between 0.5 and 0.8, which implies that new concepts with close domains have been introduced effectively. Also, we observe that for some models, such as REAL-FM-12, many new concepts were introduced and provided very good results, while in others, such as stack-fm, there were not many new concepts added. To analyze that, we manually reviewed some of the models with not many new concepts added, finding out that they also contained some obfuscated feature names that provided such results. In other words, we keep some models that contain both obfuscated and realistic feature names, which increases the difference between names for certain models. 4.5 Analysis of Results The results from the experiments indicate that the LLM can generate valid instances of UVL models with varying degrees of success. The experiments demonstrated that the Codex LLM is a suitable choice for this task, as it effectively manages syntactic aspects of the UVL and can keep the features in the same domain as the original model instances. In Experiment 1, the data supported the hypothesis that at least half of the generations would be syntactically valid. The incorrect generations were mainly due to issues with tab usage or missing relationship types, which could potentially be addressed with further fine-tuning of the LLM. In Experiment 2, the hypothesis that the generated variants would differ from the original models was also supported. The analysis showed differences in the number of features, constraints, and relationships, although the differences in cross-tree constraints were not as significant. This could be improved by adjusting the LLM’s generation process to add content at the end of the files or by specifying the inclusion of new constraints. Interestingly, for some models, such as REAL-FM5 and cfpd_library, no changes were observed, which is due to a lack of actual differences between the variants and the original models. In Experiment 3, the hypothesis that the domain of the new concepts introduced would be close to the domain of existing features was generally supported. The results of cosine similarity indicated that, in most cases, the new concepts were closely related to the original domain. However, there were some variations between models, such as REAL-FM-12, which exhibited many new concepts, while stack-fm showed minimal new concepts. The discrepancy could be due to the presence of obfuscated feature names in some models, which affected the generation process. In conclusion, the experiments demonstrated the potential of LLMs in generating valid instances of UVL models. The results suggest that LLMs can be used effectively for this task with appropriate fine-tuning and training. Furthermore, the analysis highlights areas where improvements can be made, such as addressing syntactic issues and better handling of obfuscated feature names, to enhance further LLMs’ performance in generating UVL model instances. 4.6 Threats to validity This section discusses the different threats to validity that affect the evaluation. External validity. The inputs used for the experiments presented in this paper were realistic or designed to mimic realistic feature models. SPLOT feature models are regularly developed by numerous experts involved in model upload. However, since there is no control over uploads, they may have errors and not be representative of the systems they may represent. Also, note that the generation of variants, while trying to stay in the same domain due to the stochastic nature of this approach, might not fully mimic realistic models. The major threats to external validity are: - Population validity, we do not have a guarantee that the models used were realistic. To reduce these threats, we manually reviewed the set of models existing in the SPLOT repository and Large Language Models to generate meaningful feature model instances SPLC ’23, August 28-September 1, 2023, Tokyo, Japan 0 5 10 15 20 25 REAL-FM-15 0 5 10 15 20 25 30 35 fame_dbms_fm 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 stack_fm 0 5 10 15 20 25 30 REAL-FM-7 0 2 4 6 8 10 12 REAL-FM-6 0 5 10 15 20 25 30 35 REAL-FM-19 0 10 20 30 40 50 60 arcade_game_pl_fm 0 20 40 60 80 REAL-FM-1 0 10 20 30 40 50 60 REAL-FM-20 0 2 4 6 8 10 12 aircraft_fm 0 5 10 15 20 25 connector_fm 0 2 4 6 8 10 12 14 REAL-FM-10 0 5 10 15 20 25 30 REAL-FM-14 8 10 12 14 16 REAL-FM-5 5 6 7 8 9 10 11 12 13 cfdp_library_fm 0 5 10 15 20 25 30 REAL-FM-13 0 5 10 15 20 25 REAL-FM-9 0 5 10 15 20 25 30 35 car_fm 0 5 10 15 20 25 30 REAL-FM-2 0 50 100 150 200 250 300 REAL-FM-4 0 5 10 15 20 25 30 REAL-FM-8 0 10 20 30 40 REAL-FM-18 0 5 10 15 20 25 30 REAL-FM-17 0 10 20 30 40 50 60 70 REAL-FM-11 012345 0 10 20 30 40 50 REAL-FM-16 012345 0 5 10 15 20 movies_app_fm 012345 0 10 20 30 40 50 60 smart_home_fm 012345 0 10 20 30 40 50 60 REAL-FM-3 012345 0 2 4 6 8 10 12 14 16 REAL-FM-12 012345 0 20 40 60 80 100 DELL-LAPTOP-NOTEBOOK-FM Features Tree relationships Cross-tree constraints Figure 7: Feature model properties present in the new variants only maintained those that provide some meaningful domain information - Ecological validity: We generated five variants per model as previous research was done (see Section 2). This is of extreme importance due to the stochastic nature of LLM, in which two different executions might not provide the same result. Internal validity The obtained models might depend on external factors such as model availability and access to the model. For this reason, we used the OpenAI and Cohere platforms as a black box in which we also tried to keep the prompt as clean as possible to avoid introducing fluctuations in the obtained results. 5 DISCUSSION AND FUTURE WORK In this study, we proposed an innovative approach to generating realistic variants of feature models taking advantage of LLMs such