Supplementary material: Enhancing Educational Support for JetBrains MPS with a Retrieval-Augmented LLM Chatbot: A Structured Knowledge Integration Approach

A Evaluation Questions and Expected Answers The following questions and their corresponding expected answers were used to evaluate the performance of the LLM using the RAGAS framework. These answers represent the reference content used to compute similarity and factuality scores. The model-generated answers are not included here, as they are reflected in the RAGAS evaluation scores presented in the main paper. Question 1 What is the generator configuration pattern? Expected Answer: Separating generation configuration from the model content is an effective pattern where the model contains the data to be generated, and separate configuration models—like mbeddr’s BuildConfiguration—specify how generation should occur. During generation, relevant content is copied into the configuration model, simplifying the generator’s task and allowing concurrent generation of different outputs since each output has its own model and only requires read access to the original content. Configurations can reference inputs either by entire models or specific root nodes, offering flexibility and control over the generation process. Question 2 For the generator, do we prefer Switches over Ifs? Expected Answer: We do prefer Switches over Ifs for the generator. Question 3 What are the disadvantages of using exceptions for error handling in MPS generator? Expected Answer: Generation stops immediately on an error, providing only a single error message and forcing users to fix one error at a time—a frustrating process due to repeated regenerations. Additionally, it’s difficult to trace the error back to the original input because exceptions lack detailed debugging information, making it hard to identify the problematic node even with transient models enabled. Question 4 What does maintainability of the generator mean? Expected Answer: On the one hand, it is about the ability to change, delete, or rewrite parts of the generator chain without affecting parts prior or later in the chain. On the other hand, it’s about writing readable generators that can easily be understood and reasoned about. 1 B Evaluation Questions – Automatically Generated and Manually Corrected The following questions were used for evaluating generator maintainability and design decisions: 1. What is a potential reason for MPS generators to avoid transforming nodes when they are used in tests, as mentioned in the document? 2. Based on the document, what is the significance of detecting tests in the context of maintaining MPS generators? 3. Explain the purpose and functionality of the ‘IsInTestsExpression‘ in the ‘com.mbeddr.mpsutil.blutil.genUtil‘ language. How does it help in managing complex generators that depend on context? 4. Discuss the advantages of using preprocessing scripts in Java for model transformation over reduction or weaving rules in the context of inputoutput structure differences. Provide an example scenario where preprocessing would be beneficial. 5. Explain the role of preprocessing in the generation of structured outputs, such as XML files, and discuss the advantages and disadvantages of using preprocessing scripts in this context. 6. Discuss the challenges associated with error handling during the generation process in MPS generators, particularly focusing on the use of exceptions and their impact on the user experience. 7. Explain the disadvantages of using exceptions for error handling during generation in MPS generators. How does the alternative approach using ‘genContext.show error‘ improve the error handling process? 8. Discuss the limitations of using a ‘LOOP ‘ macro or performing transformations ”in place” when generating output from a list of items or a single child node. How might these limitations affect the extensibility of the model? 9. Explain the limitations of using a ‘LOOP ‘ macro for generating output from a list of items or a single child node, and describe how the combination of ‘COP Y SRC‘/‘COP Y SRCL‘ and a reduction rule can overcome these limitations. 10. In the context of extending the behavior of a generator, why might it be impossible to include a singleton object into an ‘arraylist‘ created using a ‘LOOP ‘ macro? Provide an example scenario where this limitation could affect the extensibility of a generator. 2 11. Explain the limitations of using the ‘LOOP ‘ macro in the context of extending a generator, and describe how the rewritten code addresses these limitations. 12. Discuss the advantages of using a ‘SW IT CH‘ macro over an ‘IF ‘ macro when generating output based on a condition that is not a simple boolean property. Provide an example from the context to support your explanation. 13. Explain the advantages of using the ‘SW IT CH‘ macro over the ‘IF ‘ macro in cases where the condition is based on a model element other than a boolean property. Provide an example from the mebddr codebase to support your explanation. 14. In the context of code generation, what potential issue might arise when using an ‘IF ‘ macro to handle different types of ‘Evaluator‘ concepts? How can this issue be addressed to ensure the code generation process remains flexible and adaptable to new ‘Evaluator‘ types? 15. Explain how the concept of a ”template switch” is utilized in the context of introducing a custom ‘Evaluator‘ and describe its extensibility feature with an example. 16. Discuss the role of a default rule in a template switch and explain what happens if a template switch does not have a default case, as mentioned in the provided context. 17. Explain the purpose and functionality of the ”extends” relationship in the context of template switches, as described in the provided document. How does it affect the behavior of the original switch? 18. Discuss the limitations and potential issues associated with using predefined generator plans in the context of language extensibility and testing, as outlined in the document. Why might this approach be less favorable compared to dynamic generation plans with priorities? 19. Discuss the scenarios where using predefined generation plans in MPS might be beneficial, despite the limitations on extensibility. Why might extensibility be less of a concern or even undesirable in these contexts? 20. Explain the challenges associated with writing large sets of generators and languages that build on top of each other, as mentioned in the document. How can defining logical checkpoints or barriers in generator priorities help address these challenges? 21. Based on the Mbeddr example provided in the document, discuss how the use of over 30 generators can be managed effectively. What role do generator priorities play in ensuring the correct execution order, and how does this contribute to easier debugging? 3 22. Explain the concept of logical checkpoints/barriers in generator priorities and discuss their significance in managing cross-generator dependencies. How do these barriers aid in debugging? 23. Describe the role of the ‘mbeddr.modules.gen‘ layer in the mbeddr generator hierarchy. What is its input and output, and how does it relate to higher abstraction layers such as the ‘mbeddr.unittest‘ language? 24. Explain the role and function of the ”mbeddr.modules.gen” layer in the mbeddr generator architecture. How does it interact with higher-level abstractions and what is its significance in the generation process? 25. Describe the relationship between the higher-level abstractions and the middle layer in the mbeddr generator architecture. How do these layers ensure that the correct generation order is maintained, and what mechanisms are in place to debug the generation plan if transformations are not applied correctly? 26. Explain the role of the ‘mbeddr.statemachines‘ and ‘mbeddr.components‘ generators in the context of higher-level abstractions, and how their execution priorities are determined relative to the ‘modules.gen‘ generator. 27. Describe the purpose of the ”Maintainable MPS Generators” guide and identify the main sections outlined in its Table of Contents. How does this guide aim to assist developers working with MPS generators? 28. Explain the challenges associated with generating multiple independent outputs from a single input model, as discussed in the document. How does this affect the reuse of generators? 29. Describe the ”Generator Configuration Pattern” mentioned in the document. How does this pattern help in separating generation from the model, and what are its benefits in managing multiple outputs? 30. Explain the concept of the ”Generator Configuration Pattern” as described in the context. How does it help in maintaining loose coupling between generators and what are its benefits in terms of concurrent output generation? 31. Discuss the challenges associated with a naive solution of copying and reducing nodes in each generator. Why does this approach lead to tightly coupled generators, and how does it affect the reuse of generators? 32. Explain the process that occurs during the generation of a model in the context provided. What role does the initial copying of content play, and how does it affect the subsequent steps in the generator chain? 33. Discuss the different ways generator configurations can refer to the input model for generation. What are the implications of using a complete model reference versus specifying root nodes, and how does this choice impact the generation process? Provide examples where applicable. 4 34. Explain the role of ‘ModelRefExpression‘ in the context of generating references to root nodes of a model. How does it facilitate the handling of dependencies in the generation process? 35. In the implementation of generators, what is the significance of the line ‘nlist¡¿ copies = genContext.copy with trace nodes2copy;‘? Discuss how this approach differs from iterating over nodes and calling ‘.copy‘ on them individually. 36. Explain the advantage of using the ”copy with trace” method over iterating and calling the .copy method on nodes individually in the context of MPS (Meta Programming System). 37. Why is it important for references in the generated model to point within the model itself rather than to nodes from the original input when using the ”copy with trace” method? 38. Explain the advantage of using the ”copy with trace” method in MPS generators over manually copying nodes one by one. How does this method improve the process of handling references in the generated model? 39. Based on the context provided, what is the significance of ensuring that references in the generated model point to the copied nodes rather than the original input nodes? How does this relate to the use of mapping labels during generation? 5