Appendix to "Graph RAG for Automated Short Answer Grading with Feedback: Bridging Pedagogical Needs and Technical Capabilities"

Appendix A. Experimental Configuration (Reproducibility) • LLM (KG extraction): GPT-4o-mini (JSON mode), temperature = 0, seed = 42 • LLM (grading & feedback): GPT-4o (JSON mode), temperature = 0, seed = 42 • Embeddings: OpenAI text-embedding-3-small (1536-d) • Graph DB: Neo4j 5.x with APOC • Retrieval (GraphRAG): Vector-Cypher (top-k = 3 seed nodes + 2 neighbors, 1–2 hops) • Length constraints: 30 / 50 words • Score granularity (bins): {0.10, 0.125, 0.20, 0.25} • Decoding: Deterministic（no sampling; JSON mode） Appendix B. Knowledge Graph Schema Nodes Key Concept; Required Element; Answer Structure; Answer Component; Expected Response; Feedback Type; Common Mistake; Evaluation Criteria; Improvement; Suggestion Relationships Require, Evaluates, Weights, Contains, Precedes, Supports, Indicates, Corrects, Suggests Appendix C. Two Prompt Templates for KG Construction (Full Text) C1 Knowledge Extraction Prompt Template You are an educational assessment expert tasked with extracting key concepts and knowledge points from questions and reference answers. Extract the entities (nodes) and specify their type from the following Input text. Also extract the relationships between these nodes. The relationship direction goes from the start node to the end node. Return result as JSON using the following format: {"nodes": [ {"id": "0", "label": "the type of entity", "properties": {"name": "name of entity" }} ], "relationships": [{"type": "TYPE_OF_RELATIONSHIP", "start_node_id": "0", "end_node_id": "1", "properties": {"details": "Description of the relationship"}} ] } - Use only the information from the Input text. Do not add any additional information. - If the input text is empty, return empty Json. - Make sure to create as many nodes and relationships as needed to offer rich educational context. - An AI knowledge assistant must be able to read this graph and immediately understand the key concepts and their relationships. - Multiple documents will be ingested from different sources and we are using this property graph to connect information, so make sure entity types are fairly general. Use only the following nodes and relationships (if provided): {schema} Assign a unique ID (string) to each node, and reuse it to define relationships. Do respect the source and target node types for relationship and the relationship direction. Do not return any additional information other than the JSON in it. Examples: {examples} Input text: {text} Appendix C. Two Prompt Templates for KG Construction (Full Text) C2 Grading Logic Prompt Template You are an educational assessment expert tasked with analyzing student answers and feedback to understand the grading mechanism. Extract the entities (nodes) and specify their type from the following Input text. Also extract the relationships between these nodes. The relationship direction goes from the start node to the end node. Return result as JSON using the following format: {"nodes": [ {"id": "0", "label": "the type of entity", "properties": {"name": "name of entity" }} ], "relationships": [{"type": "TYPE_OF_RELATIONSHIP", "start_node_id": "0", "end_node_id": "1", "properties": {"details": "Description of the relationship"}} ] } - Use only the information from the Input text. Do not add any additional information. - If the input text is empty, return empty Json. - Make sure to create as many nodes and relationships as needed to offer rich grading context. - An AI knowledge assistant must be able to read this graph and immediately understand the evaluation criteria and scoring patterns. - Multiple documents will be ingested from different sources and we are using this property graph to connect information, so make sure entity types are fairly general. Use only the following nodes and relationships (if provided): {schema} Assign a unique ID (string) to each node, and reuse it to define relationships. Do respect the source and target node types for relationship and the relationship direction. Do not return any additional information other than the JSON in it. Examples: {examples} Input text: {text} Appendix D. Grading and Feedback Prompt for GraphRAG: Generating Grades and Feedback You are an educational assessment expert evaluating a student's answer based on reference materials. Based on the context information and the student's answer, assign a score and provide feedback. # New Answer: {query_text} # Reference Context: {context} # Evaluation (here we use 0.125 increments and 30 words limits): Provide an accurate score from 0.0 to 1.0, using increments of 0.125 (e.g., 0.0, 0.125, 0.25,0.375,0.5,0.625,0.75,0.875,1.0),and provide detailed feedback. Return your evaluation as JSON using the following format: {{"score": 0.0-1.0, "feedback": "<≤30 words detailed and concise critique-feedback>"}} - Use only the information from the Context to inform your evaluation. - Be fair and consistent in your scoring. Do not return any additional information other than the JSON. Appendix E. Example of Traceable Knowledge Graph. Questions: State at least 4 of the differences shown in the lecture between the UDP and TCP headers. Student Answer: A UDP header has a length of 8 bytes whereas a TCP header has a length of 20 bytes. A UDP header has a field for the packet length, unlike a TCP header. A UDP header doesn’t contain a sequence number, while a TCP header does. A UDP header neither contains an acknowledgement number but a TCP header has an extra field for that. Reference Answer: Possible Differences: The UPD header (8 bytes) is much shorter than the TCP header (20-60 bytes); The UDP header has a fixed length while the TCP header has a variable length Fields contained in the TCP header and not the UDP header : Sequence number; Acknowledgment number; Reserved; Flags/Control bits; Advertised window; Urgent Pointer; Options + Padding if the options are; UDP includes the packet length (data + header) while TCP has the header length/data offset (just header) field instead; The sender port field is optional in UDP, while the source port in TCP is necessary to establish the connection Human feedback: The response correctly identifies and states the four differences between TCP and UDP headers except that the TCP header can be between 20 and 60 bytes long. Graph RAG feedback: The student correctly identifies four differences, but inaccurately states TCP header length as fixed at 20 bytes. Appendix F. Actionable Verbs’ Pedagogical Lexicon Educational keywords improve, suggest, recommend, consider, try, focus, strength, weakness, good, well done, excellent, needs work, specific, example, detail, clarify, explain, develop Constructive keywords because, since, therefore, however, although, for example, specifically, in particular, furthermore, additionally