A Neo4j Knowledge Graph for RAG Guidance Enforcement

A Neo4j Knowledge Graph for RAG Guidance Enforcement Dr. Andrew Flinders1, Brandon Campbell1 1Northrop Grumman Corporation, 5770 Missile Wy, Roy, UT 84067 Abstract This paper explores the demands of processing one or a few documents with absolute fidelity when presented with this scenario: given a guidance document, ensure that no sensitive data is present in a new dataset. In many NLP applications, processing numerous documents with high precision is often desirable, but not mandatory. However, our specific use case demands the processing of documents with nearly perfect precision. To achieve this, we have developed a knowledge graph implementation that checks for compliance with the guidance document. This knowledge graph must precisely mirror the content of the guidance document, necessitating the retention of the original text along with transformer-produced vector embeddings for Retrieval-Augmented Generation (RAG) [1] interpretations of the database contents on each node. Our technique, leveraging RAG, is broadly applicable to any scenario requiring strict data compliance with a guidance document. Key Words: Knowledge Graph, RAG 1. Introduction Recent advancements in Artificial Intelligence (AI), particularly large language models (LLMs), have expanded their use cases far beyond basic document retrieval and text generation. Many LLM use cases have become almost ubiquitous include finding a document most similar to a given example, answering specific questions, improving or summarizing text, writing introductions, and summarizing papers, etc. In many of these scenarios, precision is desired but not required; for instance, identifying the “most similar document” or delivering an “optimal answer” can be highly subjective. However, our use case demands a non-subjective process: one where every operation is clearly defined and verifiable, to wit, ensuring strict compliance with a guidance document. This requires that we not only retrieve and process the data in question but also rigorously enforce policies, such as detecting all instances of prohibited Personally Identifiable Information (PII) or combinations thereof. The process must log each violation with exact references to the corresponding sections of the guidance document, ensuring that the enforcement mechanism is transparent and verifiable. 2. Typical Use Cases and Precision Considerations LLMs offer a range of capabilities for general-purpose tasks. For example, a user may ask an LLM to: • Find a document similar to a given one. • Provide an answer to a specific question. • Improve a paragraph or write an introduction. • Summarize an existing document. While precision is always valued, subjective criteria such as what constitutes “the most similar” document or whether a paragraph was sufficiently improved remain vague. Our challenge, however, is to as much as possible define a non-subjective but more efficient process of textual analysis. This process includes detecting all instances of PII or forbidden combinations based on the guidance document, where every detected instance is logged with a reference to the rule that was violated. For instance, if the guidance document forbids the presence of Social Security Numbers (SSN) or disallows first and last names together, the system must identify and flag every occurrence as well as every combination of occurrences accordingly. 3. Process Overview Our approach is structured around a multi-step process that integrates a Neo4j knowledge graph [2] with a RAG agent to enforce each paradigm with as much consistency as possible. The process can be summarised as follows: First, the document proposed for review is encoded using a standard LLM model, thereby generating vector embeddings. Next, the system checks the knowledge graph for rules that are relevant to the content of the proposed document. These rules, which may include, for example, “all instances of SSN are forbidden” or “first name and last name should not appear together,” are used to formulate the RAG query. The query is then executed against a Generative AI agent to determine if any parts of the document violate the specified rules, and the resulting violations are documented back into the knowledge graph. In this way, our process enforces data compliance by linking concerning content to predefined guidelines. 4. Example: HR Email and Rule Enforcement Consider the following nominal example: Email Content: "Hey Andy! I heard you moved to that new place downtown! That is really exciting! I guess getting that raise really made a difference. I just need to check a couple of details before I can process your latest HR request. Your SSN is 123-45-6789 right? And your wife’s maiden name is Cooper? But she did take your last name of Flinders when you married, is that correct? Anyway, happy to help you with this! -Billy “HR” Johnson, The HR guy" In this example, the system is provided these rules by the similarity search: All instances of SSN are forbidden. All instances of a first name and last name together are forbidden. The process would then generate a RAG query such as: "Given these rules: 1. All instances of SSN are forbidden; 2. All instances of first name and last name together are forbidden. Are any parts of the email of concern? If so, please reference exactly which part of the email is violating which rule, and format the response as a table." This query attempts to coerce the model into providing an objective analysis of the text, mapping each violation to the corresponding rule from the guidance document (Fig. 1). 5. Additional Insights and Future Directions While our technique currently employs loose similarity thresholds that have proven successful so far, there is room for further refinement. Future work could involve: Tracking the frequency of violations by specific rules to better understand compliance trends. Implementing a user feedback system to adjust sensitivity thresholds, ensuring that rules are neither over-detected nor under-detected. Investigating methods to reduce potential hallucinations in generated responses, although current observations indicate few incidents of hallucination impacting performance. Exploring more sophisticated unsupervised methods for data extraction and model tuning. 6. Conclusions A guidance document mobilized by a knowledge graph can serve as an effective enforcement mechanism for ensuring data compliance. Our approach leverages RAG to integrate the original text and rule-based vector embeddings to yield a non-subjective, verifiable process. Although our method requires further development, the technique is very flexible and allows parameters and models to be tuned for specific use cases. Further insights can be obtained by tracking user interactions and compliance metrics over time, paving the way for iterative improvements that could eventually be applied to a broader set of documents. Figure 1: A representation of the Knowledge Graph used to support the RAG operations. Acknowledgements Thanks Northrop Grumman supported this research, although we should make it clear that the use case outlined in this paper is notional and is not currently being used by NGC. References [1] Lewis, M., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., ... & Riedel, S. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems, 33. URL: https://arxiv.org/abs/2005.11401 [2] Hogan, A., Blomqvist, E., Cochez, M., D’Amato, C., Melo, G. D., Gutierrez, C., … & Polleres, A. (2021). Knowledge Graphs. arXiv preprint arXiv:2003.02320. URL: https://arxiv.org/abs/2003.02320