D2.4 AMOE - v1

Deliverable D2.4 AMOE - v1 Editor(s): Franz Deimling (FABA) Responsible Partner: Fabasoft R&D GmbH Status-Version: Final – v1.0 Date: 31.10.2024 Type: OTHER (SW) Distribution level: PU D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 2 of 25 www.emerald-he.eu Project Number: 101120688 Project Title: EMERALD Title of Deliverable: AMOE - v1 Due Date of Delivery to the EC 31.10.2024 Workpackage responsible for the Deliverable: WP2 - Methodology for knowledge extraction Editor(s): Franz Deimling (FABA) Contributor(s): Angela Fessl (KNOW) Reviewer(s): Angela Fessl (KNOW) Cristina Martínez, Juncal Alonso (TECNALIA) Approved by: All Partners Recommended/mandatory readers: WP1, WP2, WP3, WP4 and WP5 Abstract: Interim evidence extraction from policy documents that can be integrated with the certification graph Keyword List: Evidence extraction, policy documents, organisational metrics Licensing information: This work is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0 DEED https://creativecommons.org/licenses/by-sa/4.0/) Disclaimer Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. The European Union cannot be held responsible for them. D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 3 of 25 www.emerald-he.eu Document Description Version Date Modifications Introduced Modification Reason Modified by v0.1 04.09.2024 First draft version Franz Deimling (FABA) v0.2 03.10.2024 Added text and figures to “Implementation” and “Delivery and usage” Franz Deimling (FABA) v0.3 07.10.2024 Updated references and texts Franz Deimling (FABA) v0.4 14.10.2024 Updated conclusions, executive summary, future work, references, formatting Franz Deimling (FABA) v0.5 15.10.2024 Quality Assurance Review Angela Fessl (KNOW) v0.6 28.10.2024 Addressed comments of Quality Assurance review Franz Deimling (FABA) v0.7 29.10.2024 Final review Cristina Martínez/ Juncal Alonso (TECNALIA) v0.8 30.10.2024 Revised document based on the final review Franz Deimling (FABA) v1.0 31.10.2024 Submitted to the European Commission Cristina Martínez/ Juncal Alonso (TECNALIA) D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 4 of 25 www.emerald-he.eu Table of contents Terms and abbreviations ............................................................................................................... 6 Executive Summary ....................................................................................................................... 7 1 Introduction ........................................................................................................................... 8 1.1 About this deliverable .................................................................................................... 8 1.2 Document structure ....................................................................................................... 8 2 Implementation ..................................................................................................................... 9 2.1 Functional description ................................................................................................... 9 2.1.1 Fitting into the overall EMERALD Architecture ................................................. 13 2.2 Technical description ................................................................................................... 14 2.2.1 Prototype architecture ...................................................................................... 14 2.2.2 Technical specifications ..................................................................................... 16 2.3 Testing and quality management of evidence extraction method ............................. 16 2.3.1 Annotation setup ............................................................................................... 16 2.3.2 Test setup .......................................................................................................... 17 2.4 Limitations and future work ........................................................................................ 18 3 Delivery and usage .............................................................................................................. 19 3.1 Package information .................................................................................................... 19 3.2 Installation instructions ............................................................................................... 19 3.3 Instructions for use ...................................................................................................... 20 3.4 Licensing information .................................................................................................. 23 3.5 Download ..................................................................................................................... 23 4 Conclusions .......................................................................................................................... 24 5 References ........................................................................................................................... 25 List of tables TABLE 1. AMOE.01 - UPLOAD PDF DOCUMENT .................................................................................... 9 TABLE 2. AMOE.02 - PROVISION OF EXTRACTED EVIDENCE TO EVIDENCE STORE ....................................... 10 TABLE 3. AMOE.03 - REFINE EVIDENCE EXTRACTION APPROACH ............................................................ 10 TABLE 4. AMOE.04 - COMPARE RESULTS FROM MULTIPLE DOCUMENTS .................................................. 11 TABLE 5. AMOE.05 - SELECT METRICS PER DOCUMENT. ........................................................................ 11 TABLE 6. AMOE.06 - CLASSIFY DOCUMENT, SELECT RESPECTIVE METRICS (OPTIONAL) ............................... 12 TABLE 7. AMOE.07 - METRIC STATES ................................................................................................ 12 TABLE 8. OVERVIEW OF AMOE'S SOURCE CODE PACKAGE CONTENTS ....................................................... 19 D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 5 of 25 www.emerald-he.eu List of figures FIGURE 1. EMERALD COMPONENT OVERVIEW DIAGRAM ...................................................................... 13 FIGURE 2. AMOE ARCHITECTURE DIAGRAM ......................................................................................... 14 FIGURE 3. SCREENSHOT OF ANNOTATIONS IN INCEPTION ..................................................................... 17 FIGURE 4. EMERALD UI MOCK-UP CONTAINING AMOE EVIDENCE DATA (D4.3 [8])................................. 21 FIGURE 5. EMERALD UI MOCK-UP FOR AMOE.05 (D4.3 [8]) ............................................................. 21 FIGURE 6. EMERALD UI SCREENSHOT DEPICTING A LIST OF AMOE FILES ................................................. 22 FIGURE 7. EMERALD UI SCREENSHOT DEPICTING A LIST OF EXTRACTED AMOE EVIDENCE FOR AN UPLOADED POLICY FILE ............................................................................................................................. 22 FIGURE 8. EMERALD UI SCREENSHOT DEPICTING CURRENT IMPLEMENTATION OF AMOE EVIDENCE VIEW .... 23 D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 6 of 25 www.emerald-he.eu Terms and abbreviations AI Artificial Intelligence AMOE Assessment and Management of Organisational Evidence API Application Programming Interface CPU Central Processing Unit CSP Cloud Service Provider DB Database DoA Description of Action EC European Commission GA Grant Agreement to the project GPU Graphical Processing Unit GUI Graphical User Interface ID Identifier HTML Hypertext Markup Language IaaS Infrastructure as a Service KPI Key Performance Indicator KR Key Result MEDINA Predecessor project of EMERALD NLP Natural Language Processing nDCG normalized Discounted Cumulative Gain PaaS Platform as a Service PDF Portable Document Format QA Question Answering RCM Repository of Controls and Metrics SW Software TRL Technology Readiness Level UI User Interface WP Work Package D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 7 of 25 www.emerald-he.eu Executive Summary This deliverable presents the initial design, architecture, and implementation state of the Assessment and Management of Organisational Evidence (AMOE) component, an evidence extractor for policy documents. The main contributions are related to the key result KR1EXTRACT of EMERALD, a framework to continuously extract knowledge from different layers of a cloud service and prepare suitable evidence based on them. The policy document evidence extractor, developed in Task 2.3 and described in this deliverable, aims at identifying relevant text segments related to security related features, as defined in the respective EMERALD metrics based on specific controls and security requirements of various security schemes. The extracted evidence is stored in the EMERALD Evidence Store. Other related deliverables in WP2, all due at project month 12 (October 2024), provide functional and technical details on further evidence extractors from different sources, i.e., D2.2 [1] on source code evidence extraction in Task 2.2, D2.6 [2] on security and privacy preserving evidence extraction in Task 2.4, D2.8 [3] on runtime data extraction in Task 2.5. All these details contributed to D2.1 [4] on the overall information model of the certification graph in Task 2.1. This document starts by illustrating how the policy document evidence extractor fits into the overall EMERALD architecture. The main part provides functional and technical descriptions of the evidence extractor AMOE, including its purpose and scope, the (current and planned) coverage of the EMERALD requirements, the components’ internal architecture and their subcomponents. These descriptions are complemented by information on delivery and usage, as well as on limitations and future work. Finally, the document concludes with a short summary. Based on the work described in this deliverable, the policy document evidence extractor will be further extended and integrated into the EMERALD framework. This is the first iteration of the deliverable coming from Task 2.3. The second and final version of this deliverable (D2.5 [5]) with the updated extractor will be delivered in project month 24 (October 2025). D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 8 of 25 www.emerald-he.eu 1 Introduction EMERALD aims to offer a suite of tools and techniques for evidence collection, leveraging a knowledge graph-based approach. KR1-EXTRACT facilitates a unified, tool-supported methodology for continuously extracting knowledge across various layers of a cloud service— such as infrastructure, platform, runtime data, policy documents, software, and AI models. The goal of WP2 is to develop a cohesive view of the cloud service being certified by extracting and enriching knowledge from these layers and generating relevant evidence for security metrics. A key focus of this work package is the research and design of tools and techniques to extract knowledge from diverse sources. Central to this is the Evidence Store, utilizing a graphbased model that acts as a common structure, populated by all evidence extraction tools with evidence 1 . 1.1 About this deliverable The goal of this deliverable is to present the EMERALD evidence extractor tool AMOE and how it is integrated into the EMERALD framework. This report reflects the current prototype of AMOE, which was originally launched in MEDINA 2 . In EMERALD, it should be advanced to a higher TRL and improved to verify that the functionality is adapted to the needs of the EMERALD pilot use cases. EMERALD follows a knowledge graph-based approach to provide a unified view of the cloud service under certification at different layers of the service. The different evidence extraction tools are ranging from the infrastructure layer (e.g., virtual resources), to the business layer (e.g., policies and procedures), to the implementation layer (e.g., source code files), and the data layer (e.g., increasingly used AI models) in cloud applications. AMOE focuses on providing evidence based on policy documents which shall be included into the whole automated certification process. This deliverable will give insights into the technical and functional approach that AMOE uses to support the key results of the project (e.g. the use cases demonstrated by the pilots as well as the technical integration via the certification-graph and evidence extraction workflows of EMERALD). 1.2 Document structure The document is structured as follows. In Section 2, the functional and technical descriptions of AMOE are described. This covers the requirements for AMOE in the EMERALD project as well as the AMOE architecture description and how it fits into the whole EMERALD architecture. Furthermore, an overview of the testing and quality management for evidence extraction is described alongside the annotation setup. Additionally, this section includes limitations and future work to be commenced regarding AMOE. Section 3 focuses on the delivery and usage of AMOE. First the package and its contents are described, followed by installation and deployment instructions. Second, this section also provides some instructions of use, licensing information, and where to download the current public version. The deliverable is concluded in Section 4, followed by some references in Section 5. 1 For details consult the AMOE and the Evidence Store data model presented in deliverable D1.1 [9] 2 https://medina-project.eu/ D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 9 of 25 www.emerald-he.eu 2 Implementation The following subsections provide functional and technical descriptions of AMOE. 2.1 Functional description Overall purpose. AMOE is based on a prototype developed in a previous project called MEDINA2. It is designed to extract evidence based on metrics, which target specific parts of policy documents. After the extraction process, the evidence can be inspected in a GUI (Graphical User Interface) that comes with AMOE or retrieved via the API. Once the evidence results have been reviewed by a user, they can be forwarded to the EMERALD framework. Context, scope and motivation. AMOE allows to transform the organisational process of checking policy documents for their content into a technical process. Text passages can be checked against predefined goals and target values. As policy documents are rather static, compared to other evidence gathered (e.g. log files, runtime information), the evidence gathering is done once per document for a specific set of metrics and target values. In the case updates are required, the new document is processed, and additional evidence is produced. The policy evidence results are integrated into the EMERALD audit process via submission to the Evidence Storage component and subsequent processing in the Assessment component. Requirements. The relevant requirements from D1.3 [6] with their respective implementation state (partially / fully / not implemented) and a brief description of how they are / will be implemented are given in Table 1 to Table 7. Table 1. AMOE.01 - Upload PDF document Field Description Requirement ID AMOE.01 Short title Upload PDF document. Description The component shall be able to receive a PDF document via API and process its contents regarding the defined metrics. The PDF shall receive a unique ID so that it can be retrieved and deleted later on. Status Work in Progress Priority Must Component AMOE Source Component Type Technical Related KR KR1_EXTRACT, KR2_CERTGRAPH, KR8_PILOTS Related KPI KPI 1.1 Validation acceptance criteria The user can upload a document via API. The user shall be able to retrieve document meta data by using the unique id that is returned on successful upload. The process shall finish in reasonable time. Progress 90% Milestone MS2: Components V1 (M12) AMOE provides the functionality to upload a PDF document via its API. At the current implementation status, the processing is started immediately after the upload of the document is completed and a unique file ID is returned. After the metrics have been processed for the document, the extracted results can be retrieved using the unique file ID. If AMOE.05 is implemented, the processing is only done for a set of selected metrics – and started on demand (not directly after the upload). D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 16 of 25 www.emerald-he.eu taska trained model of a QA system can provide answers to a question, given a question and text as input. The top answer of the QA is provided by the system. More details and alternative approaches tested (e.g. using cosine-similarity), are described in the paper [8]. QA quality checks The question answering (QA) quality check subcomponent enables the user to compare the extracted evidence (using e.g. the keyword-based approach) with the annotations exported from the INCEpTION tool. See Section 2.3 for details on the quality management process. 2.2.2 Technical specifications The AMOE tool is written in Python >=3.12. It uses various Python libraries as well as the pdftohtml functionality from poppler utils 10 . The webservice is built on Quart 11 , the evidence extraction is based on transformers 12 , PyTorch 13 and the roberta-base-squad2 14 model from huggingface. The component is using MongoDB 15 and Redis 16 to store the data. Evidence and logs are stored in the MongoDB. Redis is used in par with the quart-session library. 2.3 Testing and quality management of evidence extraction method To test the evidence extraction method used in AMOE, the extracted data is compared to some predefined target values. This allows us to compute some scores that can be used to adjust the settings of the approach. The quality management is thus split into two parts: 1) the annotation setup and 2) the execution of tests, evaluation, and analysis of the results. 2.3.1 Annotation setup This section describes the data annotation process and preparational steps. First, the policy documents (PDFs) need to be gathered. Also, the annotation software INCEpTION 17 needs to be set up and running. Then an annotation project can be set up by adding the list of metrics to the tag set and configuring the layers to be annotated. The tag set for the metrics can be generated using an excel list/csv containing the metric ids with the support of the python program supplied in the AMOE source code “src/extract_metric_tag_list.py”. After the INCEpTION project is set up, the policy documents can be uploaded, and users can annotate the files. To annotate, the text must be selected and afterwards a metric can be assigned. If the metric or annotation is hovered, more information is displayed such as the description that has been configured in the tag set. Once the document has been annotated, the curation process can be applied to ensure high quality data. In the curation step, multiple annotations by different users are combined into a single source of truth, which can be exported and used for the quality assessment of the evidence extraction approach. Figure 3 depicts a 10 https://poppler.freedesktop.org/ 11 https://pypi.org/project/quart/ 12 https://github.com/huggingface/transformers 13 https://pytorch.org/ 14 https://huggingface.co/deepset/roberta-base-squad2 15 https://www.mongodb.com/ 16 https://redis.io/ 17 https://inception-project.github.io/ D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 17 of 25 www.emerald-he.eu screenshot of the annotation view in INCEpTION. In the specific case shown the annotated text samples are highlighted in green. Figure 3. Screenshot of annotations in INCEpTION 2.3.2 Test setup This section describes the possible tests to be commenced in the project to check the performance of the evidence extraction approach and tune it to the metrics/documents of the EMERALD project. The exported annotations (ground truth) can be compared to the results of AMOE. The most basic score that can be computed is based on the number of matches vs the total number of metrics annotated. The evidence extraction approach shall be evaluated by computing this score per pilot. For example, if for a document 28 metrics have been annotated in INCEpTION (#of annotated evidence) and AMOE retrieves the correct answer for 19 metrics (#correctly retrieved evidence), the resulting score would be ca. 0.68. The pilot documents differ depending on the concrete use case (IaaS, PaaS, …), the language, and the formatting. Different scores will be analysed to get a good overview of the different domains and the performance overall. The question answering model (roberta-base-squad2) can retrieve multiple answers that can be ranked by the score associated. This score can be interpreted as a kind of probability for how likely the answer is to be correct. These ranked answers can be used to compute scores like the nDCG 18 - normalized Discounted Cumulative Gain, which can help in tuning the approach to rank relevant answers higher. This score has so far not been applied for AMOE, as the current 18 https://en.wikipedia.org/wiki/Discounted_cumulative_gain D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 18 of 25 www.emerald-he.eu implemented approaches provide a single answer. The main benefit of the nDCG will be in the improvement of AMOE evidence extraction approaches as some interim results can be tuned. 2.4 Limitations and future work One key challenge is the limited availability of data, which restricts the system’s ability to provide comprehensive answers across diverse domains. Additionally, language-related limitations arise, as the model may struggle to extract relevant responses in languages it has not been trained on. Privacy concerns also play a significant role, particularly since the tool avoids using pre-trained publicly available AI models due to data protection requirements, further constraining the variety and quality of data inputs. The models are selected on the basis of their license as well as local operability – to make sure, the data does not have to leave the premise. The prototype operates under limited GPU and CPU resources, which hinders the processing speed and scalability of the system, affecting its overall efficiency and performance when handling complex queries or large datasets. The processing time of the current evidence extraction approach is dependent on the input size – longer documents potentially take longer to process. However, this is somewhat mitigated by applying keywords to reduce the search space (see also [8]). AMOE is designed to support in the assessment and management of policy documents, but not to fully automate the assessment. With the current design, no guarantee can be given that the AI models would retrieve the correct evidence 100%. Given the limited data set provided to the project by the different partners, results might be biased for specific use cases/metrics that are only relevant to some of the partners. The performance will be evaluated given the provided resources, since the focus of the project is on innovation rather than research. At the current state of the project, not all AMOE requirements have been implemented. AMOE.07 lists a few possible metric states that could be added to improve the traceability and usability of metrics. Furthermore, at the current state of AMOE it is impossible to determine for sure whether the target of a metric is contained in a document. To limit the processing resources and time waiting for results and, most of all, to extract more precise results the implementation of AMOE.05 is planned – selecting metrics per file to avoid processing non-sensical data. The largest remaining requirement is AMOE.03 – the improvement of the evidence extraction approach. For the remainder of the project AMOE.01, AMOE.02 and AMOE.04 will be worked on and AMOE.06 is considered optional to be addressed if enough resources are available and other requirements are completed. The AMOE GUI will not be updated further as it will be integrated into the EMERALD UI. D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 19 of 25 www.emerald-he.eu 3 Delivery and usage The following sections give a short overview of the delivery and usage of the tool. 3.1 Package information AMOE can be deployed as a Docker container. Table 8 shows an overview of the repository folders and files. Table 8. Overview of AMOE's source code package contents Folder Description / The root folder contains some helper scripts and configurations needed to build and run AMOE (e.g. Dockerfile). /kubernetes/ Contains the Kubernetes19 files for the deployment of AMOE. /metric_data/ Contains the local version of the metrics. /src Contains the source code of AMOE /src/paragraph_extraction/ Contains the code for the pre-processing pipeline. /src/qa/ Contains the code for evidence extraction using the question answering model as well as code to compute quality scores. /src/static/ Contains the stylesheets and images for the webservice. /src/templates/ Contains the HTML templates for the webservice. /src/utils/ Contains code for utility functions of the webservice such as use of other EMERALD component’s API, evidence extraction and database management. /tests/ Contains the tests for the AMOE API /clouditor-evidence-client/ Contains the repo for the generated Python client for the Evidence Store API based on their OpenAPI file. /orchestrator-client/ Contains the repo for the generated Python client for the Orchestrator API based on their OpenAPI file. /rcm-client/ Contains the repo for the generated Python client for the Repository of Controls and Metrics API based on their OpenAPI file. 3.2 Installation instructions Clone the AMOE repository. Set up a MongoDB and a Redis instance (see Kubernetes files in the repository). Set the following environment variables or variables directly in the config.py: • MONGODB_URL • MONGODB_PORT • MONGODB_USER • MONGODB_PASSWORD • REDIS_SERVICE • REDIS_PASS 19 https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 20 of 25 www.emerald-he.eu • REDIS_PORT • KEYCLOAK_URL • KEYCLOAK_REALM • KEYCLOAK_CLIENT_ID • KEYCLOAK_CLIENT_SECRET • KEYCLOAK_USER • KEYCLOAK_PASSWORD Optionally set (needed to deploy in production with EMERALD components): • CATALOGUE_API_URL • ORCHESTRATOR_API_URL • ALLOWED_ORIGINS Run python3 -m hypercorn -b 0.0.0.0 "src.app:create_app()" to deploy the service locally, or deploy with Kubernetes. 3.3 Instructions for use Instructions for use are included in the AMOE’s public GitLab repository 20 . The OpenAPI documentation can be found in the code repository in GitLab as well as retrieved for every deployment by accessing <amoe-server-url>/docs. Figure 4 and Figure 5 display mock-ups for the EMERALD UI containing information that is or will be provided by AMOE. The first depicts how the implementation in the EMERALD UI could look like to upload a policy file to AMOE as well as how the evidence data for a certain metric could be displayed. The second figure provides a draft for the functionality of AMOE.05 – which will allow to select a set of metrics to be retrieved for a file in AMOE. For testing purposes of the AMOE integration in the EMERALD UI, some pages were recreated like the overview page of uploaded documents (Figure 6), a page listing the metadata of an uploaded file and processed evidences – filtered by search term “password” (Figure 7) and a page allowing to set the assessment status and compliance comment while investigating the extracted evidence (Figure 8). 20 https://git.code.tecnalia.com/emerald/public/components/amoe-assessment-and-management-oforganizational-evidence D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 21 of 25 www.emerald-he.eu Figure 4. EMERALD UI mock-up containing AMOE evidence data (D4.3 [9]) Figure 5. EMERALD UI mock-up for AMOE.05 (D4.3 [9]) D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 22 of 25 www.emerald-he.eu Figure 6. EMERALD UI screenshot depicting a list of AMOE files Figure 7. EMERALD UI screenshot depicting a list of extracted AMOE evidence for an uploaded policy file D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 23 of 25 www.emerald-he.eu Figure 8. EMERALD UI screenshot depicting current implementation of AMOE evidence view 3.4 Licensing information The component is planned to be licenced under Apache 2.0. 3.5 Download The component code can be downloaded from the EMERALD public Gitlab repository 21 . 21 https://git.code.tecnalia.com/emerald/public/components/amoe-assessment-and-management-oforganizational-evidence D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 24 of 25 www.emerald-he.eu 4 Conclusions AMOE is designed to extract relevant information based on EMERALD metrics from different policy documents provided by the project partners. In this deliverable, the technical report of the EMERALD evidence extraction component AMOE is presented. The functional description and how AMOE fits into the general EMERALD framework is described. Furthermore, a list of sub-components is given and how they interact. AMOE is using the APIs of other EMERALD components - the Repository of Controls and Metrics to retrieve information of the metrics and security schemes, the Orchestrator to retrieve specific target values and metric configurations, and the Evidence Store to integrate the extracted results into the EMERALD framework. The clients are generated using the respective OpenAPI files. AMOE is built using Python and different NLP libraries and pre-trained AI models. The basic requirements are under development and the functionalities are offered to the EMERALD UI via a dedicated AMOE API. The open requirements are planned for the upcoming project period and described in the second version of this deliverable D2.5 [5] in M24 (October 2025). D2.4 – AMOE - v1 Version 1.0 – Final. Date: 31.10.2024 © EMERALD Consortium Contract No. GA 101120688 Page 25 of 25 www.emerald-he.eu 5 References [1] EMERALD Consortium, “D2.2 Source Evidence Extractor – v1: Evidence extraction from source code that can be integrated with the certification graph,” 2024. [2] EMERALD Consortium, “D2.6 ML model certification – v1: Security and privacy preserving evidence that can be integrated with the certification graph,” 2024. [3] EMERALD Consortium, “D2.8 Runtime evidence extractor – v1: Evidence extraction from runtime data that can be integrated with the certification graph,” 2024. [4] EMERALD Consortium, “D2.1 Graph Ontology for Evidence Storage: Description of a uniform schema for storing and linking heterogenous data,” 2024. [5] EMERALD Consortium, “D2.5 AMOE–v2,” 2025. [6] EMERALD Consortium, “D1.3 EMERALD solution architecture - v1,” 2024. [7] MEDINA Consortium, “D3.6 - Tools and techniques for collecting evidence of technical and organisational measures-v3,” 2023. [8] F. Deimling and M. Fazzolari, “AMOE: A Tool to Automatically Extract and Assess Organizational Evidence for Continuous Cloud Audit,” In: Atluri, V., Ferrara, A.L. (eds) Data and Applications Security and Privacy XXXVII. DBSec 2023. Lecture Notes in Computer Science, vol 13942. Springer, Cham. https://doi.org/10.1007/978-3-031-37586-6_22, 2023. [9] EMERALD Consortium, “D4.3 User interaction and user experience concept – v1,” 2024. [10] EMERALD Consortium, “D1.1 Data modelling and interaction mechanisms - v1,” 2024.