D1.1 Data modelling and interaction mechanisms - v1

Deliverable D1.1 Data Modelling and interaction mechanisms – v1 Editor(s): Franz Deimling Responsible Partner: Fabasoft R&D GmbH Status-Version: Final - v1.0 Date: 31.07.2024 Type: R Distribution level (SEN, PU): PU D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 2 of 39 www.emerald-he.eu Project Number: 101120688 Project Title: EMERALD Title of Deliverable: D1.1 - Data Modelling and interaction mechanisms – v1 Due Date of Delivery to the EC 31.07.2024 Workpackage responsible for the Deliverable: WP1 - Concept and methodology of EMERALD Editor(s): Franz Deimling (FABA) Contributor(s): CNR, FABA, FhG, SCCH, TECNALIA Reviewer(s): Iñaki Etxaniz, Gorka Benguria Elguezabal, Cristina Martínez (TECNALIA) Approved by: All Partners Recommended/mandatory readers: WP1, WP2, WP3, WP4, WP5 Abstract: Initial version of the overview of data models and techniques used for creating and linking the data to evidence (annotation, etc) Keyword List: Data diagram, data model, component overview Licensing information: This work is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0 DEED https://creativecommons.org/licenses/by-sa/4.0/ Disclaimer: Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. The European Union cannot be held responsible for them. D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 3 of 39 www.emerald-he.eu Document Description Version Date Modifications Introduced Modification Reason Modified by v0.1 27.03.2024 First draft version FABA v0.2 27.06.2024 Comments and suggestions received by consortium partners WP1 partners v0.3 30.06.2024 Contributions from component partners added WP2 and WP3 partners v0.4 09.07.2024 Figures and listings updated FABA, Tecnalia, FhG v0.5 19.07.2024 QA Review TECNALIA v0.6 24.07.2024 Addressed all comments received in the Internal QA review FABA v1.0 31.07.2024 Submitted to the European Commission TECNALIA D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 4 of 39 www.emerald-he.eu Table of contents Terms and abbreviations ............................................................................................................... 6 Executive Summary ....................................................................................................................... 7 1 Introduction ........................................................................................................................... 8 1.1 About this deliverable .................................................................................................... 8 1.2 Document structure ....................................................................................................... 8 2 Data Model Overview ............................................................................................................ 9 3 Component Data Models .................................................................................................... 11 3.1 Evidence Collector Data Models .................................................................................. 13 3.1.1 AI-SEC ................................................................................................................. 13 3.1.2 AMOE ................................................................................................................. 14 3.1.3 Clouditor-Discovery ........................................................................................... 15 3.1.4 Codyze ............................................................................................................... 17 3.1.5 eknows ............................................................................................................... 18 3.2 Trustworthiness System (TWS) Data Model ................................................................ 20 3.3 Mapping Assistant for Regulations with Intelligence (MARI) Data Model .................. 21 3.4 Repository of Controls and Metrics (RCM) Data Model .............................................. 22 3.5 Orchestrator Data Model............................................................................................. 25 3.6 Evidence Store Data Model ......................................................................................... 27 3.7 Assessment Data Model .............................................................................................. 27 3.8 Evaluation Data Model ................................................................................................ 28 4 Interactive Documentation ................................................................................................. 30 4.1 PlantUML ..................................................................................................................... 30 4.2 Web Service ................................................................................................................. 30 4.2.1 Implementation details ..................................................................................... 31 4.3 Data model versioning ................................................................................................. 31 5 Data Exchange and Formats ................................................................................................ 33 5.1 Interaction mechanisms between components .......................................................... 33 5.2 Sequence diagrams ...................................................................................................... 35 6 Conclusions .......................................................................................................................... 37 7 References ........................................................................................................................... 38 APPENDIX: Release 1.0.9 of Architecture and Data Modelling ................................................... 39 D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 5 of 39 www.emerald-he.eu List of figures FIGURE 1. EMERALD DATA DIAGRAM ................................................................................................ 10 FIGURE 2. OVERVIEW OF THE EMERALD COMPONENTS........................................................................ 12 FIGURE 3. OVERVIEW OF THE AI-SEC COMPONENT DATA MODEL ............................................................ 13 FIGURE 4. OVERVIEW OF THE AMOE COMPONENT DATA MODEL ............................................................ 15 FIGURE 5. OVERVIEW OF THE CLOUDITOR-DISCOVERY COMPONENT DATA MODEL ...................................... 16 FIGURE 6. CODYZE COMPONENT OVERVIEW ......................................................................................... 18 FIGURE 7. OVERVIEW OF THE EKNOWS COMPONENT DATA MODEL ........................................................... 19 FIGURE 8. OVERVIEW OF THE TRUSTWORTHINESS SYSTEM COMPONENT DATA MODEL ................................ 20 FIGURE 9. OVERVIEW OF THE MARI COMPONENT DATA MODEL .............................................................. 22 FIGURE 10. OVERVIEW OF THE RCM COMPONENT DATA MODEL ............................................................. 24 FIGURE 11. OVERVIEW OF THE ORCHESTRATOR COMPONENT DATA MODEL .............................................. 26 FIGURE 12. OVERVIEW OF THE EVIDENCE STORE COMPONENT DATA MODEL ............................................. 27 FIGURE 13. OVERVIEW OF THE ASSESSMENT COMPONENT DATA MODEL ................................................... 28 FIGURE 14. OVERVIEW OF THE EVALUATION COMPONENT DATA MODEL ................................................... 29 FIGURE 15. INTERACTIVE SVG - HIGHLIGHT NEIGHBOURS ON CLICK .......................................................... 30 FIGURE 16. LANDING PAGE OF THE INTERACTIVE DOCUMENTATION .......................................................... 31 List of listings LISTING 1. EXAMPLE OF VIRTUAL MACHINE PROPERTIES......................................................................... 17 LISTING 2. AMOE EVIDENCE IN JSON ................................................................................................. 33 LISTING 3. CLOUDITOR EXAMPLE EVIDENCE IN JSON ............................................................................. 34 LISTING 4. A EUCS REQUIREMENT MAPPING IN OSCAL ........................................................................ 35 D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 6 of 39 www.emerald-he.eu Terms and abbreviations AIC4 Artificial Intelligence Cloud Service Compliance Criteria Catalogue AMOE Assessment and Management of Organisational Evidence API Application Programming Interface AST Abstract Syntax Tree BSI Bundesamt für Sicherheit in der Informationstechnik CI/CD Continuous Integration / Continuous Delivery CLI Command Line Interface CSP Cloud Service Provider DoA Description of Action EC European Commission EUCS European Cybersecurity Certification Scheme for Cloud Services GA Grant Agreement to the project GASTM Generic Abstract Syntax Tree gRPC gRPC Remote Procedure Call (created by Google) JSON JavaScript Object Notation KPI Key Performance Indicator MARI Mapping Assistant for Regulations with Intelligence NLP Natural Language Processing OSCAL Open Security Controls Assessment Language PDF Portable Document Format PNG Portable Network Graphics RCM Repository of Controls and Metrics REST Representational State Transfer SARIF Static Analysis Results Interchange Format SVG Scalable Vector Graphics TRL Technology Readiness Level TWS Trustworthiness System UML Unified Modelling Language UUID Universally Unique Identifier WP Work Package D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 7 of 39 www.emerald-he.eu Executive Summary This deliverable, the first version of the data modelling and interaction mechanisms, provides an initial report on the data diagrams, design and documentation of the EMERALD framework and its components. The goal of the corresponding task T1.1 in work package 1 is to coordinate the different types of data shared between the components of WP2, WP3 and WP4. The deliverable provides an overview of the data model, as well as the setup of the interactive documentation. Furthermore, the data exchange and formats are described. D1.1 lays the foundation of the data model – the underlying work of Task 1.1. The resulting documentation serves as a common ground to develop the different components and their APIs. It should offer a high-level overview of the components – displaying the flow of the data. Technical details can be found in the overall data diagram and data format descriptions. Additionally, an overview per component is provided, so as not to be overwhelmed by details, and to be able to focus only on parts of the EMERALD framework. The document is structured in four main parts – the data model, the component overview, the interactive documentation and finally the data exchange and format description. It starts by giving detailed insights into the data classes used in EMERALD. The following section summarizes each component, starting with the evidence collectors (WP2) and continues with the different components of WP3. In the interactive documentation section, the technical setup of the documentation is described. Finally, the plans for the interaction mechanisms are outlined. There will be a second version of this deliverable (D1.2 in M18), which will include updates to the data model and interaction mechanisms. The next steps will be for the different components to implement the data classes and APIs which will be described in the respective component deliverables. Depending on the requirements coming from the pilots (WP5), workflows (WP4) and technical work packages (WP2 and WP3), updates to the data diagrams will be included in the release cycle of the web service and included in the future version of this deliverable. D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 8 of 39 www.emerald-he.eu 1 Introduction This section explains the goal and purpose of the deliverable, its context and its structure. 1.1 About this deliverable This deliverable is the first release of the task T1.1 “Data modelling and information sharing mechanisms” of WP1 of the EMERALD project [1]. It shall provide an overview of the data model that is used in the EMERALD framework. Furthermore, the deliverable provides an overview of each component’s data and how it is linked to other components. The goal is to provide insights of the current state of the data used in EMERALD and how it is organized. A second version of this deliverable will be D1.2, which is due to in M18. The data model will be used by all the components in collaboration with WP2 and WP3 as well as the EmeraldUI component that will be developed in WP4. The interaction mechanism between the different software components will be described and preferred data formats will be presented to facilitate data access and sharing. The task uses the existing data classes of the components and focuses on providing relevant information to the different partners, unfamiliar to the different components. Different abstraction layers will be used to provide an overview and detailed insights. The diagrams will be adjusted over the course of the project and adopted to the requirements of the different components. In order not to lose track of any changes, dedicated processes (see Section 4.3) have been set up to check this. 1.2 Document structure The document is organized into four main sections: • Data model • Component overview • Interactive documentation • Data exchange and formats The data model overview section, Section 2, depicts and describes the current state of the whole data model used in EMERALD. It shall give detailed insights into the inter-component relationships of the EMERALD data. In order to have a more abstract view and not get lost in the details, an overview of the components is provided in Section 3. This section contains a subsection dedicated to each EMERALD component. Section 4 describes the deployment and core implementation of the interactive documentation approach used to share the data model within the EMERALD project. There are three subsections, starting with a section describing PlantUML and how it is used to create the diagrams. This is followed by a description of the web service. Finally, the process on versioning and updating the diagrams is described. Section 5 describes the different formats used in the project and how the components communicate. The deliverable is summarized in Section 6. Finally, the current release of the interactive documentation can be found in the APPENDIX: Release 1.0.9 of Architecture and Data Modelling . D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 9 of 39 www.emerald-he.eu 2 Data Model Overview This section describes the current version of the EMERALD data model. The model describes the different data classes as well as their connections within and between components. The goal is to provide insights to developers and users of the EMERALD framework. Therefore, the data diagram is presented in an interactive system 1 that is explained in more detail in Section 4. There are different abstraction layers, to allow for a “drill down” on the details. Figure 1 shows the resulting data model for the whole EMERALD framework 2 . It depicts each component in a separate box, whereas the background colour denotes the EMERALD work package to which it is related. Evidence collection components (WP2) are coloured in orange, and WP3 components are coloured in teal. Each component box contains the data classes that are relevant for other developers and inter-component communication. Component specific information can be found in the respective subsection of Section 3. This version of the data model is loosely based on the data model that was created in a similar predecessor project called MEDINA 3 - the MEDINA data model was reported in deliverable D5.2 4 . The EMERALD project uses some of the components that were part of the MEDINA data model – such the Evidence Store, the Orchestrator, the Repository of Controls and Metrics (RCM) and the Trustworthiness System. Data classes related to components not relevant to EMERALD were excluded. Please note that the Questionnaire is a subcomponent of the RCM and is therefore shown with a dedicated box in Figure 1. However, as the Questionnaire data model is quite large, it is not shown in Figure 1 but in the RCM component overview (see Figure 10). 1 https://models.emerald.digital.tecnalia.dev/ 2 Please note that an enlarged view of the EMERALD data model is available in APPENDIX: Release 1.0.9 of Architecture and Data Modelling. 3 https://medina-project.eu/ 4 https://medina-project.eu/wpcontent/uploads/2023/05/MEDINA_D5.2_MEDINA_RequirementsDetailed_architectureDevOps_infrastr ucture_v2_v1.0.pdf D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 16 of 39 www.emerald-he.eu Figure 5. Overview of the Clouditor-Discovery component data model D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 17 of 39 www.emerald-he.eu Listing 1. Example of Virtual Machine properties 3.1.4 Codyze The Codyze component is a static source code analysis tool which analyses source code of applications comprising Cloud services and assesses security-relevant implementation details. The analysis report presents implementation details that meet or respectively violate specified security requirements. As part of a CI/CD pipeline, Codyze acts as a quality and compliance gate allowing only the delivery of applications that meet security requirements and preventing it otherwise. Each update to the application’s source code or new release can trigger an execution of the CI/CD pipeline and thereby Codyze. In addition, manual or scheduled assessments are possible. Codyze is developed in Kotlin 7 and uses a graph-based representation of source code utilizing the concept of a code property graph. The resulting representation is largely programming language agnostic. Thus, it facilitates the implementation of generic, reusable source code 7 https://en.wikipedia.org/wiki/Kotlin_(programming_language) message VirtualMachine { option (resource_type_names) = "VirtualMachine"; option (resource_type_names) = "Compute"; option (resource_type_names) = "CloudResource"; option (resource_type_names) = "Resource"; google.protobuf.Timestamp creation_time = 2132; string id = 15888 [(buf.validate.field).required = true]; bool internet_accessible_endpoint = 11229; map<string, string> labels = 12634; string name = 5434 [(buf.validate.field).required = true]; // The raw field contains the raw information that is used to fill in the fields of the ontology. string raw = 17236; ActivityLogging activity_logging = 17610; AutomaticUpdates automatic_updates = 7698; repeated string block_storage_ids = 14852; BootLogging boot_logging = 4303; EncryptionInUse encryption_in_use = 5839; GeoLocation geo_location = 17337; MalwareProtection malware_protection = 5352; repeated string network_interface_ids = 150; OSLogging os_logging = 14872; repeated Redundancy redundancies = 11599; RemoteAttestation remote_attestation = 16051; optional string parent_id = 7061; ResourceLogging resource_logging = 17205; UsageStatistics usage_statistics = 4834; } D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 18 of 39 www.emerald-he.eu analysis techniques. Currently, Codyze supports the programming languages C, C++, Java, Go and Python. Within EMERALD, Codyze interacts with the Orchestrator to orchestrate its analysis, and reports its findings as evidence to the Evidence Store (see Figure 6). Thereby, Codyze generates an analysis report in SARIF 8 (CodyzeSarif). This report contains raw evidence from Codyze’s analysis, which is persisted to the Evidence Store to facilitate further analysis externally to Codyze. Moreover, Codyze processes the findings in the SARIF report into evidence for the EMERALD framework. Each finding is converted into a CodyzeEvidence that identifies the analysed Cloud Service (cloudServiceId), specifies the analysed resource (resource), links it to the underlying SARIF report (sarifId), classifies the finding according to the EMERALD ontology (ontologyRef) and summarizes the result (result). Details on the approach of the Codyze component and its related Task 2.2 will be reported in dedicated deliverables D2.2 “Source Evidence Extractor–v1” (M12) and D2.3 “Source Evidence Extractor–v2” (M24). Figure 6. Codyze component overview 3.1.5 eknows The eknows component – based on a platform for multi-language reverse engineering and documentation generation – extracts evidence from source code files. The source code files are collected from the Cloud Service environment at certain points in time. A set of predefined triggers will be available (e.g., once a week/month/etc., or upon changes) to configure the points in time according to the respective use case. eknows stores the collected files, as well as relevant metadata related to the sources (e.g., from code repositories) and metrics. eknows uses static code analysis to extract evidence. The Java-based software platform provides a modular, extensible set of software components for (i) source code parsing using language8 Static Analysis Results Interchange Format (SARIF), https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarifv2.1.0.html D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 19 of 39 www.emerald-he.eu specific frontends (currently more than 16 programming languages, including Java and Python) (extraction), (ii) transformation of parsed source code into a generic abstract syntax tree (GASTM), (iii) structural and language-independent analysis of security-related information, and (iv) reporting of analysis results for security metrics. The extracted and analysed raw evidence is then forwarded to the Evidence Store component. At the moment of writing, eknows comprises two main data classes (see Figure 7): EknowsSourceCodeFile and EknowsEvidence. EknowsSourceCodeFile serves as an internal representation of the source code file to be analysed, identified by a unique identifier (id). It contains attributes denoting the repository from where to collect the file (repositoryPath), the name of the file (filename), the corresponding Cloud Service (cloudServiceId), the generated abstract syntax tree (AST) model for the parsed source code (compilationUnit), and further metadata, such as the modificationDate. EknowsEvidence is the internal representation of the extracted data of the source code file (fileId). This is created for a security metric (metricId) during the extraction process. The evidence result (result) is stored in the EknowsEvidence class, as well as closely related attributes, such as lineOfCode or analyzerVersion. The internal data class of eknows will change in the next few months, according to the requirements defined for the EmeraldUI in D4.1 [3] and further needs of the pilot partners. EknowsEvidence is related to the Orchestrator (Cloud Service) and the Evidence Store (assessment results, raw evidence). EknowsSourceCodeFile is related to the Orchestrator (Cloud Service). The information from eknows can be accessed via APIs (Java, REST and/or CLI (Command Line Interface)) and used via the upcoming EmeraldUI. Details on the approach of the eknows component and its related Task 2.2 will be reported in dedicated deliverables D2.2 “Source Evidence Extractor–v1” (M12) and D2.3 “Source Evidence Extractor–v2” (M24). Figure 7. Overview of the eknows component data model D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 20 of 39 www.emerald-he.eu 3.2 Trustworthiness System (TWS) Data Model The TWS component securely stores the information and associated metadata of evidence and assessment results on the Blockchain to be able to guarantee its integrity and transparency through the EmeraldUI. Due to the use of Blockchain, sensitive information such as evidence and assessment results are not stored and just a summary of them is recorded on the Blockchain through identifiers and hashes. In fact, in the case of assessment results, two different hashes are included: the assessment result itself and the compliance comments. The evidence and assessment result themselves are kept in a local storage - Evidence Store and Assessment components respectively. In addition, TWS also records metadata information to provide some context. In the case of evidence, they are usually related to specific Cloud Services (cloudServiceId) and the cloud resources to which they refer (resourceId). In the case of an Assessment Result, the requirement to which it refers (requirementId), and the associated evidence identifiers considered in the assessment (evidenceIds) are also stored. Finally, for both evidence and assessment results, recording information about the timestamp when they were created (timestamp) is also useful. As a result, Figure 8 summarises the current data model for evidence (TrustworthyEvidence) and assessment results (TrustworthyAssessmentResult) to be recorded on the Blockchain-based TWS. It also shows the interactions with other components: i) with the Assessment component, which provides information to be recorded in the TWS, and from where the TWS retrieves the actual evidence and assessment results to validate their integrity; ii) with the EmeraldUI, which provides a graphical interface for users to automatically validate the integrity status of the Evidence and Assessment Results. Details on the approach of the TWS component and its related Task 3.5 have been reported in D3.1 [4]. Figure 8. Overview of the Trustworthiness System component data model D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 21 of 39 www.emerald-he.eu 3.3 Mapping Assistant for Regulations with Intelligence (MARI) Data Model MARI – Mapping Assistant for Regulations with Intelligence - is the component that using Deep Learning and state-of-the-art NLP tools is able to create an automatic association between: • A security control and a security metric • Two security controls from two different certification schemes. MARI is based on the previous work in MEDINA's Metric Recommender 9 [5], which took the description of an EUCS security requirement in natural language, the description of a list of metrics, again in natural language, and as a result returned the list of metrics in descending order of relevance. To do this, the textual descriptions of the metrics and requirements are transformed into feature vectors by pre-trained models (in particular, the best association results in MEDINA were obtained with fastText 10 ). A K-d tree is computed on the feature vectors of the metrics, which can be used to select the k closest neighbours of the requirement vector, based on the shortest Euclidean distance. Thus, we were able to obtain a metric-requirement association. At the time of writing, the development of MARI is focused on the internal architecture rather than on interactions with other EMERALD components. Also, we are considering different clustering techniques and different embedding production techniques. We will also extend the MARI functionalities to deal with more certification schemes (in fact, the automatic association between controls from different schemes is a novelty of EMERALD), and as per the work description, we will develop different strategies to work with (e.g., take a subset of metrics that are useful to get a certain level of certification). Figure 9 shows a first approach of the MARI data model., based on the EUCS scheme [6]. The RCM data classes SecurityMetric and SecurityRequirement are taken as input to produce two new data classes, SecurityRequirementsAssociation and MetricRequirementAssociation. These associations are the results of MARI processing. Please note that this data model is subject to change in the coming releases due to the introduction of other certification schemes. In addition, a refined internal data class, data storage, and calls to the component will be reviewed over the coming months, both based on requirements from other components that MARI interacts with, and as the component evolves. Details on the approach of MARI component and its related Task 3.3 have been reported in D3.1 [4]. 9 https://git.code.tecnalia.com/medina/public/nl2cnl-translator 10 https://fasttext.cc/ D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 22 of 39 www.emerald-he.eu Figure 9. Overview of the MARI component data model 3.4 Repository of Controls and Metrics (RCM) Data Model The Repository of Controls and Metrics (RCM) provides a central point in EMERALD framework where the certification schemes are stored and managed. The repository can contain different schemes and includes a complete information of each scheme, with the corresponding categorization. A first approach of the RCM internal data model is based on the EUCS scheme [6] (see Figure 10) 11 , while it is subject to change in the coming releases due to the introduction of other schemes (e.g., BSI C5 12 or AIC4 13 are some foreseen ones). Because of this, the principal data classes implemented in the RCM are SecurityControlFramework, SecurityCategory, SecurityControl and SecurityRequirement, that reflect the organization of the EUCS framework. Along with these, some other auxiliary entities are implemented, such as SimilarControls – to support mapping among controls of different schemes - and ImplementationGuidelines – to help the user with the implementation of the requirements. RCM also incorporates the definition of the SecurityMetric class used in EMERALD to define what to measure to assess the collected evidence. The RCM classes have interactions with other EMERALD components as follows: 11 Please note that an enlarged view of the RCM data model is available in APPENDIX: Release 1.0.9 of Architecture and Data Modelling. 12 https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/CloudComputing/ComplianceControlsCatal ogue/2020/C5_2020.pdf?__blob=publicationFile&v=3 13 https://www.bsi.bund.de/SharedDocs/Downloads/EN/BSI/CloudComputing/AIC4/AI-Cloud-ServiceCompliance-Criteria-Catalogue_AIC4.html D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 23 of 39 www.emerald-he.eu • SecurityControlFramework, SecurityControl, SecurityMetric and SecurityRequirement are related with the Orchestrator, which needs also to internally manage the schemes. • SecurityMetric is also related with the AMOE and the Assessment components. • SecurityMetric and SecurityRequirement are also shared with the MARI component. Another functionality offered by the RCM is a Questionnaire to provide users the possibility to perform a self-assessment to check compliance with the EUCS scheme. The Questionnairerelated data classes, which are enclosed in a box in the diagram (see Figure 10), are as follows: Questionnaire, QuestionnairePurpose, QuestionnaireLevel, Question, QuestionnaireAnswer, QuestionnaireNonConformities, and jhiUser. All these entities are devoted to (i) Implement several questions per requirement, (ii) manage the responses given; (iii) calculate the results for this specific user, and (iv) offer the degree of compliance with the EUCS scheme regarding the selected assurance level. Finally, the EmeraldUI component is also related with the data entities used in the RCM in order to provide the final user with a graphical view of the schemes contained in the RCM and all the associated information. Details on the approach of the RCM component and its related Task 3.2 have been reported in D3.1 [4]. D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 24 of 39 www.emerald-he.eu Figure 10. Overview of the RCM component data model D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 25 of 39 www.emerald-he.eu 3.5 Orchestrator Data Model The Orchestrator is the central management and orchestration component in EMERALD. Its main purpose is to hold all dynamic information about the current audit process, such as the Target Of Evaluations, the evaluated Cloud Services, all Assessment Results, and the final Certificate state (see Figure 11). Furthermore, it fetches static data from the RCM, such as the available schemes and its associated metrics. For performance reasons this data (SecurityControlFramework, SecurityControlCategory, SecurityControl, SecurityRequirement and SecurityMetric) is cached in the Orchestrator. The most important dynamic data classes are: • CloudService, which holds the logical representation of a single service, which aims to be certified. • TargetOfEvaluation, which takes an existing cloud_service_id and combines it with one dedicated security catalogue to produce a Certificate. • Certificate, which is the data class representing different states and is related to the EvaluationResults. • Control, which is the neutral representation of either a control, requirement or objective (this definition of Control is similar to the term defined in OSCAL 14 ). Since every SecurityControlFramework/security scheme uses different names, the Orchestrator normalizes them in the Control data class. In addition, each Control can have subcontrols, which allows to include different SecurityControlFrameworks in EMERALD. Details on the approach of the Orchestrator component and its related Task 3.1 have been reported in D3.1 [4]. 14 https://pages.nist.gov/OSCAL/resources/concepts/terminology/#control D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 32 of 39 www.emerald-he.eu using text-based diagrams and git + GitLab 21 ). Different versions of the diagrams can be stored in commits, and merge requests can be created to deal with changes to the diagrams. The process to add changes to the data model was defined as follows: major changes are completed in a separate branch – when finished, a merge request should be created in the EMERALD GitLab and the changes will be reviewed to check for inconsistencies and breaks to the interactive, web-service-based deployment. After the review, the new version will be merged, which triggers the build pipeline and a new release will be deployed to the EMERALD Kubernetes cluster. The latest release version of the diagrams will then be available to all developers and can be retrieved at https://models.emerald.digital.tecnalia.dev/. If there are any problems, or additional diagrams are needed, Gitlab’s issue functionality can be used to document, communicate and coordinate the required changes. 21 https://gitlab.com/ D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 33 of 39 www.emerald-he.eu 5 Data Exchange and Formats This section provides a short overview of the planned data exchange approach, as well as the formats used. Although all EMERALD components use different data types, they all communicate in a standardized way and format, which speeds up development, as components do not need to build special data connectors for different tools. 5.1 Interaction mechanisms between components The interaction between the components will be implemented using REST 22 – representational state transfer. Each component is using and/or serving REST-APIs that are documented in the OpenAPI 23 specification files. This helps developers to share the different endpoints and allows for code for client interfaces to be generated. Some components may also offer gRPC connections (Remote Procedure Call framework by Google) to share data between closely related components such as Evidence Store and Assessment. The most common format for REST-API will be JSON 24 , as it allows for easy access of attributevalue pairs and arrays. In EMERALD, some components are based on the predecessor versions developed in MEDINA and have existing APIs following the same approach. These APIs can be extended and adjusted to the needs of the EMERALD framework. Listing 2 shows the JSON for a piece of evidence that is sent from AMOE to the Evidence Store. Similarly, Listing 3 shows a more extensive example for data represented in JSON and how it is used by some EMERALD components, such as Clouditor-Discovery. 22 https://en.wikipedia.org/wiki/REST 23 https://en.wikipedia.org/wiki/OpenAPI_Specification 24 https://en.wikipedia.org/wiki/JSON { "id":"b11a1b4b-4cff-4135-afbb-f6e30364d881", "timestamp":"2024-06-26T18:23:45.123456", "cloud_service_id":"3f1c2e4c-8bd5-45d1-a6a3-0f9a9a8e4d35", "tool_id":"amoe", "raw":"password must contain more than 15 characters", "resource": {"id":"165483","type":["PolicyDocument"]} } Listing 2. AMOE evidence in JSON D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 34 of 39 www.emerald-he.eu Some components will offer data import / export functionality. The Repository of Controls and Metrics is planning to allow import of security schemes using the OSCAL 25 format. The API description and more details on the format will be described in the future deliverable D3.3 “Evidence assessment and Certification–Implementation-v1” (M12). The OSCAL format allows different file types and data formats such as YAML 26 and JSON. Listing 4 shows a tentative example of the mapping of an EUCS Requirement in OSCAL. It can be seen how the parts of the Control (ops-02) are specified using the OSCAL elements ”id”, “title”, ”properties”, and also with 25 https://pages.nist.gov/OSCAL/ 26 https://en.wikipedia.org/wiki/YAML { "id": "/subscriptions/XXXXX/resourcegroups/democlouditorhappy/providers/microsoft .storage/storageaccounts/democlouditordiagnostics", "cloudServiceId": "00000000-0000-0000-0000-000000000000", "toolId": "Clouditor Evidences Collection", "properties": { "@type": "type.googleapis.com/clouditor.ontology.v1.ObjectStorageService", "creationTime": "2023-07-09T10:35:18.246911100Z", "id": "/subscriptions/XXXXX/resourcegroups/democlouditorhappy/providers/microsoft .storage/storageaccounts/democlouditordiagnostics", "labels": { "owner": "clouditor" }, "name": "democlouditordiagnostics", "raw": "/*...*/", "geoLocation": { "region": "westeurope" }, "httpEndpoint": { "url": "https://democlouditordiagnostics.[file,blob].core.windows.net/", "transportEncryption": { "enabled": true, "enforced": true, "protocol": "TLS", "protocolVersion": 1.2, "cipherSuites": [] } }, "parentId": "/subscriptions/XXXXX/resourcegroups/democlouditorhappy" } } Listing 3. Clouditor example evidence in JSON D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 35 of 39 www.emerald-he.eu ”parts” and ”prose”; the Requirements are implemented with “parts” within the upper “parts” of Control. The Requirement ID (OPS-02.3) is specified with “properties”, and the requirement itself with “prose”. 5.2 Sequence diagrams To illustrate the interactions between the components, sequence diagrams will be created and extended in the future work of Task 1.1. Additional documentation will be provided which can be included in the interactive PlantUML diagrams. At the time of writing this deliverable, the "controls": [ { "id": "ops-02", "title": "CAPACITY MANAGEMENT - MONITORING", "properties": [ { "name": "label", "value": "OPS-02" } ], "parts": [ { "id": "ops_02_obj", "name": "control-objective", "prose": "The capacities of critical resources such as personnel and IT resources are monitored." }, { "id": "ops-02_smt", "name": "statement", "parts": [ { "id": "ops-02_smt.3", "name": "item", "properties": [ { "name": "label", "value": "OPS-02.3" } ], "prose": "The provisioning and de-provisioning of cloud services shall be automatically monitored to guarantee fulfilment of OPS-02.1" } ] } ] } ] Listing 4. A EUCS Requirement mapping in OSCAL D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 36 of 39 www.emerald-he.eu sequence diagrams for AMOE and eknows have already been integrated into the diagram collection. The sequence diagrams will be included in future deliverables of EMERALD WP1, in particular in D1.3 “EMERALD solution architecture-v1” (M12) and D1.4 “EMERALD solution architecture-v2” (M24). D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 37 of 39 www.emerald-he.eu 6 Conclusions This document provides an overview of the overall EMERALD data model, as well as a more detailed view of the component data models. The general data model is loosely based on the data model of the predecessor project MEDINA. However, to increase the TRL of the reused MEDINA components and adjust them to the EMERALD framework, all component diagrams received updates. Furthermore, it was extended with additional components, such as AI-SEC or eknows. The data model is presented in a web service, to allow interactive investigation of the different diagrams. The diagrams are based on text instructions using PlantUML and then rendered in SVG files. This allows the diagrams to be versioned and the various functionalities of the EMERALD GitLab repository can be used to manage and coordinate the updates. The basic idea of this interactive documentation is to start with an abstract overview (landing page) and then drill down to the different components of interest. The different classes and components of the diagrams can be clicked/hovered to navigate and highlight direct connections. Finally, this deliverable describes the main data format that will be used for data exchange between EMERALD components and external sources – JSON. To provide more insight, an example for AMOE and Clouditor-Discovery evidence have been provided. The Repository of Controls and Metrics (RCM) will provide import/export functionality of security schemes in OSCAL format – for which a JSON example was also provided. The data diagrams will be updated according to the needs and changes of the different components. These changes will be subject to the described processes in this deliverable, shared with the consortium in different version releases, and deployed in the EMERALD Kubernetes infrastructure. The updates will be collected and described in a second version of this deliverable (D1.2 – Data modelling and interaction mechanisms v2), planned to be submitted in M18 of the project (end of April 2025). D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 38 of 39 www.emerald-he.eu 7 References [1] EMERALD Consortium, “EMERALD - Annex 1 - Description of Action - GA 101120688,” 2022. [2] EMERALD Consortium, “D2.1 Graph Ontology for Evidence Storage,” 2024. [3] EMERALD Consortium, “D4.1 Results of the UI-UX requirements analysis and the work processes–v1,” 2024. [4] EMERALD Consortium, “D3.1 Evidence assessment and Certification–Concepts-v1,” 2024. [5] MEDINA Consortium, “D5.5 MEDINA integrated solution-v3,” 2023. [Online]. Available: https://medina-project.eu/wp-content/uploads/2023/09/MEDINA_D5.5_MEDINAintegrated-solution-v3_v1.0.pdf. [Accessed July 2024]. [6] ENISA, “EUCS - Cloud Services Scheme,” [Online]. Available: https://www.enisa.europa.eu/publications/eucs-cloud-service-scheme. [Accessed July 2024]. D1.1 – Data Modelling and interaction mechanisms – v1 Version 1.0 – Final. Date: 31.07.2024 © EMERALD Consortium Contract No. GA 101120688 Page 39 of 39 www.emerald-he.eu APPENDIX: Release 1.0.9 of Architecture and Data Modelling In order to allow the readers of this document to consult the documentation and data model themselves, the current version of the files have been archived in a zip file. The contents are images of the different data models, as well as a webpage to aid in navigation. The 1.0.9 release version of the interactive documentation is available here: D1.1 Appendix Release 1.0.9 of Architecture and Data Modelling To open the interactive documentation locally, you need to extract the zip file. Then navigate to the “architecture_and_data_model” folder and open the index.html file in a common web browser.