D5.5: Validation and Evaluation Report (first version)

D5.5: Validation and Evaluation Report (first version) PROJECT Project Number 101120962 Project Acronym RESCALE Project Title Revolutionised Enhanced Supply Chain Automation with Limited Threats Exposure Start Date 01.10.2023 Programme HORIZON-CL3-2022-CS-01-02 DELIVERABLE Deliverable Type Report Workpackage WP5 Deliverable Lead UNSPMF Editors Danijela Boberic Krsticev (UNSPMF) Contributors All partners Dissemination Level Public Abstract The current document is an initial validation and evaluation report on the fulfillment of the functional and non-functional requirements identified in D2.3. As part of the validation process, each component is validated individually and within the more extensive integrated system to ensure seamless interoperability and alignment with predefined requirements. In addition, the project defined a set of Key Performance Indicators (KPIs) that serve as benchmarks to measure the project’s success in achieving its research and development objectives, and this deliverable includes a report on the status of those KPIs. Furthermore, this deliverable presents an evaluation survey designed to collect user feedback during software demonstrations at info days and similar events, providing valuable insight into the software’s usability, usefulness, and adoption potential among stakeholders. Through this deliverable, RESCALE establishes a structured and dynamic framework for continuous validation and evaluation, reinforcing the reliability, security, and overall effectiveness of its solutions while aligning with project goals and industry standards. Disclaimer The information in this document is provided “as is”, and no guarantee or warranty is given that the information is fit for any particular purpose. The content of this document reflects only the author’s view – the European Commission is not responsible for any use that may be made of the information it contains. The users use the information at their sole risk and liability. This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101120962 D5.5: Validation and Evaluation Report (first version) Internal Reviewers 1. Sascha Kattelmann - (PST) 2. Daniel Asztalos - (CC) Revisions Version Date Partner Overview 0.1 10/01/2025 UNSPMF ToC 0.2 14/03/2025 All partners First round of input 0.3 23/03/2025 All partners Second round of input 0.4 25/03/2025 UNSPMF Aligning terminology and formatting across sections 0.5 31/03/2025 PST, CC, AEGIS, ISI Internal review started 0.6 10/04/2025 All partners Review comments addressed 1.0 11/04/2025 UNSPMF Final version RESCALE – Public – Page 2 / 148 Table of Contents 1 Introduction 10 1.1 Scope&Contribution............................... 10 1.2 Relation to Work Packages, Deliverables, and Activities . . . . . . . . . . . . . 11 1.3 Contribution to WP5 and Project Objectives . . . . . . . . . . . . . . . . . . . 11 1.4 DocumentStructure................................ 12 2 Software Architecture Overview 14 3 Component-Level Validation 17 3.1 Static Code Analysis Module . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.1.1 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.1.2 EvaluationResults ............................ 19 3.1.3 KPIIndicators............................... 28 3.2 DynamicTestingModule............................. 33 3.2.1 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2.2 EvaluationResults ............................ 34 3.2.3 KPIIndicators............................... 52 3.3 TrustOR...................................... 56 3.3.1 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.3.2 EvaluationResults ............................ 58 3.3.3 KPIIndicators............................... 69 3.4 LedgerInfrastructure ............................... 72 3.4.1 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.4.2 EvaluationResults ............................ 73 3.4.3 KPIIndicators............................... 80 3.5 Security Assurance Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 3.5.1 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 81 3.5.2 EvaluationResults ............................ 82 3.5.3 KPIIndicators............................... 83 3.6 StateManagerModule .............................. 85 3.6.1 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.6.2 EvaluationResults ............................ 85 3.6.3 KPIIndicators............................... 90 3.7 Dashboard..................................... 91 3.7.1 Technical Requirements . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.7.2 EvaluationResults ............................ 92 3.7.3 KPIIndicators............................... 98 4 System-Level Validation 99 4.1 TechnicalRequirements.............................. 99 4.2 ImpactKPIIndicators...............................101 4.2.1 Key Impact KPIs and Their Evaluation Framework . . . . . . . . . . . 101 4.3 MVPEvaluationResults .............................103 4.3.1 Test Scenario 1: Supply Chain Security Validation and TBOM Generation103 4.3.2 Test Scenario 2: TBOM Validation Workflow . . . . . . . . . . . . . . 104 4.3.3 System Integration Sequence Diagram . . . . . . . . . . . . . . . . . . 105 3 D5.5: Validation and Evaluation Report (first version) 5 User Evaluation 108 6 Conclusion 109 A RESCALE Evaluation Questionnaire 110 B SAVE-ME Example 117 C DSCG Example 119 D TBOM file example 129 E SASTER-cli Example 133 RESCALE – Public – Page 4 / 148 List of Figures 1 RESCALE High-level Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2 Example Execution of the SSCG Generator as Part of PST’s CI System . . . . . . . 21 3 Example Execution of the SASTer-cli Tool..................... 26 4 TrustOR component downloaded and run with Docker CLI . . . . . . . . . . . . . 58 5 Part of TrustOR repository README.md . . . . . . . . . . . . . . . . . . . . . . 59 6 TrustOR image OS/ARCH details as displayed in Harbor . . . . . . . . . . . . . . 60 7 TrustOR Swagger and OpenAPI json . . . . . . . . . . . . . . . . . . . . . . . . . 60 8 Schema details as shown in TrustOR Swagger . . . . . . . . . . . . . . . . . . . . 62 9 MongoDB collections used by TrustOR, as seen via a Mongo Express dashboard . 63 10 Request / Response of endpoint for getting TBOM metadata . . . . . . . . . . . . 64 11 logs / steps mapped to the TBOM lifecycle introduced in D4.1 . . . . . . . . . . . 65 12 TrustOR endpoints for receiving SBOMs, SSCGs & DSCGs . . . . . . . . . . . . 65 13 Blockchain metadata for specific TBOM . . . . . . . . . . . . . . . . . . . . . . . 68 14 Example of valid and not-valid TBOM . . . . . . . . . . . . . . . . . . . . . . . . 69 15 TBOM visible via Dashboard also added in the blockchain . . . . . . . . . . . . . 70 16 Endpoint for receiving a TBOM file . . . . . . . . . . . . . . . . . . . . . . . . . 71 17 Endpoint for validating a TBOM file . . . . . . . . . . . . . . . . . . . . . . . . . 71 18 Ledger Infrastructure component downloaded and run with Docker CLI . . . . . . 73 19 Example of README.md file from the RESCALE GitLab repository . . . . . . . 74 20 Ledger Docker image based on Hyperledger Besu available in RESCALE Harbor registry. ........................................ 75 21 Extract of the README.md from the Trust Storage client repository in the RESCALE GitLabrepository.................................... 76 22 Chapter 3 structure from D4.4 - TBOM Security and Trust Mechanism . . . . . . . 77 23 Metrics described in Section 3.1.2 of D4.4 - TBOM Security and Trust Mechanism 78 24 HashManagersourcecode .............................. 79 25 Data structures in HashManager . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 26 ResponseTime .................................... 83 27 State Manager SSCG Generation process . . . . . . . . . . . . . . . . . . . . . . . 86 28 State Manager SSCG Generation process . . . . . . . . . . . . . . . . . . . . . . . 86 29 State Manager docker pull from harbor . . . . . . . . . . . . . . . . . . . . . . . . 87 30 State Manager running as docker image . . . . . . . . . . . . . . . . . . . . . . . 87 31 State Manager OpenAPI Documentation . . . . . . . . . . . . . . . . . . . . . . . 88 32 MachineReadableI/O ................................ 89 33 MongoDB Collections utilised by State Manager . . . . . . . . . . . . . . . . . . 89 34 State Manager shown as a Keycloak client inside a Keycloak Realm . . . . . . . . 90 35 SSCGInspection ................................... 93 36 Getting the Access Token from the Dashboard to initiate SSCG generation . . . . . 94 37 DSCGInspection ................................... 95 38 Getting the Access Token and the Serial Number from the Dashboard . . . . . . . 96 39 Dashboard Docker pull from harbor . . . . . . . . . . . . . . . . . . . . . . . . . 96 40 Dashboard running as Docker image . . . . . . . . . . . . . . . . . . . . . . . . . 96 41 Producer’sView.................................... 97 42 Consumer’sView ................................... 97 43 User Authentication and Component Registration . . . . . . . . . . . . . . . . . . 106 44 StaticAnalysisPhase .................................106 5 D5.5: Validation and Evaluation Report (first version) 45 DynamicAnalysisPhase ...............................107 46 TBOMGeneration ..................................107 RESCALE – Public – Page 6 / 148 List of Tables 1 Project’s objectives related to this deliverable . . . . . . . . . . . . . . . . . . . . 12 2 List of Initial Functional and Non-Functional Requirements related to the Static CodeAnalysisModule ................................ 17 3 List of KPIs related to the Static Code Analysis Module . . . . . . . . . . . . . . . 29 4 List of Initial Functional and Non-Functional Requirements related to the Dynamic TestingModule .................................... 33 5 List ofKPIs related to the Dynamic Testing Module . . . . . . . . . . . . . . . . . 53 6 List of Initial Functional and Non-Functional Requirements related to the TrustOR component....................................... 57 7 List of KPIs related to the TrustOR Component . . . . . . . . . . . . . . . . . . . 69 8 List of Initial Functional and Non-Functional Requirements related to the Ledger Infrastructurecomponent............................... 72 9 List of KPIs related to the Ledger Infrastructure Component . . . . . . . . . . . . 80 10 List of Initial Functional and Non-Functional Requirements related to the Continuous Security Assurance component . . . . . . . . . . . . . . . . . . . . . . . . . 82 11 List of KPI Indicators related to the Continuous Security Assurance component . . 83 16 Traceability Matrix of Initial Functional and Non-Functional Requirements . . . . 99 7 D5.5: Validation and Evaluation Report (first version) List of Abbreviations API Application Programming Interface. 17 BOV Bill of Vulnerabilities. 64 CD Continuous Deployment. 19, 27 CI Continuous Integration. 19, 27 CLI Command-Line Interface. 20, 27 CPA Correlation Power Analysis. 44 CPE Common Platform Enumeration. 82 CVE Common Vulnerabilities and Exposures. 18, 26, 27, 32, 39, 53, 56 CWE Common Weakness Enumeration. 22, 23, 26, 27, 39, 52 DL Deep Learning. 17, 22–24, 32 DSCG Dynamic Supply Chain component Guarantee. 5, 14, 15, 35, 36, 39, 44, 45, 50, 56, 65, 70, 72, 94, 104, 105 DuT Device Under Test. 45, 46 FN False Negative. 30, 31 FP False Positive. 30, 31 FPGA Field Programmable Gate Array. 46 IAM Identity and Access Management. 14, 105 iKPI Impact Key Performance Indicator. 101 JSON JavaScript Object Notation. 22, 27, 36, 38, 40, 45, 46, 48, 51, 83, 88, 92, 97 KPI Key Performance Indicator. 1, 3, 7, 10, 15, 17, 28–32, 52, 53, 55, 56, 69, 71, 80, 81, 83, 84, 90, 101, 109 LLM Large Language Model. 24 ML Machine Learning. 17, 25, 26, 28, 30, 44, 45, 49 MVP Minimum Viable Product. 10, 17, 99, 109 NVD National Vulnerability Database. 15 SARIF Static Analysis Results Interchange Format. 21–28 RESCALE – Public – Page 8 / 148 D5.5: Validation and Evaluation Report (first version) SBOM Software Bill of Materials. 5, 17, 56, 65, 70, 72, 104, 105 SSCG Static Supply Chain component Guarantee. 5, 14, 15, 19–21, 25–27, 35, 36, 56, 65, 70, 72, 92, 95, 97, 104, 105 SSL Secure Sockets Layer. 61 TBOM Trusted Bill of Materials. 3, 5, 6, 10, 14, 15, 44, 56, 57, 62–65, 67–73, 78–82, 84, 103–105, 107 TDC Time-to-Digital Converter. 46 TN True Negative. 30, 31 TP True Positive. 30, 31 TRL Technology readiness levels. 11 TVLA Test Vector Leakage Assessment. 44 XML Extensible Markup Language. 83 RESCALE – Public – Page 9 / 148 D5.5: Validation and Evaluation Report (first version) Dynamic Testing module Management module TrustOR Communication mechanisms Trust Storage Repositories Low Level Hardware Assessment Dynamic HW Analyzer Static code analysis module Static Code Analyzer Formal Verifier Standardization Guidelines (CVE/CWE) Security Assurance TBOM Generator TBOM Validator Dashboard External git Repositories SSCG Dynamic Software Testing Ledger Infrastructure SSCG Generator DSCG Generator Auth / IAM State manager Ledger Adapter DSCG TBOM Assessment Domain Security & Trust Domain Management Domain Figure 1: RESCALE High-level Architecture RESCALE – Public – Page 16 / 148 D5.5: Validation and Evaluation Report (first version) 3 Component-Level Validation Ensuring the reliability and accuracy of individual RESCALE’s components is a critical step in the validation process. Before full integration testing, component-level validation focuses on assessing each module independently to verify its functionality, performance, and compliance with predefined requirements. This process helps identify potential defects early, facilitating more efficient debugging and reducing the risk of RESCALE system failures. The individual components of the RESCALE solution, as presented in Figure 1, have been validated against requirements and KPIs previously defined in D2.3. The results of this validation are detailed in the following subsections. 3.1 Static Code Analysis Module The Static Code Analysis Module processes software projects, specifically their source code and the SBOM associated with it, to identify and assess potential security vulnerabilities and weaknesses. It employs advanced ML/DL enhanced static code analyzers, which go beyond traditional methods to detect issues more effectively. These advanced analyzers are particularly effective at identifying security vulnerabilities (e.g., injection flaws, buffer overflows), code smells, API misuse, and performance anti-patterns. The ML/DL components play a crucial role in reducing false positives, enhancing the accuracy of the analysis compared to traditional static analysis tools. 3.1.1 Technical Requirements The Static Code Analysis Module currently consists of several tools: Execution Engine (orchestrates all tools based on a given configuration), SSCG Generator, SAVE-ME, SASTER-cli, IVEE (Intelligent Vulnerabilities Exposure Engine), DetectEr. Each of these tools were validated against the MVP to assess the extent to which the functional requirements have been fulfilled till M18. Those assessments provide insights into the module’s effectiveness, accuracy, and efficiency in real-world scenarios. Table 2 provides an overview of the identified functional and non-functional requirements, including their description, the tools (inside the Static Code Analysis Module) associated with their implementation, the method used for validation, and the current status of each requirement. Initial specification of RESCALE’s functional and nonfunctional requirements is presented in D2.3 - Technology Analysis and Requirements Specification (first version). Table 2: List of Initial Functional and Non-Functional Requirements related to the Static Code Analysis Module TRID Title Tool Validation method Status TR003 Source code collection Execution Engine Demonstration Achieved RESCALE – Public – Page 17 / 148 D5.5: Validation and Evaluation Report (first version) Table 2 – continued from previous page TRID Title Tool Validation method Status TR004 Static code scanning SAVE-ME, SASTER-cli, IVEE Demonstration In progress TR005 False Positive/Negative analysis of static analysis report SAVE-ME, SASTER-cli Demonstration In progress TR006 Show the results of static analysis SSCG Generator, SASTER-cli, SAVE-ME, IVEE, DetectEr Demonstration In progress TR007 Generate SSCG SSCG Generator Demonstration In progress TR012 Tools/components should be able to “download and run” Static Code Analysis Module & Execution Engine Demonstration In progress TR013 Documentation Completeness Static Code Analysis Module, SAVE-ME, SASTER-cli Demonstration In progress TR014 Docker Compatibility SSCG Generator, SAVE-ME, SASTER-cli, IVEE, DetectEr Demonstration In progress TR016 Automated Tool/Component Accessibility Static Code Analysis Module incl. all tools Demonstration In progress TR017 Secure Communication Protocol Support SSCG Generator Demonstration Scheduled for M20 TR018 Machine Readable Data Input/Output SSCG Generator, SASTER-cli, IVEE, SAVE-ME, DetectEr Demonstration In progress TR022 Modules-level ingesting of crucially important data sources (CVE, etc.) SASTER-cli, SAVE-ME Demonstration Scheduled for M21 TR023 Security/Cryptographic Primitive Operations Execution Engine Demonstration Scheduled for M22 TR024 ML/DL for Data Classification SAVE-ME Demonstration In progress TR025 Software Property Verification DetectEr, IVEE Demonstration In progress TR026 Static Code Analysis Scalability Static Code Analysis Module incl. all tools Demonstration Scheduled for M22 RESCALE – Public – Page 18 / 148 D5.5: Validation and Evaluation Report (first version) Table 2 – continued from previous page TRID Title Tool Validation method Status TR027 Static Code Analysis and Formal Verification Accuracy SASTER-cli, IVEE, SAVE-ME Demonstration In progress TR028 Static Code Analyzer and Formal Verifier Integration DetectEr, IVEE Demonstration Scheduled for M22 TR029 Machine Learning Integration SASTER-cli, SAVE-ME Demonstration Achieved TR030 Language Support SAVE-ME, SASTER-cli, IVEE, DetectEr Demonstration Achieved for Erlang, Python, C, C++ 3.1.2 Evaluation Results The following evaluation results provide an overview of test cases and corresponding evidence demonstrating the Static Code Analysis Module’s compliance with the defined requirements. This evaluation aims to verify that each tool operates according to its intended purpose and meets the criteria outlined in the requirement and Table 2 presented in the previous section. A dedicated evaluation is carried out separately for each tool. 3.1.2.1 Execution Engine The Execution Engine is the central coordinator of the Static Code Analysis Module. It makes the source code available to analyzers, processes configuration settings into specific analysis actions, and manages the execution of each analyzer. Based on their results, it triggers the SSCG generation and ensures the output is pushed to the RESCALE platform. Acting as the glue of the module, it ensures all components work together smoothly to deliver verified code analysis results. TR003 Source code collection: The Execution Engine relies on source code being injected into the Docker container of the Static Code Analysis Module. This is commonly done in an automated way in CI/CD systems. The injected source code is then forwarded to the static code analyzers of the module. TR012 Tools/components should be able to “download and run”: The Static Code Analysis Module can be downloaded and started as Docker container, together with a specific configuration. This configuration is then ingested into the Execution Engine which translates it to a set of actions, e.g. the execution of analysis tools on source code and the generation of the SSCG. The user only needs to provide such configuration to the Docker container to run it, there is no need to compile or build anything. RESCALE – Public – Page 19 / 148 D5.5: Validation and Evaluation Report (first version) 3.1.2.2 SSCG Generator The SSCG Generator, developed in Erlang for the RESCALE project, is a CLI tool designed for generating and managing SSCGs. The SSCG Generator consumes test reports from a single static code analysis tool and processes them into a single SSCG, which is a CycloneDX document [15]. It then sends the SSCG to the RESCALE platform, given certain configuration parameters like a URL endpoint and an authentication token. TR006 Show the results of the static analysis: The SSCG Generator bundles all test reports from all analyzers of the static code analysis module together into a SSCG as outlined in D3.1. This functionality has already been verified by the pilot partners but is still subject to improvements as the tools are still in development (see also D5.3). TR007 Generate SSCG: The SSCG is generated as outlined in D3.1 and can be published to the RESCALE platform. This functionality has already been verified by pilot partners (see D5.3) but is still subject to improvements as the tools are still under development. TR012 Tools/components should be able to “download and run”: The SSCG Generator can be downloaded as part of the static code analysis module and, given an appropriate configuration (e.g. access token for the RESCALE platform and a correct URL pointing to it) it works out of the box. The tool can also be used on its own as it has a CLI interface with documented options. This CLI interface is documented in D3.1. As shown in D5.3 it is possible for the pilot partners to utilize the SSCG Generator as part of the static code analysis module without any problems. An example run of the SSCG Generator is illustrated in Figure 2. TR014 Docker Compatibility: The SSCG Generator is embedded into the static code analysis module, which is a single Docker container. The user just has to download this container to make use of all tools related to that module. The compatibility with Docker has been verified by the pilot partners as well (see D5.3). TR016 Automated Tool/Component Accessibility: The SSCG Generator is indirectly configured using the configuration of the static code analysis module. The user of the module does not even directly touch the SSCG Generator; it can be used without even being aware of it. This mode of operation has also been verified by the pilot partners (see D5.3). TR017 Secure Communication Protocol Support: The SSCG Generator sends the SSCG to the RESCALE platform. For this purpose it is required to use authentication and encryption mechanics such that RESCALE infrastructure can be verified and data in transit can’t be read by a man-in-the-middle attack. A typical example of such a mechanism is TLS. It is currently under discussion how this topic is approached and a conceptual solution is expected by M21. RESCALE – Public – Page 20 / 148 D5.5: Validation and Evaluation Report (first version) Figure 2: Example Execution of the SSCG Generator as Part of PST’s CI System TR018 Machine Readable Data Input/Output: The SSCG Generator requires the analyzer tools to provide their test reports in a certain folder. All files in this folder are then processed into an SSCG. While the Generator itself has no hard requirements on the file content format, it is expected that the analyzer tools output their data in SARIF format (which is a standard for static code analysis data, see D3.1). The test reports are encoded with base64 and inserted as a string into the SSCG, which itself is a CycloneDX document. Hence, the overall SSCG, including the actual test reports, is machine processable. 3.1.2.3 SAVE-ME SAVE-ME is an adaptation of the CodeBERT[6] model specifically fine-tuned to detect insecure code patterns in Erlang. The model operates at the function level, making it suitable for analyzing individual Erlang functions in isolation. This is a critical capability given Erlang’s functional programming paradigm and its emphasis on concurrent processes, where functions often encapsulate isolated logic and are triggered independently. Unlike traditional monolithic codebases, Erlang applications are typically composed of many small, self-contained functions, making function-level analysis particularly effective for identifying localized vulnerabilities. TR004 Static code scanning: The SAVE-ME module satisfies the requirement for static code scanning by analyzing Erlang source code without execution, identifying potential security vulnerabilities and coding issues. The scanning process is fully automated and operates on a per-function basis to ensure a finegrained analysis. Specifically, SAVE-ME processes a given input directory containing Erlang source files. It systematically scans each file, extracts individual functions, and passes them through the model RESCALE – Public – Page 21 / 148 D5.5: Validation and Evaluation Report (first version) for analysis. The model then classifies each function based on the presence of security vulnerabilities, providing a vulnerability type prediction and a corresponding confidence score. Currently, SAVE-ME identifies vulnerabilities using the following categories: •BAD ARG – Invalid or unexpected arguments passed to a function. •BAD GENERATOR – Errors in generator expressions, such as invalid list comprehensions. •BAD KEY – Accessing non-existent keys in maps or dictionaries. •BAD MAP – Incorrect usage or construction of map data structures. •BAD RECORD – Misuse of record fields or accessing undefined fields. •NO MATCHING CASE CLAUSE – Missing or unhandled cases in case expressions. •NO MATCHING FUNCTION CLAUSE – Pattern mismatch in function clauses with no fallback. •NO MATCH OF RHS – Pattern mismatch when binding the right-hand side of a match expression. As the dataset expands, we plan to refine these classifications by mapping detected vulnerabilities to CWE categories. This will enhance the tool’s ability to align with industry standards and provide more structured vulnerability insights. The ongoing dataset development will facilitate this transition, ensuring broader and more meaningful classification capabilities in future iterations. The results are structured and written according to the SARIF format, ensuring compatibility with security workflows and automated reporting systems. This approach satisfies the static scanning requirement by allowing SAVE-ME to operate on a function level rather than analyzing entire files simultaneously, making it modular, scalable, and adaptable. Future improvements will continue refining detection accuracy and expanding the types of vulnerabilities identified. TR005 False Positive/Negative analysis of static analysis report: The SAVE-ME tool enables developers to evaluate the reliability of its vulnerability assessments by providing confidence scores alongside its predictions. Since SAVE-ME is a DL model rather than a traditional rule-based static analyzer, it does not pinpoint specific problematic lines in the code. Instead, it classifies entire Erlang functions and outputs a confidence score indicating the likelihood that a particular function contains a vulnerability. TR006 Show the results of static analysis: The output of SAVE-ME’s static analysis is structured in JSON format following the SARIF schema and passed along the pipeline to the SSCG Generator for further processing (for more details, see TR018 below). The tool can also display the raw results in text format to the user running this tool. TR013 Documentation Completeness: RESCALE – Public – Page 22 / 148 D5.5: Validation and Evaluation Report (first version) As the tool develops, its documentation is consistently revised and enhanced. Additionally, we provide straightforward and comprehensive Docker documentation. Some details can be found in Chapter 4 of the RESCALE Deliverable D3.1, and the documentation will be extended as the project progresses. TR014 Docker Compatibility: The SAVE-ME sub-module satisfies the requirement for Docker compatibility by being encapsulated within a Docker container. This ensures ease of deployment, portability, and integration into existing security workflows. A dockerized version of SAVE-ME can be found in the RESCALE Harbor repository [18]. SAVE-ME is both shipped as part of the Static Code Analysis Module container and provided as a standalone Docker container in the RESCALE Harbor. Users can invoke SAVE-ME using a simple Docker command, providing the necessary input and output directories. The results are generated in SARIF format, ensuring compatibility with static analysis tools. Moreover, SAVE-ME is automatically triggered within the Static Code Analysis Module when an Erlang project is detected, making its integration seamless and transparent to the end user. For more details on SAVE-ME’s Docker implementation, refer to Chapter 4 of the RESCALE Deliverable D3.1. TR018 Machine Readable Data Input/Output: To facilitate user interaction and verification, SAVE-ME produces its results in SARIF, a standardized format that allows security reports to be reviewed systematically. Developers can examine the reported vulnerabilities, assess the associated confidence scores, and determine whether a flagged issue represents a true positive (an actual security concern) or a false positive (an issue incorrectly reported as a vulnerability). By structuring its output in the SARIF format, SAVE-ME ensures compatibility with existing security tools and workflows. Future improvements will focus on refining confidence estimation and integrating mechanisms to incorporate user feedback, further improving the model’s precision and reducing false positives. An example of the SAVE-ME output can be found in Appendix B. TR022 Modules-level ingesting of crucially important data sources: Currently, SAVE-ME is working with Erlang-specific designations of vulnerabilities. As the project continues and the dataset used to train the DL model is expanded, support will be added for CWEs. TR024 ML/DL for Data Classification: The SAVE-ME tool satisfies the requirement for DL-based data classification by utilizing a fine-tuned deep-learning model to determine whether an Erlang function handles vulnerable or non-vulnerable data. This classification is important for enforcing proper security measures, ensuring that vulnerable data is appropriately handled and protected against potential leaks or misuse. RESCALE – Public – Page 23 / 148 D5.5: Validation and Evaluation Report (first version) To achieve this, SAVE-ME processes each Erlang function individually, analyzing its structure, parameter usage, and data flow. The model, based on the fine-tuned CodeBERT [6] variant, learns contextual patterns that distinguish vulnerable data handling from general function logic. Unlike rule-based approaches that rely on fixed keyword lists, this DL-driven approach allows SAVE-ME to generalize across different coding styles and usage patterns, improving accuracy and adaptability. To train and refine the classification model, SAVE-ME requires a labeled dataset of Erlang functions, where each function is tagged as either vulnerable or secure. This dataset is currently being constructed using InfERL [10], a static analysis tool, to analyze various Erlang projects and extract labeled examples. The initial labels undergo manual inspection and expert validation to enhance accuracy and minimize false positives. By integrating expert-reviewed ground truth data, SAVE-ME ensures that its DL model continuously improves in precision and reliability. The classification results, along with confidence scores, are exported in SARIF format, allowing seamless integration with security analysis tools and automated workflows. As the dataset expands and the model is further refined, SAVE-ME will enhance its ability to recognize complex data-handling scenarios in Erlang, making static code analysis more robust and adaptable to evolving security needs. While its open-source publication and potential reuse by other RESCALE tools are topics under consideration, these aspects remain subject to future discussion and agreement within the consortium. For further details on the CodeBERT model, SAVE-ME model, and the dataset, please refer to Chapter 4 of the RESCALE Deliverable D3.1. TR026 Static Code Analysis Scalability: Once the DL model is fully trained, steps will be taken to ensure its smooth operation with increasing size and number of inputs. TR027 Static Code Analysis and Formal Verification Accuracy: High precision of vulnerability prediction will be ensured by rigorous testing in different application scenarios and through prepared benchmarks. See KPI1.2 and KPI2.1 in the Section 3.1.3. TR029 Machine Learning Integration: SAVE-ME leverages a pre-trained CodeBERT-based model which is fine-tuned for vulnerability classification of Erlang source code (for more details, see TR030 below). TR030 Language Support: The SAVE-ME tool is designed to support static code analysis for Erlang, leveraging a pretrained CodeBERT-based model fine-tuned for vulnerability classification. While traditional static analysis tools often require hand-crafted rules for each supported language, SAVE-ME benefits from the transfer learning capabilities of large language models (LLMs). By using the pre-trained CodeBERT model, SAVE-ME can be adapted to different programming languages through additional fine-tuning, making it a flexible and scalable approach to vulnerability detection. This method allows the tool to learn language-specific patterns and RESCALE – Public – Page 24 / 148 D5.5: Validation and Evaluation Report (first version) security risks without needing entirely new rule sets, unlike conventional static analysis tools that require manual rule definitions for each language. Although SAVE-ME is currently tailored for Erlang, its underlying approach offers a path for extending support to other languages. Fine-tuning on additional datasets can enable the model to generalize across languages like Python and C/C++, ensuring adaptability to different coding environments while maintaining the efficiency of deep learning-based static analysis. 3.1.2.4 DetectEr DetectEr is a runtime verification tool for Erlang programs that monitors system executions against formally specified correctness properties. It uses a logic-based specification language to define expected behaviors and dynamically checks these properties during program execution. In the scope of RESCALE, the pilot partner PST makes DetectEr usable for its embedded software (step 1) and generates formal models for checking specific pieces of software (step 2) running on embedded devices. The development of DetectEr is currently still focused on step 1. Hence, a detailed evaluation of DetectEr with respect to technical requirements is currently not meaningful. It is expected that DetectEr can be integrated into the static code analysis module within the next three months, such that at least TR006, TR012, TR013, TR014 and TR016 will be covered in this time frame. 3.1.2.5 SASTER-cli SASTER-cli is a command-line tool that performs static code analysis by using and orchestrating multiple static code analyzers. It supports a plethora of static code analyzers (e.g., Bandit, Semgrep, Flawfinder, etc.) and is designed for modularity and automation. The tool is capable of detecting software vulnerabilities across different programming languages and includes a ML component to reduce false positives. TR004 Static code scanning: The SASTER-cli tool fulfills the requirement for static code scanning by orchestrating multiple static code analyzers. The tool processes source code, runs the selected analyzers and aggregates their outputs in SARIF format. SASTER-cli can be currently executed via the following command: python3 saster-cli.py scan --offline -m Flawfinder,Bandit,Semgrep,Horusec,Bearer,Cppcheck -o results.sarif project/ --verbose This command initiates analysis with the selected modules, with the results being in SARIF format. Each analyzer operates independently under the hood, and their outputs are later given as input to the SSCG Generator. Figure 3 shows the execution of the tool. RESCALE – Public – Page 25 / 148 D5.5: Validation and Evaluation Report (first version) 2. Each benchmark application is analyzed using the InfERL tool guided by an Erlang expert. 3. The number of actual vulnerabilities present in the benchmark application is documented, providing a reference point for evaluation. 4. The assessment results from SAVE-ME are compared with the documented vulnerabilities, and the overall accuracy is computed as expressed in Eq. 1 and the following text. The goal for this metric is to exceed the average of 98% across all tested applications. To evaluate SAVE-ME’s performance, a set of benchmark applications will be utilized, where the vulnerabilities are manually or synthetically inserted and validated by security experts based on their knowledge. The module will be executed on these applications, and its results will be cross-referenced with the established ground truth. Since SAVE-ME utilizes a DL-based vulnerability classification model, its performance will improve as the dataset used for training expands. The ongoing development of a labeled dataset of Erlang functions, generated using InfERL and refined through expert validation, will further enhance the accuracy of its vulnerability assessments. KPI2.2: Detect more than 90% of known vulnerabilities at hardware and software level for the pilot’s vulnerable hardware and software solutions Rather than attempting to detect every possible vulnerability, the strategy involves defining a prioritized list of known vulnerabilities that are most relevant to the pilot’s environment and ensuring that the tools, supported by manual verification where necessary, can detect 90% of them. At present, a comprehensive assessment of all vulnerabilities in the pilots’ software projects is not available. To address this, we will curate a representative subset of known vulnerabilities from relevant public databases (e.g., CVE) and known security issues associated with the software components used in the pilot. This subset will serve as a practical benchmark for evaluating the detection capabilities of RESCALE tools. This approach ensures that the KPI remains meaningful and achievable while adapting to the practical constraints of the pilot’s current state of knowledge. Furthermore, the primary detection methods employed (static code analysis tools) are inherently targeted at software and are not applicable to hardware. The latter is covered by dynamic analysis tools. KPI2.3: Detect at least two (2) unknown vulnerabilities Currently, no new vulnerabilities have been detected using RESCALE static code analysis tools. After the static code scans are completed on the pilot projects using the final tools, all flagged issues should be carefully and manually reviewed and analyzed to determine whether they represent exploitable unknown vulnerabilities or false positives. The project leverages a combination of traditional static analysis techniques and modern ML/DLenhanced analyzers. While these advanced tools are primarily trained on known patterns, their learning-based nature provides the potential to identify anomalous or previously unclassified RESCALE – Public – Page 32 / 148 D5.5: Validation and Evaluation Report (first version) code patterns. This opens the possibility—albeit a narrow one—for detecting previously unknown vulnerabilities that may not be captured by conventional rule-based tools. Importantly, the tools in use are not narrowly tailored to specific vulnerability classes. Instead, they aim to provide broad coverage across a wide range of potential security issues. This general-purpose capability, combined with expert manual analysis, increases the chances of surfacing novel or overlooked vulnerabilities within the pilot systems. 3.2 Dynamic Testing Module The Dynamic Testing Module identifies vulnerabilities across hardware, firmware, and software in the supply chain. It employs various testing techniques tailored to each component type, such as fuzzing, runtime instrumentation, and protocol testing. This module integrates hardware and software testing components to conduct dynamic assessments and report results to the State Management Module. More details about this module can be found in Deliverable 3.3. 3.2.1 Technical Requirements Table 4 presents a mapping of relevant functional requirements to their implementation and performance results inside the Dynamic Testing Module. The Dynamic Testing Module currently consists of several tools (DSCG Generator, RAISE, EvoMaster, InSpectre Gadget, FATex and Dynamic Hardware Analyzer) and they are validated in order to check which requirements are fulfilled in this iteration of software development. Those assessments provide insights into the module’s effectiveness, accuracy, and efficiency in real-world scenarios. Table 4: List of Initial Functional and Non-Functional Requirements related to the Dynamic Testing Module TRID Title Tool Validation method Status TR008 Flexible dynamic analysis applied to software/- firmware using multiple optimization techniques (classic, fuzzing and ML/DL based) RAISE, EvoMaster Demonstration In progress TR009 Dynamic analysis applied to hardware InSpectre Gadget, Dynamic Hardware Analyzer Demonstration Achieved TR010 Show the results of dynamic analysis EvoMaster, RAISE, FATex, InSpectre Gadget, Dynamic Hardware Analyzer Demonstration Achieved RESCALE – Public – Page 33 / 148 D5.5: Validation and Evaluation Report (first version) Table 4 – continued from previous page TRID Title Tool Validation method Status TR011 Generate DSCG DSCG Generator Demonstration Achieved TR012 Tools/components should be able to ”download and run” All included tools Demonstration Achieved TR013 Documentation Completeness All included tools Demonstration In progress TR014 Docker Compatibility All included tools Demonstration Achieved TR016 Automated Tool/Component Accessibility All included tools Demonstration In progress TR017 Secure Communication Protocol Support RAISE Demonstration In progress TR018 Machine-Readable Data Input/Output All included tools Demonstration In progress TR022 Modules-level ingesting of crucially important data sources (CVE, etc.) FATex Demonstration Achieved TR031 Hardware Assessment Leakage Assessment metric Dynamic Hardware Analyzer Demonstration In progress TR032 Device under Test Taylored Hardware Assessment Dynamic Hardware Analyzer Demonstration In progress TR033 Hardware Assessment Trace Collection speed Dynamic Hardware Analyzer Demonstration In progress TR034 Remote Side Channel Trace Collection Dynamic Hardware Analyzer Demonstration In progress TR035 Hardware Security Assessment Reporting Dynamic Hardware Analyzer Demonstration In progress TR047 Low-level Hardware Assessment – Exploitability InSpectre Gadget Demonstration In progress TR048 Low-level Hardware Assessment – Targets InSpectre Gadget Demonstration Achieved TR049 Low-level Hardware Assessment – Analysis Time InSpectre Gadget Demonstration Achieved TR050 Low-level Hardware Assessment – Reporting InSpectre Gadget Demonstration Achieved 3.2.2 Evaluation Results In the following section, the evaluation results for each identified tool will be presented, demonstrating how each tool contributes to the validation of the defined requirements. RESCALE – Public – Page 34 / 148 D5.5: Validation and Evaluation Report (first version) 3.2.2.1 DSCG Generator The DSCG Generator is designed to streamline the processing of test reports from multiple dynamic testing analysis tools. It consolidates these reports into a single DSCG, formatted as a CycloneDX document, ensuring standardized and efficient data representation. Once generated, the DSCG is transmitted to the RESCALE platform, leveraging key configuration parameters such as a URL endpoint, authentication token, a list of integrating tools, and a shared volume for seamless communication between multiple containers or tools. TR011 Generate DSCG: The DSCG Generator is responsible for collecting reports from dynamic analyzer tools, ensuring that all Dynamic Testing Modules/Tools provide data in a common, machine-readable format. It integrates essential details from these reports, retrieves relevant information from SSCG, and ingests DSCG data from various analysis modules. By processing and standardizing this information according to the common DSCG format specification, the DSCG Generator produces the final, unified DSCG report. TR012 Tools/components should be able to ”download and run”: The Dockerized version of the DSCG Generator are accessible via the Rescale Harbor repository and available for all RESCALE users to download and run. Pilot owners can already achieve this by downloading the DSCG Generator image from the Rescale Harbor and running it within their own pilot environment. TR013 Documentation Completeness: As the tool develops, its documentation is consistently revised and enhanced. Additionally, we provide straightforward and comprehensive Docker documentation. TR014 Docker Compatibility: The DSCG Generator is built on Docker and is created and executed using Docker commands. Before using the DSCG Generator, users must ensure the following resources are prepared: • Required Files: –wrapper.py –Dockerfile –sscg.json • Shared Data Volume: –Ensure the shared-data volume used by the analyzer tool’s Docker container is available. This volume is mounted using the -v shared-data:/output option. The user needs to download the Docker image from Rescale Harbor and then run the downloaded container with the necessary environment variables and shared volume, for example: RESCALE – Public – Page 35 / 148 D5.5: Validation and Evaluation Report (first version) docker run --rm -it -v $(pwd)/output:/output -e SSCG_UID="bc2d4166-497f-42d0-86fc-3164f9cd1093" -e AUTH_TOKEN="myToken" -e RESCALE_URL="https://sm.dev.rescale-project.eu" harbor.rescale-project.eu/wp3/dscg-generator:latest --tools-list "evomaster,raise" Once the container completes processing, if needed, copy the generated output to your local system using: docker cp <container_id>:/output/. $(pwd)/output TR018 Machine-Readable Data Input/Output: The DSCG Generator takes the DSCG portion of the analyzer tool’s output (in JSON format) along with SSCG as input (in CycloneDX format) and produces output in CycloneDX format. The example showcasing the DSCG output, which is generated by integrating reports from two dynamic analyzer tools, RAISE and EvoMaster can be found in Appendix C. 3.2.2.2 RAISE RAISE is a tool developed to improve REST API fuzzing by adding a custom genetic algorithm to RESTler. It makes use of RESTler’s existing request structure and combines it with smarter search strategies. The goal is to increase the precision and coverage of tests, making it easier to find more complex vulnerabilities in web APIs. TR008 Flexible dynamic analysis applied to software/firmware using multiple optimization techniques (classic, fuzzing and ML/DL based): RAISE is a stateful REST API fuzzer, based on RESTler [1] in the process of being enhanced with genetic algorithms. An adaptive fuzzing framework that dynamically adjusts test cases based on real-time feedback is implemented. As the fuzzer explores different API paths, it will refine and prioritize the most promising mutation strategies to maximize code coverage and trigger potential security vulnerabilities. To achieve this, RAISE will integrate a machine learning model that leverages genetic algorithms to optimize fuzzing paths. Genetic algorithms will help in selecting, mutating, and evolving test cases by focusing on those that produce unexpected responses, increase API coverage, or expose security vulnerabilities. The fuzzing process will be designed to evolve iteratively, where test cases leading to unique or critical responses will be given higher priority for further mutations. RAISE requires a Swagger (OpenAPI) file as an input to construct fuzzing test cases. The Swagger file provides a structured definition of the API, including endpoints, request methods, parameters, expected data types, and constraints. By processing this file, RAISE can generate well-formed API requests while strategically mutating parameters to explore edge cases and uncover vulnerabilities. RESCALE – Public – Page 36 / 148 D5.5: Validation and Evaluation Report (first version) TR010 Show the results of dynamic analysis: RAISE categorizes the detected issues into bug buckets based on response patterns, failure types, and API behavior to easily prioritize and classify them. Here is the bug-bucketing classification: •Response Code-Based Categorization – 5XX Errors (Server Failures): Indicate crashes, unhandled exceptions, or improper error handling in the API backend. Example: A fuzzed input causes an internal server error (500 response). – 4XX Errors (Client-Side Failures): Indicate authentication, authorization, or input validation weaknesses. Example: sending a request in which no optional fields are defined, but the server sends a Bad Request 400 message. – Unexpected 2XX Responses: Occur when invalid inputs are incorrectly accepted by the API, indicating logic flaws. Example: Sending an invalid JSON body but still receiving a 200 OK response or an unauthenticated request bypasses access control and receives a 200 response instead of a 401. •Failure Type Categorization – Authentication and Authorization Failures: Occur when unauthorized access is granted. – Unhandled Exceptions and Server Crashes: Caused by malformed requests, type mismatches, or boundary condition errors. – Schema Violations: API responses that do not conform to the expected Open API/Swagger schema. Once the bugs are categorized, RAISE provides reports in the CycloneDX format. TR012 Tools/components should be able to ”download and run”: The dockerized version of RAISE is available on Rescale Harbor repository. Please see TR014 below for additional info. TR013 Documentation Completeness: As the tool evolves, the documentation is constantly being adjusted and updated. The tool has a concise Docker documentation. TR014 Docker Compatibility: Once the RAISE docker image is downloaded from RESCALE Harbor, it can be run simply with the standard docker command: sudo docker run --rm --network=host \ -v "$(pwd)/app/input:/app/input" \ -v "$(pwd)/output:/output" \ -e SWAGGER_FILE_NAME="swagger.json" \ -e AUTH_TOKEN="myAuthToken" \ -e SSCG_UID=SSCGSerialId \ RESCALE – Public – Page 37 / 148 D5.5: Validation and Evaluation Report (first version) -e RESCALE_URL="https://rescale.org" \ raise "python /app/raise.py swagger.json /app/input /app" The Docker file has the following environment variables that can be adjusted: •TOOL INPUT DIR - is the path to the directory where the Swagger file is stored, •SWAGGER FILE NAME - is the name of the Swagger file, •TOOL NAME - is the tool name (eg., raise). TR017 Secure Communication Protocol Support: RAISE inherited security communication protocols TLS 1.2+ and HTTPS from RESTler. It has an option –token via the command line for the Bearer token. TR018 Machine-Readable Data Input/Output RAISE uses a Swagger specification as input (see TR008 for RAISE), and the output is in JSON format (see TR010 for RAISE). Example of RAISE’s output: "claims": [ { "bom-ref": "Claim: RAISE found 6 instances of 500 errors!", "evidence": [ "Evidence: 500 - Internal Server Error encountered at /api/blog/posts/189", "Evidence: 500 - Internal Server Error encountered at /api/blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/blog/posts" ] } ] "evidence": [ { "bom-ref": "Evidence: 500 - Internal Server Error encountered at /api/blog/posts/189", "description": "Evidence gathered by RAISE for 500", "data": [ { "name": "RAISE Gadget Output", "contents": { "attachment": { "contentType": "application/json", "encoding": "base64", "content": "ewogICJyZXF1ZXN0IjogewogICAgIlJlcXVlc..." } } RESCALE – Public – Page 38 / 148 D5.5: Validation and Evaluation Report (first version) } ] } ] Also, RAISE is planned to categorize errors and use Common Weakness Enumeration (CWE). 3.2.2.3 FATex The Firmware Analysis Toolkit (FAT) is a powerful tool designed to automate firmware analysis and emulation for embedded devices. It leverages QEMU [16] for emulation and Binwalk [11] for firmware extraction, enabling researchers and security professionals to analyze firmware without requiring physical hardware. This approach simplifies vulnerability detection and firmware reverse engineering. TR010 Show the results of dynamic analysis: FATex is a completely redesigned and improved version of FAT, incorporating automated supply chain analysis along with vulnerability mapping and detection. Once the firmware is emulated through QEMU, FATex retrieves the vulnerability data related to the firmware’s binary files/executables via an OpenAPI and cross-references it with the CVE database. It extracts firmware binary images, identifies third-party libraries and binaries within the image, and checks for known vulnerabilities associated with these components. All identified vulnerabilities are then compiled into a DSCG report, enhancing the efficiency and accuracy of firmware security analysis with a special focus on software supply-chain visibility. TR012 Tools/components should be able to ”download and run”: The Dockerized version of the FATex is accessible via the Rescale Harbor repository and available for all RESCALE users to download and run. FATex is easy to execute on a given binary, with the default target configuration as follows: USAGE: fatex --firmware <firmware_path> --testcase <testcase_type> OPTIONS: • -f, –firmware Path to the firmware file. • -t, –testcase [Possible values: network, binary] – Specifies the type of test case. • -v, –verbosity Displays detailed step-by-step output and more specific error messages. • -h, –help Displays the help information. • -V, –version Shows the version of FATex. EXAMPLE: RESCALE – Public – Page 39 / 148 D5.5: Validation and Evaluation Report (first version) $fatex -f /app/my_firmware.tar -t binary This requirement has been fulfilled by pilot owners through downloading the FATex image from the Rescale Harbor and deploying it within their respective pilot environments. TR013 Documentation Completeness: As the tool develops, its documentation is consistently revised and enhanced. Additionally, we provide straightforward and comprehensive Docker documentation. The complete documentation can be found at https://gitlab.rescale-project.eu/rescale/dyn-fatex. TR014 Docker Compatibility: FATex is built on Docker and, after being downloaded from Rescale Harbor, is executed using Docker commands. docker run --rm -it -v $(pwd)/input:/input -v shared-data:/output -e SSCG_UID="urn:uuid:bc2d4166-497f-42d0-86fc-3164f9cd1093" -e FASTCVE_URL="http://fastcve:8000" --network binare-network harbor.rescale-project.eu/wp3/fatex_image:latest --verbose fatex -f /input/WNAP320_V2.0.3_firmware.tar -t binary TR018 Machine-Readable Data Input/Output: FATex processes standard firmware image binaries (e.g., .bin), making them machine-readable. The output is generated in JSON format, following the template below, which includes the declarations field: "declarations":{ "claims":[ { "bom-ref":"Claim: ReSCALE Test Suite FATex binaries", "evidence":["Evidence: binaries found", "Evidence: vulnerabilties identified","Evidence: FastCVE output" ] } ], "evidence":[ { "bom-ref":"Evidence: binaries found", "description":"Base64 representation of FATex binary check.", "data":[ { "name":"ReSCALE Test Suite FATex output", "contents":{ "attachment":{ "contentType":"application/json", "encoding":"base64", "content":"W3sibmFtZSI6ImZpcm13YXJlLXVwZ3Jh..." } } } RESCALE – Public – Page 40 / 148 D5.5: Validation and Evaluation Report (first version) ] }, { "bom-ref":"Evidence: FastCVE output", "description":"Base64 representation of FATex binary check with FastCVE output.", "data": [ { "name":"ReSCALE Test Suite FATex + FastCVE output", "contents":{ "attachment":{ "contentType":"application/json", "encoding":"base64", "content":"W3sibmFtZSI6ImZpcm13YXJlLXVwZ3..." } } } ] }, { "bom-ref":"Evidence: vulnerabilties identified", "description":"Base64 representation of FATex found vulnerabilties in CycloneDX format.", "data": [ { "name":"ReSCALE Test Suite FATex vulnerability output", "contents":{ "attachment":{ "contentType":"application/json", "encoding":"base64", "content":"eyJ2dWxuZXJhYmlsaXRpZXMiOlt7ImJvbS1y..." } } } ] } ] } TR022 Modules-level ingesting of crucially-important data sources (CVE, etc): The tool analyzes the following vulnerabilities as part of its output format: { "vulnerabilities": [ { "bom-ref": "vuln-cve-2011-2716", "id": "CVE-2011-2716", "source": { "name": "NVD" }, "description": "The DHCP client (udhcpc) in BusyBox before 1.20.0 allows RESCALE – Public – Page 41 / 148 D5.5: Validation and Evaluation Report (first version) TR013 Documentation Completeness: The complete documentation can be found at https://vusec.github.io/inspectre-gadget/. TR014 Docker Compatibility: InSpectre Gadget provides full Docker support: =$docker run --rm -it \ -v $(pwd)/input:/input \ -v $(pwd)/output:/output \ -e AUTH_TOKEN="myAuthToken" \ -e SSCG_UID=SSCGSerialId \ -e RESCALE_URL="https://rescale.org" \ inspectre_image /app/inspectre /input/binary.so TR018 Machine-Readable Data Input/Output: The input binary is a standard (e.g., ELF) binary, thus machine-readable. The output format is JSON, with the following template: { "declarations": { "claims": [ { "bom-ref": "Claim: InSpectre Gadget found {num-gadgets} exploitable gadgets!", "evidence": [ "Evidence: {evidence}" ] } ], "evidence": [ { "bom-ref": "Evidence: {evidence}", "description": "Evidence gathered by InSpectre Gadget", "data": [ { "name": "InSpectre Gadget Output", "contents": { "attachment": { "contentType": "text/plain", "encoding": "base64", "content": "{base64-output}" } } } ] } ], ] } } RESCALE – Public – Page 48 / 148 D5.5: Validation and Evaluation Report (first version) TR022 Modules-level ingesting of crucially-important data sources (CVE, etc): The tool reasons over the following vulnerabilities (part of the output format): "vulnerabilities": [ { "bom-ref": "vuln-cve-2024-2201", "id": "CVE-2024-2201", "source": { "name": "NIST" }, "description": "A cross-privilege Spectre v2 vulnerability allows attackers to bypass all deployed mitigations, including the recent Fine(IBT), and to leak arbitrary Linux kernel memory on Intel systems.", "affects": [ { "ref": "pkg:hex/input-binary" } ], "ratings": [ { "score": 4.7, "severity": "Medium", "method": "CVSSv3", "vector": "CVSS:3.1/AV:L/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N" } ] } 3.2.2.6 EvoMaster EvoMaster (integrated with FuzzDB) combines automated REST API testing with a comprehensive repository of security payloads. EvoMaster uses search-based and evolutionary testing to explore API behaviors, while FuzzDB provides known attack vectors (e.g., SQLi, XSS, command injection). This integration enables intelligent fuzzing and vulnerability detection, enhancing both functional and security testing capabilities in black-box and gray box testing. TR008 Flexible dynamic analysis applied to software/firmware using multiple optimization techniques (classic, fuzzing and ML/DL based): EvoMaster is the core component in the API fuzzing tool, which is designed and being implemented for security testing and includes a novel integration of three tools(EvoMaster, FuzzDB, coverage.py) and ML models. It has several algorithms implemented, which can be chosen depending on the type of testing. It supports black-box and white-box testing using Evolutionary Algorithms enhanced with Adaptive Hypermutation. FuzzDB is a dictionary of attack patterns and primitives for black-box application fault injection and resource discovery. It is being integrated with EvoMaster. The integration of EvoMaster with the FuzzDB payload repository and the use of ML for fuzzing vector evolution in black-box mode will provide flexibility and optimization techniques to expose unexpected responses and security vulnerabilities. RESCALE – Public – Page 49 / 148 D5.5: Validation and Evaluation Report (first version) TR010 Show the results of dynamic analysis The output of EvoMaster fuzzing on a REST API consists of various artifacts that help analyze the testing process and its findings. The results are categorized into several key components: 1. Discovered Issues & Bugs EvoMaster detects unexpected responses and potential vulnerabilities. • HTTP Errors: (e.g., 500 Internal Server Error, 400 Bad Request) • Assertion Violations: When an expected output does not match actual behavior • Unhandled Exceptions: Indicating robustness issues in the API • Performance Issues: Such as slow responses or excessive memory usage 2. Test Cases (JUnit/Python) EvoMaster generates automated test cases in formats such as JUnit (Java/Kotlin) or Python scripts, which can be used for regression testing. These test cases reproduce the API calls that were found to trigger issues or cover edge cases. 3. Security Findings If EvoMaster is used in security testing mode, it may report: • Injection vulnerabilities (e.g., SQLi, XSS) • Authorization bypass • Broken authentication issues • Exposed sensitive data These findings are linked to FuzzDB payloads . 4. Logs & Execution Reports A summary report is created and included in the output of the tool. This report contains pieces of information such as: detected issues analyzing the schema, consumed search budget, number of covered targets, evaluated tests and evaluated actions, timestamps of API calls, needed budget (percentage of running time), passed time, potential faults and a number of test cases in a python script that can be used to reproduce the faults. All this running information is concluded in a report, which is consumed by the DSCG generator. The report is in json format and contains elements compatible with part of Cyclone DX standard. These elements will be included in the final DSCG. TR012 Tools/components should be able to ”download and run”: The tool and its components can be easily downloaded from the Gitlab repository of the RESCALE project or from the harbor image repository. Running the tool is simple and environmentindependent as it is containerized. The command for pulling the image from RESCALE harbor repository is: docker pull harbor.rescale-project.eu/wp3/evomaster_image:latest RESCALE – Public – Page 50 / 148 D5.5: Validation and Evaluation Report (first version) TR013 Documentation Completeness: The existing documentation of the tool is being updated in parallel with the process of implementing its final version. There is also documentation in Gitlab that describes the use of the docker container. TR014 Docker Compatibility: The tool is containerized and can be run alone or as part of an integrated workflow. The container can be run with the command that follows and contains the minimum parameters needed by the tool. The command is written for the Windows environment. docker run -it --rm ^ -v .\output:/output --network=host ^ -e AUTH_TOKEN="myToken" ^ -e SSCG_UID="urn:cdx:bc2d4166-497f-42d0-86fc-3134f8cd1092" ^ -e RESCALE_URL="https://rescale.org" ^ -e DRY_RUN="false" ^ -e AUTH_HEADER="bearer myToken" ^ -e SUT_URL="http://192.168.2.2:5000/openapi" ^ -e MAX_TIME="5s" ^ --name emCont evomaster_image TR018 Machine-Readable Data Input/Output: The basic input required for EvoMaster to run on a REST API is an OpenAPI/Swagger specification in JSON format, describing the API endpoints, request parameters, and responses. The output of the tool is a JSON file in CycloneDX format. Example: { "declarations": { "claims": [ { "bom-ref": "urn:cdx:bc2d4166-497f-42d0-86fc-3134f8cd1092", "evidence": [ "Evidence: Faulty endpoints" ] } ], "evidence": [ { "bom-ref": "urn:cdx:bc2d4166-497f-42d0-86fc-3134f8cd1092", "description": "Faulty endpoints", "data": [ { "name": "RescTest_faults_Test.py", "contents": { "attachment": { "content": "zWyJjb250ZW50LXR..." } } RESCALE – Public – Page 51 / 148 D5.5: Validation and Evaluation Report (first version) } ] }, { "bom-ref": "urn:cdx:bc2d4166-497f-42d0-86fc-3134f8cd1092", "description": "Summary report", "data": [ { "name": "Summary report", "contents": { "attachment": { "content": "tYXN0ZXIub3JnG1swbSB0byBsZWFybi..." } } } ] } ] }, "vulnerabilities": [ { "id": "test_0_with500", "source": { "name": "Evo Master" }, "affects": [ { "ref": "/ping" } ], "description": "# (500) GET:/ping", "detail": "# Found 2 potential faults. Type-codes: 100, 200", "ratings": [ { "severity": "unknown" } ], "recommendation": "Reproduce the test using the code in the evidence content item." } ] } The use of CWE categorization of the errors is in progress. 3.2.3 KPI Indicators This section summarizes KPIs and their status in relation to the Dynamic Testing Module and the tools it incorporates. RESCALE – Public – Page 52 / 148 D5.5: Validation and Evaluation Report (first version) Table 5: List ofKPIs related to the Dynamic Testing Module KPI Title Tool Validation Method Status KPI1.2 Accuracy of security audits carried out within the project more than 95% for the specified use cases RAISE, EvoMaster, InSpectre Gadget, FATex, Dynamic Hardware Analyzer Evaluation measures based on precision In progress KPI2.1 Highly accurate vulnerability assessment algorithms more than 98% for known benchmark applications RAISE, EvoMaster, InSpectre Gadget, FATex, Dynamic Hardware Analyzer Evaluation measures based on precision In progress KPI2.2 Detect more than 90% of known vulnerabilities at hardware and software level for the pilot’s vulnerable hardware and software solutions RAISE, EvoMaster, InSpectre Gadget, FATex, Dynamic Hardware Analyzer Detection results based on dynamic analysis In progress KPI2.3 Detect at least two (2) unknown vulnerabilities RAISE, EvoMaster, InSpectre Gadget, FATex, Dynamic Hardware Analyzer Detection results based on dynamic analysis Discovered 2+ vulnerabilities (4 CVEs and counting) KPI1.2 Accuracy of security audits carried out within the project more than 95% for the specified use cases: This KPI will be monitored across all tools integrated into the Dynamic Testing Module. Currently, the evaluation strategy has been designed specifically for the RAISE and EvoMaster tool. A similar approach will be adopted for the remaining tools. To ensure reliable security assessments, RAISE and EvoMaster will utilize an accuracy metric known as precision defined as: Precision =TP T P +FP (3) where: •TP (True Positives): Correctly identified API vulnerabilities. •FP (False Positives): Incorrectly flagged vulnerabilities that do not actually exist. The development of RAISE requires the creation of a ground truth dataset containing labeled HTTP statuses that can indicate security issues. The verification process includes two key stages: RESCALE – Public – Page 53 / 148 D5.5: Validation and Evaluation Report (first version) •Genetic Algorithm Integration: RAISE will prioritize high-impact fuzzing sequences by evolving test cases that maximize API coverage and exploit potential weaknesses. •Manual Review and Refinement: Security experts will analyze flagged issues to reduce false positives and improve classification models. As RAISE continues to be developed, its accuracy will be iteratively refined through AI-driven optimizations and continuous testing on real-world API datasets. The goal is to achieve and maintain a 95% or higher accuracy in detecting API security vulnerabilities. Once fully developed, RAISE will provide a state-of-the-art security auditing tool that significantly enhances REST API fuzzing through adaptive learning and intelligent test case generation. The integration of EvoMaster with the FuzzDB repository improves the precision and effectiveness of automated security audits, ensuring that vulnerabilities are detected with a precision rate exceeding 95% for the specified use cases. This is achieved through a combination of intelligent test generation, comprehensive payload injection, and adaptive fuzzing techniques. The FuzzDB repository, a well-established collection of known attack vectors and security test payloads, is integrated into EvoMaster’s fuzzing mechanism. This allows the tool to: • Inject malicious payloads (e.g., SQL injection, XSS, command injection) into API parameters. • Simulate real-world attack scenarios to evaluate API resilience. • Perform context-aware input mutation, refining test cases based on observed system behavior. Using FuzzDB’s structured data set, EvoMaster reduces false negatives, ensuring that security threats are thoroughly exposed. Validation and metrics to ensure accuracy >95%: • False positives are minimized through multi-step verification of detected vulnerabilities. • False negatives are reduced by iteratively refining test coverage using feedback-driven mutation. • The precision and recall of security tests are continuously evaluated, maintaining an accuracy threshold above 95%. The precision-based formula is used to quantitatively ensure the accuracy of security audits. This tells you how many of the issues reported by EvoMaster are actual vulnerabilities. EvoMaster will also use Recall metric defined as: Recall =T P T P +FN (4) where: RESCALE – Public – Page 54 / 148 D5.5: Validation and Evaluation Report (first version) •TP (True Positives): Correctly identified API vulnerabilities. •FP (False Positives): Incorrectly flagged vulnerabilities that do not actually exist. •FN (False Negative): Real vulnerability that the tool failed to detect. This indicates how many of the actual vulnerabilities were correctly found. KPI2.1 Highly accurate vulnerability assessment algorithms more than 98% for known benchmark applications: This KPI will be monitored across all tools integrated into the Dynamic Testing Module. Currently, the evaluation strategy has been designed specifically for the RAISE tool. A similar approach will be adopted for the remaining tools. In the context of RAISE, this KPI measures the accuracy of the tool in detecting vulnerabilities in known benchmark applications. The goal is to achieve an accuracy of more than 98%, ensuring that the tool reliably identifies security vulnerabilities in real-world and synthetic test scenarios. To systematically assess RAISE’s effectiveness, the evaluation follows these key steps: 1. Benchmark application: An application under test, where the vulnerabilities are already known. The benchmark application contains a predefined set of vulnerabilities, either based on an established knowledge base of known security issues or inserted through a structured vulnerability injection methodology. 2. Analysis using RAISE: Each benchmark application is analyzed using RAISE’s advanced fuzzing techniques, which incorporate genetic algorithms to improve the efficiency of security testing. 3. Ground truth comparison: The number of actual vulnerabilities present in the benchmark application is documented, providing a reference point for evaluation. 4. Accuracy computation: The assessment results from RAISE are compared with the documented vulnerabilities, and the overall accuracy is computed using standard precisionbased metrics. The goal is to exceed an average accuracy of 98% across all tested applications. To evaluate RAISE’s performance, a set of benchmark applications will be utilized, where vulnerabilities are manually or synthetically inserted and validated by security experts. The module will be executed on these applications, and its results will be cross-referenced with the established ground truth. Since RAISE utilizes a genetic algorithm-based fuzzing model, its performance will improve as the dataset used for optimization expands. The ongoing development of a labeled dataset of API vulnerabilities, generated using RESTler’s fuzzing framework and refined through genetic selection, will further enhance the accuracy of its vulnerability assessments. By continuously refining its mutation strategies and improving its benchmarking dataset, RAISE will be progressing toward achieving the target of 98% accuracy in vulnerability assessments across known benchmark applications. RESCALE – Public – Page 55 / 148 D5.5: Validation and Evaluation Report (first version) KPI2.2 Detect more than 90% of known vulnerabilities at hardware and software level for the pilot’s vulnerable hardware and software solutions: This KPI will be monitored across all tools integrated into the Dynamic Testing Module but in this moment we have focused on identifying side-channel vulnerabilities across various platforms, including FPGA, CPU, and other hardware. We will validate this KPI across all available tools related to Dynamic Testing Module and report its status in the Deliverable D5.6. The Hardware analyzer specifically tests for side-channel attacks, such as power analysis, electromagnetic emissions, and timing attacks, on these different platforms. We are in the process on identifying relevant CVEs if applicable and selecting side-channel attacks residing in the academic literature. By cross-referencing the hardware with known CVEs or relevant research works on side-channel attacks, we will be able to assess potential weaknesses and conduct empirical tests to detect side-channel leakage and related risks, ensuring thorough security analysis across the hardware and software. The component must have the ability to detect more than 90% of these selected CVEs (if applicable) or other known vulnerabilities from the literature, within the pilot environment. KPI2.3 Detect at least two (2) unknown vulnerabilities: The KPI highlights the discovery of significant security issues and multiple CVEs, showcasing the project’s contribution to advancing security research. Currently, InSpectre Gadget has already identified more than 2 potential vulnerabilities. Other tools from Dynamic Testing Module will be applied to further evaluate this KPI and potentially uncover more unknown vulnerabilities. InSpectre Gadget focused on identifying microarchitectural vulnerabilities in modern computing architectures. Key achievements include: •SLAM: Discovery of a new covert channel amplifying the impact of Spectre attacks on CPUs supporting Intel LAM, exposing vulnerabilities in noncanonical address translation paths. •Spectre-MTE: Identification of a new speculative oracle in ARM MTE-based solutions capable of disclosing tag assignments based on random tagging. •Native BHI: First demonstration of native cross-privilege Spectre v2 attacks, resulting in CVE-2024-2201 . •GhostRace: Discovery of speculative race conditions, representing a novel class of speculative execution vulnerabilities, leading to CVE-2024-2193 and CVE-2024-26602. 3.3 TrustOR TrustOR facilitates the generation, validation, and management of TBOMs by securely aggregating information from the Assessment Domain, including Software Bill of Materials (SBOM)s, Static Supply Chain component Guarantee (SSCG)s and Dynamic Supply Chain component Guarantee (DSCG)s. By establishing trusted references between these elements, TrustOR creates a cohesive security profile that can be verified and traced throughout the supply chain lifecycle. RESCALE – Public – Page 56 / 148 D5.5: Validation and Evaluation Report (first version) 3.3.1 Technical Requirements The validation process for TrustOR focuses on assessing its capability to securely manage data which are prerequisites for TBOMs and provide reliable API access for stakeholders. The following table presents the technical requirements mapped to TrustOR’s implementation, along with their validation methods and current status. Table 6: List of Initial Functional and Non-Functional Requirements related to the TrustOR component TRID Title Tool Validation method Status TR001 The TBOM should be based on an existing BOM Standard Format TrustOR Demonstration Achieved TR002 The TBOM must have a well defined structure Format TrustOR Demonstration Achieved TR012 Tools/components should be able to ”download and run” TrustOR Demonstration Achieved TR013 Documentation Completeness TrustOR Demonstration In progress TR014 Docker Compatibility TrustOR Demonstration Achieved TR015 OpenAPI-compliant API Documentation TrustOR Demonstration Achieved TR017 Secure Communication Protocol Support TrustOR Demonstration Achieved TR018 Machine-Readable Data Input/Output TrustOR Demonstration Achieved TR021 RESCALE common Storage Repository TrustOR Demonstration Achieved TR041 TBOM Life cycle management TrustOR Demonstration In progress TR042 TBOM Generation - prerequisites gathering TrustOR Demonstration Achieved TR043 TBOM Generation - merging information TrustOR Demonstration Achieved TR044 TBOM Generation - security & trust TrustOR Demonstration In progress TR045 TBOM Validation TrustOR Demonstration In progress TR046 TBOM Validation - check TBOM is up-to-date TrustOR Demonstration In progress RESCALE – Public – Page 57 / 148 D5.5: Validation and Evaluation Report (first version) Figure 10: Request / Response of endpoint for getting TBOM metadata Figure 11 also gives the perspective of the TBOM lifecycle, as presented in the Deliverable 4.1 ”Specification of TBOM Structure” in relation to these logs/steps. Steps 1 and 2 take place sequentially inside TrustOR component. Steps 3 and 4 are work in progress and occur asynchronously when the Security Assurance component submits a new Bill of Vulnerabilities (BOV). It is important to point out that every time a new BOV is submitted, steps 3 and 4 take place again. This practically means that TrustOR updates the TBOM’s BOV, sets the potentially new severity level and then deprecates the old TBOM / adds the updated one by utilizing the relative functions provided by the Blockchain. RESCALE – Public – Page 64 / 148 D5.5: Validation and Evaluation Report (first version) Figure 11: logs / steps mapped to the TBOM lifecycle introduced in D4.1 TR042 - TBOM Generation - prerequisites gathering: Figure 12 displays TrustOR’s Swagger API endpoints dedicated to submitting the prerequisite security assessment artifacts (SBOM, SSCG, and DSCG) required for TBOM generation, satisfying requirement TR042. Figure 12: TrustOR endpoints for receiving SBOMs, SSCGs & DSCGs TR043 - TBOM Generation - merging information: In the following json from Listing 3, which is a valid CycloneDX part of a TBOM example, the reader can notice the externalReferences to the related SBOM, SSCG and DSCG urls: ... "components": [ { RESCALE – Public – Page 65 / 148 D5.5: Validation and Evaluation Report (first version) "externalReferences": [ { "comment": "Link to BOV", "type": "distribution", "url": "https://sm.dev.rescale-project.eu /bovGet/urn:uuid:1b37a02f-1267-4f57-a2f3-2f8d0177996b" } ], "name": "BOV", "purl": "pkg:hex/[email protected]", "type": "file" }, { "bom-ref": "dscg-ref", "description": "a link to the DSCG of: grisp", "externalReferences": [ { "comment": "DSCG URL", "type": "distribution", "url": "https://sm.dev.rescale-project.eu /dscgGet/urn:uuid:3e671687-391b-41f5-a30f-a58921a69b78" } ], "name": "DSCG", "purl": "pkg:hex/[email protected]", "type": "file" }, { "bom-ref": "sbom-ref", "description": "a link to the SBOM of: grisp", "externalReferences": [ { "comment": "SBOM URL", "type": "distribution", "url": "https://sm.dev.rescale-project.eu /sbomGet/urn:uuid:7b0ba9e7-d293-4d05-a923-9527728610b9" } ], "name": "SBOM", "purl": "pkg:hex/[email protected]", "type": "file", "version": "1" }, { "bom-ref": "sscg-ref", "description": "a link to the SSCG of: grisp", "externalReferences": [ { "comment": "SSCG URL", "type": "distribution", "url": "https://sm.dev.rescale-project.eu RESCALE – Public – Page 66 / 148 D5.5: Validation and Evaluation Report (first version) /sscgGet/urn:uuid:bc2d4166-497f-42d0-86fc-3134f9cd1073" } ], "name": "SSCG", "purl": "pkg:hex/[email protected]", "type": "file" } ], ... "dependencies": [ { "ref": "bov-ref" }, { "ref": "dscg-ref" }, { "ref": "sbom-ref" }, { "ref": "sscg-ref" }, { "dependsOn": [ "bov-ref", "dscg-ref", "sbom-ref", "sscg-ref" ], "ref": "tbom1-metadata" } ], ... Listing 3: Part from TBOM displaying ref dependencies TR044 - TBOM Generation - Security & Trust: In Figure 13 the metadata of a Blockchain transaction is displayed. In particular they are related to TBOM with serialNumber: urn:uuid:0495bf94-f202-4fb0-95d4-86f374b66091 and one can spot the hash of the TBOM object that was added to the blockchain as well as the transactionHash and the blockNumber. RESCALE – Public – Page 67 / 148 D5.5: Validation and Evaluation Report (first version) Figure 13: Blockchain metadata for specific TBOM TR045 - TBOM Validation: Figure 14 demonstrates two different cases of validating a TBOM with the relative endpoint provided by TrustOR: /api/utils/validate tbom/serial number The user has to provide the TBOM serialNumber in both cases and internally TrustOR communicates with the blockchain to check whether the TBOM hash is valid. On the left, we see the case of a valid TBOM and on the right we see the case of a TBOM which is not valid, because its hash did not exist in the blockchain. RESCALE – Public – Page 68 / 148 D5.5: Validation and Evaluation Report (first version) Figure 14: Example of valid and not-valid TBOM TR046 - TBOM Validation - check TBOM is up-to-date With the current design, TR046 is also covered by TR045, since every security update coming from the Security Assurance component directly triggers the TBOM updating process, which practically includes the deprecation of the existing TBOM and the creation of a new, updated one. This way, active and valid TBOMs always include the latest updates, whereas outdated TBOMs are deprecated objects in the Blockchain and as a result, trying to validate them will return: False. 3.3.3 KPI Indicators Table 7: List of KPIs related to the TrustOR Component KPI Title Tool Validation Method Status KPI3.1 Write at least 3 software and hardware TBOM entries in the RESCALE public blockchain TrustOR Demonstration In progress KPI3.2 Generate at least 3 software and hardware TBOM files TrustOR Demonstration In progress KPI3.3 Update and validate at least 3 software and hardware TBOM enabled applications TrustOR Demonstration Preliminary stage For KPIs 3.1, 3.2 and 3.3, the quantitative aspect are not yet be covered. At the current stage, RESCALE – Public – Page 69 / 148 D5.5: Validation and Evaluation Report (first version) one test input has been used to demonstrate the functionality end-to-end. This serves as a proofof-concept and confirms that the TrustOR component and the Trust Orchestrator framework can support addition of TBOMs with minimal overhead. The mechanism is in place, operational and more TBOMs can be added seamlessly once tooling from WP3 generates valid SBOMs, SSCGs and DSCGs for specific software or hardware components. To fully address this KPI, we are coordinating with pilot-partners to obtain TBOMs from at least two more applications. The infrastructure is ready to accommodate them, ensuring that the KPI targets can be met. KPI3.1 - Write at least 3 software and hardware TBOM entries in the RESCALE public blockchain In Figure 15 we see a test component on the left, that has an available TBOM, specifically with serialNumber: urn:uuid:urn:uuid:d96c6a6c-5710-49f9-9335-04c5c33f06fd. On the right, we see the information from TrustOR which reveals that it has been added to the blockchain, along with the relative metadata, like: the hash of the TBOM, the transactionHash and the blockNumber. Figure 15: TBOM visible via Dashboard also added in the blockchain KPI3.2 - Generate at least 3 software and hardware TBOM files Figure 16 displays the TrustOR endpoint that returns a TBOM payload. This can easily be wrapped in a .json file and downloaded by a user via a browser application. For simplicity, this screenshot displays only a part of the TBOM but the reader can see the full TBOM at the appendix D. RESCALE – Public – Page 70 / 148 D5.5: Validation and Evaluation Report (first version) Figure 16: Endpoint for receiving a TBOM file KPI3.3 - Update and validate at least 3 software and hardware TBOM enabled applications Figure 17 includes the validation part of this KPI. On the left, the endpoint for uploading and validating a TBOM is being demonstrated and on the bottom right, the returned result shows that the TBOM is valid. Provided a TBOM file, this endpoint looks up the TBOM serialNumber and (if found) compares them and then validates it against its hash in the blockchain. Figure 17: Endpoint for validating a TBOM file RESCALE – Public – Page 71 / 148 D5.5: Validation and Evaluation Report (first version) 3.4 Ledger Infrastructure Ledger Infrastructure is a blockchain-based platform that securely stores and manages TBOM integrity data in the form of hashes. By leveraging decentralized ledger technology, it creates an immutable record of TBOM components, SBOMs, SSCGs, and DSCGs. This setup ensures that any alterations to TBOM data are detectable through hash mismatches, providing a tamper-evident audit trail. The validation process for Ledger Infrastructure focuses on assessing its capability to securely store and verify TBOM hashes, ensuring reliable data integrity and stakeholder access throughout the supply chain lifecycle. 3.4.1 Technical Requirements Table 8: List of Initial Functional and Non-Functional Requirements related to the Ledger Infrastructure component TRID Requirement Title Component Validation Method Status TR012 Tools/components should be able to ”download and run” Ledger Infrastructure Deployment and accessibility tests Achieved TR013 Documentation Completeness Ledger Infrastructure Repository README.md Achieved TR014 Docker Compatibility Ledger Infrastructure Demonstration Achieved TR016 Automated Tool/Component Accessibility Ledger Infrastructure Demonstration Achieved TR017 Secure Communication Protocol Support Ledger Infrastructure Demonstration Achieved TR018 Machine-Readable Data Input/Output RESCALE Framework Demonstration Achieved TR019 Flexible choice of Ledger/Blockchain for integration Ledger Infrastructure Integration testing In Progress TR020 Decoupled blockchain deployment Ledger Infrastructure Architecture review Achieved TR036 Provide blockchain underlying infrastructure Ledger Infrastructure Assessment of blockchain environments Achieved TR037 Provide a homogeneous metrics for blockchain comparison Ledger Infrastructure Metrics evaluation Achieved TR038 Provide smart contracts deployment strategy Ledger Infrastructure Smart contract testing Achieved TR039 Smart contract data structure Ledger Infrastructure Data structure validation Achieved RESCALE – Public – Page 72 / 148 D5.5: Validation and Evaluation Report (first version) TR040 Implement strategies for bugs-free deployment of smart contracts Ledger Infrastructure Code review and testing In progress TR041 TBOM Life cycle management Ledger Infrastructure Lifecycle testing In progress 3.4.2 Evaluation Results TR012 - Tools/components should be able to ”download and run”: As seen in Figure 18, the Ledger Infrastructure component can be easily downloaded with the following Docker command: docker pull harbor.rescale-project.eu/rescale-all/besu:2025-02-11 and run with: docker run -p 127.0.0.1:8545:8545 \ harbor.rescale-project.eu/rescale-all/besu:2025-02-11 Figure 18: Ledger Infrastructure component downloaded and run with Docker CLI TR013 - Documentation Completeness: RESCALE – Public – Page 73 / 148 D5.5: Validation and Evaluation Report (first version) • adherence to secure coding practices, such as input validation and proper error handling; • implementation of access control measures using modifiers • conduct a basic security audit using automated tools. TR041 - TBOM Life cycle management: The TBOM lifecycle management was comprehensively described in deliverable D4.1 – Specification of the TBOM Structures. Building upon this specification, the implementation phase has focused on developing the necessary infrastructure to support these lifecycle requirements. As already documented in previous requirements (TR038, TR039), the Ledger Infrastructure provides the required tools under the form of smart contract methods to manage core states (Add, Read, Deprecate) of the TBOM life cycle. To fully support all the supply chain requirements of the RESCALE project, the smart contract will be enhanced with additional methods and data structures: • enhanced management of TBOM ownership with hierarchical accounts functionality; • support for additional metadata bound to TBOM entries. 3.4.3 KPI Indicators Table 9: List of KPIs related to the Ledger Infrastructure Component KPI Title Tool Validation method Status KPI3.1 Write at least 3 software and hardware TBOM entries in the RESCALE public blockchain Ledger Infrastructure Demonstration In progress KPI3.3 Update and validate at least 3 software and hardware TBOM enabled applications Ledger Infrastructure Demonstration In progress KPI3.1 - Write at least 3 software and hardware TBOM entries in the RESCALE public blockchain: This KPI will be achieved during the following phases of the project. All the necessary technologies are in place and only necessitate that all other components of RESCALE be composed and run for the demonstration. The Ledger Infrastructure has demonstrated its readiness to fulfill this KPI through several technical requirements that have already been achieved: • TR012 confirms that the Ledger Infrastructure can be easily downloaded and run using Docker commands, providing the foundation for blockchain operations RESCALE – Public – Page 80 / 148 D5.5: Validation and Evaluation Report (first version) • TR014 validates the Docker compatibility of the Ledger Infrastructure, specifically utilizing a dockerized image of Hyperledger BESU • TR016 demonstrates the automated accessibility through a dedicated Python library that provides a streamlined interface for blockchain interactions • TR038 shows the successful deployment of smart contracts with key operations (Add, Read, Deprecate) that will be used to write TBOM entries to the blockchain • TR041 confirms that the Ledger Infrastructure provides the required tools to manage core states of the TBOM life cycle KPI3.3 - Update and validate at least 3 software and hardware TBOM enabled applications: This KPI will also be achieved during the following phases of the project. The technical foundation is completely established and only requires the integration of all RESCALE components for the demonstration. The readiness of the Ledger Infrastructure to support this KPI is evidenced by: • TR012 and TR014 which ensure the Ledger Infrastructure’s availability and compatibility with Docker environments • TR016 which provides the Python library interface needed for applications to interact with the blockchain • TR018 which ensures machine-readable data input/output through standardized formats, facilitating seamless integration with other systems and components • TR038 which implements the smart contract functionality necessary for updating and validating TBOM entries • TR041 which provides the tools to manage the TBOM lifecycle, including updates and validation operations 3.5 Security Assurance Module The Continuous Security Assurance Mechanism operates on a Security and Privacy Assurance Platform that dynamically evaluates the relevance and accuracy of Trust-Based Object Models (TBOMs) for hardware and software components in the face of emerging vulnerabilities and faults. Whenever new security issues arise that were not accounted for during the initial creation of a TBOM, the platform activates a targeted update process, ensuring that the supply chain remains secure and resilient over time. 3.5.1 Technical Requirements RESCALE – Public – Page 81 / 148 D5.5: Validation and Evaluation Report (first version) Table 10: List of Initial Functional and Non-Functional Requirements related to the Continuous Security Assurance component TRID Requirement Title Component Validation Method Status TR051 New Static/dynamic report Continuous Security Assurance Integration testing Achieved TR052 Periodic assessment Continuous Security Assurance Integration testing Achieved TR053 Response Time Continuous Security Assurance Integration testing Achieved TR054 High Availability Continuous Security Assurance Demonstration Achieved TR055 Interoperability Continuous Security Assurance Integration testing and Demonstration Achieved 3.5.2 Evaluation Results TR051 - New Static/dynamic report The system assesses at run time if the TBOM reports about a software or hardware components remains valid against currently known vulnerabilities. The system must be able to ingest the results of the static and dynamic analyzer, get the identified vulnerabilities from the National Vulnerability Database and based on the CPE of the assets (software or hardware), find the vulnerabilities that can apply. TR052 - Periodic assessment The security assurance system assesses at specific intervals (e.g., daily), if the valid TBOMs remain valid against currently known vulnerabilities retrieved from the National Vulnerability Database. Based on the CPE of the assets (software or hardware), the system should identify applicable vulnerabilities. TR053 - Response Time The security assurance system provides responses to actions/assessments within seconds under normal operation conditions. This can be measured analyzing the time that the provided services take to reply to the requests. RESCALE – Public – Page 82 / 148 D5.5: Validation and Evaluation Report (first version) Figure 26: Response Time TR054 - High Availability The security assurance system will be available 99.5% of the time, excluding planned maintenance windows. The security assurance platform itself has been implemented in such a way to provide 24/7 functionality, the cloud instances availability will be directly related to the SLA given by the cloud providers, which is higher than 99.5%. TR055 - Interoperability The security assurance system exchanges data with other components and tools via welldefined APIs, supporting formats such as JSON and XML. The following is an example of one API used in STS that specify JSON as response type, with the use of the ”produces = MediaType.APPLICATION JSON VALUE”. ... @PostMapping( path = "/organisations/{organisationId}/projects/{projectId} /owners/{ownerId}", consumes = MediaType.MULTIPART_FORM_DATA_VALUE, produces = MediaType.APPLICATION_JSON_VALUE) public ResponseEntity<SuccessMessage> importSBOM( @PathVariable @Positive Long projectId, @PathVariable @Positive Long organisationId, @PathVariable UUID ownerId, @RequestParam("sbomFormat") @Valid SBOMFormatEnum sbomFormat, @RequestPart("file") MultipartFile file) { ... 3.5.3 KPI Indicators Table 11: List of KPI Indicators related to the Continuous Security Assurance component TRID Requirement Title Component Validation Method Status KPI2.1 Highly accurate vulnerability assessment algorithms more than 98% for known benchmark applications Continuous Security Assurance Demonstration In progress RESCALE – Public – Page 83 / 148 D5.5: Validation and Evaluation Report (first version) KPI2.2 Detect more than 90% of known vulnerabilities at hardware and software level for the pilot’s vulnerable hardware and software solutions Continuous Security Assurance Demonstration In progress KPI4.3 Detection of more than 95% of total insecure components during piloting activities for known benchmark applications Continuous Security Assurance Demonstration In progress KPI2.1 – Highly accurate vulnerability assessment algorithms (98%) for known benchmark applications This KPI will be addressed through the execution of standard benchmarking tests in the near future. The objective is to validate the accuracy and reliability of the vulnerability assessment algorithms by applying them to recognized benchmark applications and verifying that they meet or exceed the 98% accuracy threshold. To evaluate its performance, a set of benchmark TBOMs will be utilized, where vulnerabilities are manually or synthetically inserted and validated by security experts. The module will be executed on these TBOMs, and its results will be crossreferenced with the established ground truth. KPI2.2 – Detection of more than 90% of known vulnerabilities at the hardware and software level for the pilot’s vulnerable hardware and software solutions This KPI will be achieved at the end of the piloting activities. The necessary technologies supporting the Continuous Security Assurance component have already been developed and are fully operational. Final integration with the rest of the RESCALE architecture is needed to enable the full system to perform end-to-end vulnerability detection. KPI4.3 – Detection of more than 95% of total insecure components during piloting activities for known benchmark applications This KPI will be achieved at the end of the piloting activities. The Continuous Security Assurance infrastructure is in place and technically capable of identifying insecure components with high accuracy. Final integration with the rest of the RESCALE architecture is needed to enable the full system to perform end-to-end vulnerability detection during piloting activities. To evaluate its performance, a set of insecure libraries/components will be utilized, where vulnerabilities are manually or synthetically inserted and validated by security experts. The module will be executed on the generated TBOMs, and its results will be cross-referenced with the established ground truth. RESCALE – Public – Page 84 / 148 D5.5: Validation and Evaluation Report (first version) 3.6 State Manager Module The State Manager acts as the central orchestrator for the RESCALE platform’s backend, handling API requests from all components outside the management layer, providing API endpoints for the rest of the RESCALE components to enable their functionalities, such as the Dashboard, the Static Code Analysis Module, Dynamic Testing Module, the TrustOR, and the Security Assurance pipeline, allowing for inter-connectivity, and enabling the data flow inside the RESCALE platform. 3.6.1 Technical Requirements TRID Requirement Title Component Validation Method Status TR007 Generate SSCG State Manager Integration testing with Static Analysis Module Achieved TR011 Generate DSCG State Manager Integration testing with Dynamic Analysis Module In progress TR012 Tools/components should be able to ”download and run” State Manager Deployment and accessibility tests Achieved TR013 Documentation Completeness State Manage README.md and OpenAPI In progress TR014 OpenAPI-compliant API Documentation State Manager OpenAPI validation via Swagger In progress TR018 Machine-readable Data Input/Output State Manager Demonstration Achieved TR021 RESCALE Common Storage Repository State Manager Demonstration Achieved TR056 User role-based access control, Authentication State Manager Demonstration Achieved TR057 User Notifications State Manager Demonstration Scheduled for M21 3.6.2 Evaluation Results TR007 - Generate SSCG: The State Manager receives the SSCG report from the Static Analysis Module, extracts information such as the Serial Number and the version, associates it with the corresponding component and forwards it to TrustOR for persistent storing as shown in Figure 27. The Dashboard retrieves the SSCG for visualization through the State Manager. RESCALE – Public – Page 85 / 148 D5.5: Validation and Evaluation Report (first version) Figure 27: State Manager SSCG Generation process TR011 - Generate DSCG: The State Manager receives the DSCG report from the Dymanic Analysis Module, extracts information such as the Serial Number, associates it with the corresponding component and SSCG report and forwards it to TrustOR for persistent storing as shown in Figure 28. Figure 28: State Manager SSCG Generation process TR012 - Tools/components should be able to ”download and run”: State Manager is availRESCALE – Public – Page 86 / 148 D5.5: Validation and Evaluation Report (first version) able for download via harbor as docker image as depicted in Figure 29. Figure 29: State Manager docker pull from harbor Afterwards, it can be run with the docker compose command as shown in Figure 30 Figure 30: State Manager running as docker image TR013 - Documentation Completeness: Documentation for the State Manager component, about its configurations and deployment details, can be found at the README.md under RESCALE’s State Manager GitLab repository. TR014 - OpenAPI-compliant API Documentation: The State Manager provides an OpenAPIcompliant API documentation. It is implemented using Node.js with the Express framework and leverages Swagger to generate an interactive endpoint documentation as shown in Figure 31. RESCALE – Public – Page 87 / 148 D5.5: Validation and Evaluation Report (first version) Figure 31: State Manager OpenAPI Documentation TR018 - Machine-readable Data Input/Output: State Manager handles API requests and responses using structured, machine-readable formats. Specifically, it follows the JSON and Cyclone DX format, ensuring seamless integration with other components within the RESCALE ecosystem. The API endpoints, documented via Swagger, define the expected input/output structures, enabling automated processing and interoperability with external tools. Figure 32 shows an example of such a procedure, using POSTMAN to send API requests in a machine-readable input/output. On the upper part, the request is shown, structured in JSON format, and on the lower part the response, following a JSON format. RESCALE – Public – Page 88 / 148 D5.5: Validation and Evaluation Report (first version) Figure 32: Machine Readable I/O TR021 - RESCALE Common Storage Repository: State Manager uses a internal database implemented in MongoDB, to store information about RESCALE Users and their associated components. Figure 33 showcases the MongoDB collections used by State Manager. Figure 33: MongoDB Collections utilised by State Manager TR056 - User role-based access control, Authentication: This requirement is fulfilled through the integration of Keycloak, which provides robust authentication and role-based access conRESCALE – Public – Page 89 / 148 D5.5: Validation and Evaluation Report (first version) Figure 38: Getting the Access Token and the Serial Number from the Dashboard TR012 - Tools/components should be able to ”download and run”: The Dashboard component is available for download via harbor as docker image as depicted in Figure 39. Figure 39: Dashboard Docker pull from harbor Afterwards, it can be run with the docker compose command as shown in Figure 40 Figure 40: Dashboard running as Docker image TR056 - User role-based access control, Authentication: Figure 41 demonstrates the interface and functionalities available to the Producer role. Producers have the ability to register a component, view their registered components, access the RESCALE – Public – Page 96 / 148 D5.5: Validation and Evaluation Report (first version) SSCG, and generate a static analysis report. The role-based access control mechanism ensures that only producers can perform these actions, restricting access to functionalities intended for consumers. Figure 41: Producer’s View Figure 42 illustrates the interface and functionalities available to the Consumer role. Consumers have an overview of all registered components, allowing them to perform dynamic analysis on the available components. Additionally, they can inspect the JSON output of the static analysis if it is available, but they do not have the rights to perform the static analysis themselves. Figure 42: Consumer’s View TR057 - User Notifications: The implementation of TR057 has not yet started. Consequently, at this stage, no evaluation activities have been conducted, and evidence for its validation is not available. The progress on this requirement will be reported in future updates once implementation begins. RESCALE – Public – Page 97 / 148 D5.5: Validation and Evaluation Report (first version) 3.7.3 KPI Indicators KPI Title Component Validation Method Status KPI 4.4 Construction of an informative mechanism for both security and non-security experts Dashboard Demonstration In progress The Dashboard serves as the main interface for platform users, offering tailored views based on user roles (e.g., producer or consumer). It is designed to support both interaction and visualization of assessment workflows and security information. It provides differentiated views and options for different user roles. It displays vulnerabilities by severity levels, aiding prioritization and quick risk assessment. Additionally, it includes filters, search, and component version tracking to facilitate navigation. RESCALE – Public – Page 98 / 148 D5.5: Validation and Evaluation Report (first version) 4 System-Level Validation System-level validation ensures that the integrated components function correctly as a cohesive unit, meeting overall system requirements and performance expectations. It assesses the interactions between components, their interoperability, and the system’s ability to handle realworld scenarios. This process involves comprehensive testing techniques, including integration testing, end-to-end validation, stress testing, and scenario-based simulations, to verify that the system operates reliably under various conditions. By identifying inconsistencies, bottlenecks, and potential failures at this stage, system-level validation plays a crucial role in guaranteeing the robustness and effectiveness of the final product. In this phase of the project, a system-level validation will be conducted on the integrated MVP solution. This upcoming validation phase will confirm that all components work together seamlessly in real-world scenarios, fulfilling the project’s functional and non-functional requirements. 4.1 Technical Requirements Table 16 presents the traceability matrix of the initial functional and non-functional requirements. This matrix serves as a structured overview linking each identified requirement to its corresponding system components. It ensures that all functional and non-functional aspects are accounted for and properly addressed throughout the development process, providing clear visibility and traceability from the early stages of requirements definition to implementation. The matrix provides an overview of the requirements’ implementation status, categorized as Scheduled (not yet started but there is plan), In Progress (partially implemented), and Yes (fully implemented). Also, the presented traceability matrix illustrates which system components need to collaborate in order to fulfill each specific functional or non-functional requirement. The matrix highlights dependencies and integration points necessary for successful implementation and validation. Table 16: Traceability Matrix of Initial Functional and Non-Functional Requirements Req. ID Software Static Code Analysis Module Dynamic Testing Module TrustOR Ledger Infrastructure Security Assurance Module State Manager Module + Dashboard TR001 Yes TR002 Yes TR003 Yes TR004 In progress TR005 In progress TR006 In progress Yes TR007 In progress Yes TR008 In progress TR009 Yes RESCALE – Public – Page 99 / 148 D5.5: Validation and Evaluation Report (first version) Table 16 – continued from previous page Req. ID Software Static Code Analysis Module Dynamic Testing Module TrustOR Ledger Infrastructure Security Assurance Module State Manager Module TR010 Yes Yes TR011 Yes In progress TR012 In progress Yes Yes Yes Yes TR013 In progress In progress In progress Yes In progress TR014 In progress Yes Yes Yes In progress TR015 Yes TR016 In progress In progress Yes TR017 Scheduled In progress Yes Yes TR018 In progress In progress Yes Yes Yes TR019 Yes TR020 Yes TR021 Yes Yes TR022 Scheduled Yes TR023 Scheduled TR024 In progress TR025 In progress TR026 Scheduled TR027 In progress TR028 Scheduled TR029 Yes TR030 Yes TR031 In progress TR032 In progress TR033 In progress TR034 In progress TR035 In progress TR036 Yes TR037 Yes TR038 Yes TR039 Yes TR040 In progress TR041 In progress In progress TR042 Yes TR043 Yes TR044 In progress TR045 In progress TR046 In progress TR047 In progress TR048 Yes TR049 Yes TR050 Yes TR051 Yes RESCALE – Public – Page 100 / 148 D5.5: Validation and Evaluation Report (first version) Table 16 – continued from previous page Req. ID Software Static Code Analysis Module Dynamic Testing Module TrustOR Ledger Infrastructure Security Assurance Module State Manager Module TR052 Yes TR053 Yes TR054 Yes TR055 Yes TR056 Yes TR057 Scheduled 4.2 Impact KPI Indicators The table below presents a traceability matrix that links each Impact Key Performance Indicator (iKPI) to the evaluation method and RESCALE module/tool responsible for its validation. This approach ensures that all impact measurements are systematically aligned with the project’s core technologies and validation mechanisms. The traceability matrix provides: • A structured overview of impact KPIs, defining their scope and objectives. • Validation methods used to assess their effectiveness, such as benchmark testing, stakeholder feedback, and compliance mapping. • Mapping to RESCALE tools, ensuring that each iKPI is supported by the appropriate module for measurement and verification. By incorporating this structured approach, the RESCALE project can demonstrate the effectiveness of its technical solutions and the broader impact on supply chain security, regulatory frameworks, and industry adoption. 4.2.1 Key Impact KPIs and Their Evaluation Framework The following iKPIs serve as benchmarks for assessing RESCALE’s impact and the modules/- tools that contribute to their validation. iKPI Description Validation Method RESCALE Module/- Tool iKPI1 Increased trust and adoption of blockchain-assisted TBOM updates by 30% Stakeholder surveys, blockchain audit Trust Orchestrator (TrustOR), Ledger Infrastructure RESCALE – Public – Page 101 / 148 D5.5: Validation and Evaluation Report (first version) iKPI2 AI models developed for enhancing security testing accuracy (>5 models) AI model validation, component integration reports Static Code Analysis Module, Dynamic Testing Module iKPI3 90% vulnerability detection coverage in secure supply chains Pilot case assessments, benchmark testing Static Code Analysis, Dynamic Testing Module iKPI4 RESCALE compliance with NIS directive in 2 pilot scenarios Regulatory compliance mapping, pilot feedback Trust Orchestrator (TrustOR), Security Assurance Module iKPI5 Security assessment accuracy increased by 20% Accuracy evaluation, audit reports Ledger Infrastructure, Trust Orchestrator (TrustOR) iKPI6 Integration of three security modalities in assurance Dynamic testing, runtime monitoring Security Assurance Module, Dynamic Testing Module iKPI7 Detection of 95% insecure mechanisms in pilot activities Static & dynamic testing tool performance Dynamic Testing Module iKPI8 TBOM components’ selfassessed preparedness increased by 50% Stakeholder feedback, pilot reports MVP Evaluation, Dashboard iKPI9 Dynamic vulnerability assessment achieves 98% accuracy Benchmark testing, system evaluation Dynamic Testing Module, Security Assurance Module iKPI10 Security auditing tools achieve 95% accuracy Validation through security audits Security Assurance Module, Dashboard iKPI11 RESCALE TrustOR adoption rate exceeds 80% in pilots Adoption tracking, user feedback Trust Orchestrator (TrustOR), Dashboard iKPI12 Business models proposed based on TrustOR (>2 models) Market analysis, pilot engagement Trust Orchestrator (TrustOR), MVP Evaluation iKPI13 Five AI-assisted security testing models validated AI model integration results Security Assurance Module, AI-driven Analysis iKPI14 Validation of RESCALE outside project scope in one cybersecurity supply chain External pilot validation MVP Evaluation, Trust Orchestrator (TrustOR) iKPI15 Achieve 95% cyberattack resilience during pilots Stress testing, security assessments Dynamic Testing Module iKPI16 Perception of security via TBOM and TrustOR increased by 20% Stakeholder perception studies Dashboard, Trust Orchestrator (TrustOR) iKPI17 Reduce time for vulnerability detection by 10% Detection speed benchmarking Static Code Analysis, Dynamic Testing Module iKPI18 At least one emerging technology trend embedded in RESCALE roadmap Innovation tracking, technology reports High-Level Architecture (D2.5), Technology Analysis (D2.3) RESCALE – Public – Page 102 / 148 D5.5: Validation and Evaluation Report (first version) iKPI19 Impact demonstrated on three secure EU-based products/services Exploitation assessment MVP Evaluation, Dashboard iKPI20 RESCALE TBOM solutions demonstrated in two supply chain domains Adoption case studies Trust Orchestrator (TrustOR), Ledger Infrastructure iKPI22 More than three early adopters of the RESCALE platform Stakeholder adoption tracking MVP Evaluation, Dashboard iKPI26 Security testing mechanisms achieve 95% accuracy Validation testing, traceability analysis Static Code Analysis, Security Assurance Module iKPI27 80% acceptance of piloted technologies in two environments Pilot acceptance surveys MVP Evaluation, Dashboard 4.3 MVP Evaluation Results This section presents end-to-end testing scenarios performed to validate the integrated components of the RESCALE ecosystem. These tests aim to verify: • The interoperability of security assurance mechanisms, • The detection of vulnerabilities through dynamic and static analysis, • The usability and effectiveness of the RESCALE dashboard for tracking security metrics, • The integration of blockchain mechanisms for supply chain transparency. 4.3.1 Test Scenario 1: Supply Chain Security Validation and TBOM Generation Objective: Assess the security of a supply chain scenario by utilising RESCALE’s security assurance framework. Components Integrated: • Trust Orchestrator (TrustOR), • Static and Dynamic Testing Modules, • RESCALE Dashboard, • State Manager, • Security Assurance Module, RESCALE – Public – Page 103 / 148 D5.5: Validation and Evaluation Report (first version) • Blockchain, Infrastructure for TBOM Storage. Test Flow: 1. Component Registration: The producer logs in to the Rescale Platform and registers a software or hardware component from the RESCALE dashboard as shown in Figure 43. 2. Static Analysis Execution: The Static Code Analysis Module runs the static analysis tools for the component under test. The results of each tool are combined into an SSCG and SBOM output, which are forwarded to the RESCALE Platform. See Figure 44. 3. Dynamic Testing Execution: The Dynamic Testing Module performs runtime security validation and forwards the generated DSCG output to the RESCALE Platform. See Figure 45. 4. TBOM Creation: When the SSCG and DSCG are generated for a component, the TrustOR initiates the TBOM generation procedure, where a TBOM entry is created and added into the blockchain. 5. Security Assurance Monitoring: Afterwards, the Security Assurance Module continuously monitors the TBOM’s security status. See figure 46 6. Dashboard Reporting: All results are aggregated and displayed in the RESCALE Dashboard for end-user review. Expected Outcome: • Identification of all known vulnerabilities in the supply chain using RESCALE’s AIpowered static and dynamic analysis. • Secure storage of the TBOM via blockchain. • Continuous monitoring and reporting of TBOM security statuses. 4.3.2 Test Scenario 2: TBOM Validation Workflow Objective: Validate the process of generating and securing a TBOM within the RESCALE framework. Components Integrated: • State Manager • TrustOR • Security Assurance Module • Blockchain Infrastructure • RESCALE Repositories RESCALE – Public – Page 104 / 148 D5.5: Validation and Evaluation Report (first version) Test Flow: 1. Login to RESCALE Dashboard: Authenticate using valid credentials to access the RESCALE Platform. 2. Locate the Component associated with the TBOM under validation: Use the search or filter options in the Dashboard to find the target component. 3. Validate TBOM Integrity: Use blockchain metadata shown on the Dashboard to verify that the TBOM hash matches what is stored on the blockchain. 4. From the Component View, inspect Component’s Reports for further insights about the vulnerabilities detected: These include vulnerability reports included in the SSCG, SBOM, DSCG, and updates from the Bill of Vulnerabilities through continuous monitoring. Expected Outcome: • Successful response from Rescale Platform if TBOM is found in the blockchain. • Continuous monitoring for validation of software and hardware components. • Automated compliance checks within the security assurance framework. 4.3.3 System Integration Sequence Diagram A sequence diagram is included to illustrate the full end-to-end testing flow, showcasing: • User authentication and role-based access control (IAM) (see Figure 43), • Component registration and security validation steps (see Figure 43), • How static and dynamic security validation feed into TBOM generation (see Figures 44 and 45 ), • Blockchain integration and verification of TBOM authenticity, • Final security audit and dashboard reporting. These scenarios are derived from the RESCALE High-Level Architecture (D2.5) and the established test flows. They demonstrate the seamless integration of security assurance, blockchainbased transparency, and real-time vulnerability monitoring within the RESCALE ecosystem. Further test scenarios and additional integration validation will be described in future iterations of this document. RESCALE – Public – Page 105 / 148 2 No experience What is the size of your company?1.4 Small (1–50 employees) Medium (51–200 employees) Large (201–1000 employees) Very Large (1001+ employees) I’m not sure / Prefer not to say In which country is your company based?1.5 2 Security Assessment and Vulnerability Detection How long does it currently take to detect vulnerabilities in your software or hardware supply chain?2.1 iKPI-17 Less than a day 1–3 days 4–7 days More than a week Which methods or tools does your organization use for evidence-based assessments of security and 2.2 privacy properties of software and hardware components? iKPI-6 Static Code Analysis Dynamic Analysis Formal Verification Risk Assessment Frameworks Security Audits and Penetration Testing Threat Modeling Integration of AI/ML Models for Security and Privacy Detection Manual Code Reviews and Security Reviews Other By using RESCALE, to what extent do you believe the time required to detect vulnerabilities in your 2.4 supply chain be improved? iKPI-17 No improvement Slight improvement (1–5%) Moderate improvement (6–10%) Significant improvement (11–20%) Very significant improvement (more than 20%) I cannot estimate * * * * * DRAFT VERSION 3 To what extent do you believe that using the RESCALE platform will improve your organization's self-2.5 assessed preparedness levels for cybersecurity threats and incidents? iKPI-8 No improvement expected Some minor improvements, but overall impact is limited. Noticeable improvements, but some gaps remain. Major improvements in cybersecurity preparedness. Fully enhances preparedness and significantly reduces risks. To what extent do you believe RESCALE improves system resilience against cyberattacks?2.6 KPI1.1 No improvement Slight improvement Moderate improvement Significant improvement Very significant improvement I cannot estimate Has RESCALE helped identify vulnerabilities that were previously unknown in your system?2.7 KPI2.3 Yes No Not applicable Did using RESCALE improve your team's understanding of cybersecurity vulnerabilities?2.8 iKPI25 No improvement Slight improvement Moderate improvement Significant improvement Very significant improvement I cannot estimate 3 Compliance with Security Standards What Security Standards and proven guidelines has your organization adopted?3.1 General Data Protection Regulation (GDPR) I do not know / It does not concern ISO/IEC 15408-1:2022 (Evaluation criteria for IT security) ISO/IEC 27001:2013 (Information Security Management System) ISO/IEC 27002:2022 (Code of Practice for Information Security Controls) ISO/IEC 27005:2022 (Guidance on managing information security risks) ISO/IEC 27034 (Application Security) ISO/IEC 27035 (Information security incident management) NIS 2 Directive (EU Directive on Security of Network and Information Systems) NIST SP 800-30 (Guide for Conducting Risk Assessments) * * * * * DRAFT VERSION 4 NIST SP800-61 (Computer Security Incident Handling Guide) To what extent do you believe the RESCALE platform can assist your organization in adopting and 3.2 aligning with cybersecurity standards, such as the NIS Directive, GDPR, and other security frameworks? iKPI-4 Not at all To a small extent To a moderate extent To a large extent Completely 4 Adoption and Acceptance of RESCALE Solution How well do you think the RESCALE platform provides informative resources for both security and non-4.1 security experts? KPI4.4 Very poorly Poorly Neutral Well Very well What improvements would you suggest to make the information provided more accessible for both 4.2 security and non-security experts? KPI4.4 In your opinion, how beneficial is the use of blockchain technology in managing TBOM for security of 4.3 software supply chains? iKPI-1,iKPI-16: Not beneficial at all Slightly beneficial Moderately beneficial Very beneficial Extremely beneficial To what extent can introduction of blockchain-assisted TBOM updates increas your trust in the security 4.4 of software supply chain? iKPI-1,iKPI-16 Not at all Slightly increased trust Moderately increased trust Significantly increased trust Fully increased trust * * * * DRAFT VERSION 5 What are the main factors that would encourage or discourage your organization from fully adopting 4.5 blockchain-assisted TBOM management processes? iKPI-1 To what extent do you agree with the following statement: The technologies proposed and piloted 4.6 through the RESCALE project are effective and adaptable across different environments. iKPI-27 Strongly disagree Disagree Neutral Agree Strongly agree By what percentage do you estimate that the adoption of RESCALE can lead to cost savings in your 4.7 company? iKPI-24 Less than 5% 5-10% 10-15% 15-20% More than 20% I cannot estimate How likely are you to adopt the RESCALE platform in your organization?4.8 KPI4.2,iKPI-11 Very unlikely Unlikely Neutral Likely Very likely Contact [email protected] * * * DRAFT VERSION 6 DRAFT VERSION D5.5: Validation and Evaluation Report (first version) B SAVE-ME Example { "$schema": "https://json.schemastore.org/sarif-2.1.0", "version": "2.1.0", "runs": [ { "tool": { "driver": { "name": "Erlang Analyzer", "fullName": "Erlang Vulnerability Analyzer", "version": "1.0.0", "informationUri": "https://sample_site.com", "rules": [ { "id": "vulnerability-detection", "name": "Vulnerability Detection", "fullDescription": { "text": "Detects vulnerabilities in Erlang functions" } } ] } }, "results": [ { "ruleId": "vulnerability-detection", "message": { "text": "No vulnerability detected with confidence 3.59. Confidence for potential vulnerability: -3.28." }, "locations": [ { "physicalLocation": { "artifactLocation": { "uri": "file:///Users/Shared/Files From d.localized/Materijali /_RESCALE/save-me/test_dir/sample_erlang_module_n.erl" }, "region": { "startLine": 19, "endLine": 19 } } } ], "properties": { "confidence": 3.586867332458496 } }, { RESCALE – Public – Page 117 / 148 D5.5: Validation and Evaluation Report (first version) "ruleId": "vulnerability-detection", "message": { "text": "No vulnerability detected with confidence 3.59. Confidence for potential vulnerability: -3.30." }, "locations": [ { "physicalLocation": { "artifactLocation": { "uri": "file:///Users/Shared/Files From d.localized/Materijali /_RESCALE/save-me/test_dir/sample_erlang_module_n.erl" }, "region": { "startLine": 26, "endLine": 26 } } } ], "properties": { "confidence": 3.591543436050415 } } ] } ] } RESCALE – Public – Page 118 / 148 D5.5: Validation and Evaluation Report (first version) C DSCG Example { "bomFormat": "CycloneDX", "specVersion": "1.6", "serialNumber": "urn:uuid:267cdf51-3dee-4757-9ed5-a9ba005eaa4d", "version": 1, "metadata": { "timestamp": "2025-03-22T06:24:38.534788Z", "authors": [ { "name": "Some Company/person that wants to generate a DSCG for their binary", "email": "[email protected]" } ], "tools": { "components": [ { "type": "application", "name": "ReSCALE DSCG Generator", "version": "1.0.0", "description": "A ReSCALE certified tool to generate DSCGs", "purl": "pkg:hex/[email protected]" } ] } }, "declarations": { "targets": { "components": [ { "bom-ref": "ref_component_grisp", "description": "GRiSP Erlang Runtime Library", "externalReferences": [ { "type": "vcs", "url": "https://github.com/grisp/grisp" } ], "licenses": [ { "license": { "id": "Apache-2.0" } } ], "name": "grisp", "purl": "pkg:hex/[email protected]", "type": "application", "version": "2.6.0" RESCALE – Public – Page 119 / 148 D5.5: Validation and Evaluation Report (first version) } ] }, "claims": [ { "bom-ref": "Claim: RAISE found 6 instances of 500 errors!", "evidence": [ "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts/189", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts/186", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts/184", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts/184", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts/189", "Evidence: 500 - Internal Server Error encountered at /api/ blog/posts/189" ] }, { "bom-ref": "urn:uuid:bc2d4166-497f-42d0-86fc-3164f9cd1093", "evidence": [ "Evidence: Faulty endpoints" ] } ], "evidence": [ RESCALE – Public – Page 120 / 148 D5.5: Validation and Evaluation Report (first version) { "bom-ref": "Evidence: 500 - Internal Server Error encountered at /api/blog/posts/189", "description": "Evidence gathered by RAISE for 500", "data": [ { "name": "RAISE Gadget Output", "contents": { "attachment": { "contentType": "application/json", "encoding": "base64", "content": " ewogICJyZXF1ZXN0IjogewogICAgIlJlcXVlc3REYXR ..." } } } ] }, { "bom-ref": "Evidence: 500 - Internal Server Error encountered at /api/blog/posts", "description": "Evidence gathered by RAISE for 500", "data": [ { "name": "RAISE Gadget Output", "contents": { "attachment": { "contentType": "application/json", "encoding": "base64", "content": " ewogICJyZXF1ZXN0IjogewogICAgIlJlcXVlc3REYXR ..." } } } ] }, { "bom-ref": "Evidence: 500 - Internal Server Error encountered at /api/blog/posts", "description": "Evidence gathered by RAISE for 500", "data": [ { "name": "RAISE Gadget Output", "contents": { "attachment": { "contentType": "application/json", "encoding": "base64", "content": " ewogICJyZXF1ZXN0IjogewogICAgIlJlcXVlc3REYXR RESCALE – Public – Page 121 / 148 D5.5: Validation and Evaluation Report (first version) { "alg": "SHA-1", "content": "eee3fa7f88a4a3496903865e15b8fbf8065c1bd3" } ], "type": "bom", "url": "urn:cdx:bc2d4166-497f-42d0-86fc-3164f9cd1093/1" } ] } RESCALE – Public – Page 128 / 148 D5.5: Validation and Evaluation Report (first version) D TBOM file example { "bomFormat": "CycloneDX", "specVersion": "1.5", "serialNumber": "urn:uuid:d96c6a6c-5710-49f9-9335-04c5c33f06fd", "version": 1, "metadata": { "component": { "bom-ref": "tbom1-metadata", "name": "tbom1/metadata", "type": "file" }, "supplier": { "name": "rescale-project.eu" }, "timestamp": "2025-03-28T15:20:04.001499+00:00", "tools": { "components": [ { "name": "TBOM Generator", "type": "application" }, { "description": "Python library for CycloneDX", "externalReferences": [ { "type": "build-system", "url": "https://github.com/CycloneDX/cyclonedx-python-lib/actions" }, { "type": "distribution", "url": "https://pypi.org/project/cyclonedx-python-lib/" }, { "type": "documentation", "url": "https://cyclonedx-python-library.readthedocs.io/" }, { "type": "issue-tracker", "url": "https://github.com/CycloneDX/cyclonedx-python-lib/issues" }, { "type": "license", "url": "https://github.com/CycloneDX/cyclonedx-python-lib/blob/ main/LICENSE" }, { "type": "release-notes", "url": "https://github.com/CycloneDX/cyclonedx-python-lib/blob/ main/CHANGELOG.md" RESCALE – Public – Page 129 / 148 D5.5: Validation and Evaluation Report (first version) }, { "type": "vcs", "url": "https://github.com/CycloneDX/cyclonedx-python-lib" }, { "type": "website", "url": "https://github.com/CycloneDX/cyclonedx-python-lib/#readme" } ], "group": "CycloneDX", "licenses": [ { "license": { "id": "Apache-2.0" } } ], "name": "cyclonedx-python-lib", "type": "library", "version": "9.1.0" } ] } }, "components": [ { "bom-ref": "bov-ref", "description": "a link to continuously-updated Bill of Vulnerabilities of : grisp", "externalReferences": [ { "comment": "Link to BOV", "type": "distribution", "url": "https://sm.dev.rescale-project.eu/bovGet/urn:uuid:2c25b307fb70-486f-9a8e-3eee4c3ec046" } ], "name": "BOV", "purl": "pkg:hex/[email protected]", "type": "file" }, { "bom-ref": "dscg-ref", "description": "a link to the DSCG of: grisp", "externalReferences": [ { "comment": "DSCG URL", "type": "distribution", "url": "https://sm.dev.rescale-project.eu/dscgGet/urn:uuid:3e671687 -391b-41f5-a30f-a58921a69b78" RESCALE – Public – Page 130 / 148 D5.5: Validation and Evaluation Report (first version) } ], "name": "DSCG", "purl": "pkg:hex/[email protected]", "type": "file" }, { "bom-ref": "sbom-ref", "description": "a link to the SBOM of: grisp", "externalReferences": [ { "comment": "SBOM URL", "type": "distribution", "url": "https://sm.dev.rescale-project.eu/sbomGet/urn:uuid:7b0ba9e7d293-4d05-a923-9527728610b9" } ], "name": "SBOM", "purl": "pkg:hex/[email protected]", "type": "file", "version": "1" }, { "bom-ref": "sscg-ref", "description": "a link to the SSCG of: grisp", "externalReferences": [ { "comment": "SSCG URL", "type": "distribution", "url": "https://sm.dev.rescale-project.eu/sscgGet/urn:uuid:bc2d4166 -497f-42d0-86fc-3134f9cd1073" } ], "name": "SSCG", "purl": "pkg:hex/[email protected]", "type": "file" } ], "services": null, "externalReferences": null, "dependencies": [ { "ref": "bov-ref" }, { "ref": "dscg-ref" }, { "ref": "sbom-ref" }, { RESCALE – Public – Page 131 / 148 D5.5: Validation and Evaluation Report (first version) "ref": "sscg-ref" }, { "dependsOn": [ "bov-ref", "dscg-ref", "sbom-ref", "sscg-ref" ], "ref": "tbom1-metadata" } ] } RESCALE – Public – Page 132 / 148 D5.5: Validation and Evaluation Report (first version) E SASTER-cli Example { "$schema": "https://json.schemastore.org/sarif-2.1.0.json", "version": "2.1.0", "runs": [ { "tool": { "driver": { "name": "Bandit", "organization": "PyCQA", "rules": [ { "id": "B404", "name": "blacklist", "properties": { "tags": [ "security", "external/cwe/cwe-78" ], "precision": "high" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/blacklists/ blacklist_imports.html#b404-import-subprocess" }, { "id": "B602", "name": "subprocess_popen_with_shell_equals_true", "properties": { "tags": [ "security", "external/cwe/cwe-78" ], "precision": "high" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/plugins/ b602_subprocess_popen_with_shell_equals_true.html" }, { "id": "B608", "name": "hardcoded_sql_expressions", "properties": { "tags": [ "security", "external/cwe/cwe-89" ], "precision": "low" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/plugins/ b608_hardcoded_sql_expressions.html" }, RESCALE – Public – Page 133 / 148 D5.5: Validation and Evaluation Report (first version) { "id": "B201", "name": "flask_debug_true", "properties": { "tags": [ "security", "external/cwe/cwe-94" ], "precision": "medium" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/plugins/ b201_flask_debug_true.html" }, { "id": "B403", "name": "blacklist", "properties": { "tags": [ "security", "external/cwe/cwe-502" ], "precision": "high" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/blacklists/ blacklist_imports.html#b403-import-pickle" }, { "id": "B324", "name": "hashlib", "properties": { "tags": [ "security", "external/cwe/cwe-327" ], "precision": "high" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/plugins/ b324_hashlib.html" }, { "id": "B105", "name": "hardcoded_password_string", "properties": { "tags": [ "security", "external/cwe/cwe-259" ], "precision": "medium" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/plugins/ b105_hardcoded_password_string.html" RESCALE – Public – Page 134 / 148 D5.5: Validation and Evaluation Report (first version) }, { "id": "B301", "name": "blacklist", "properties": { "tags": [ "security", "external/cwe/cwe-502" ], "precision": "high" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/blacklists/ blacklist_calls.html#b301-pickle" }, { "id": "B113", "name": "request_without_timeout", "properties": { "tags": [ "security", "external/cwe/cwe-400" ], "precision": "low" }, "helpUri": "https://bandit.readthedocs.io/en/1.7.10/plugins/ b113_request_without_timeout.html" } ], "version": "1.7.10", "semanticVersion": "1.7.10" } }, "invocations": [ { "executionSuccessful": true, "endTimeUtc": "2024-10-01T08:35:40Z", "toolConfigurationNotifications": [ { "message": { "text": "syntax error while parsing AST from file" }, "level": "error", "locations": [ { "physicalLocation": { "artifactLocation": { "uri": "file:///data/uploads/user_1/project_1/files/ python_example_1.py" } } } RESCALE – Public – Page 135 / 148 D5.5: Validation and Evaluation Report (first version) ] } ] } ], "properties": { "metrics": { "_totals": { "loc": 139, "nosec": 0, "skipped_tests": 0, "SEVERITY.UNDEFINED": 0, "CONFIDENCE.UNDEFINED": 0, "SEVERITY.LOW": 4, "CONFIDENCE.LOW": 2, "SEVERITY.MEDIUM": 3, "CONFIDENCE.MEDIUM": 2, "SEVERITY.HIGH": 4, "CONFIDENCE.HIGH": 7 }, "/data/uploads/user_1/project_1/files/python_example_1.py": { "loc": 56, "nosec": 0, "skipped_tests": 0 }, "/data/uploads/user_1/project_1/files/python_example_2.py": { "loc": 14, "nosec": 0, "skipped_tests": 0, "SEVERITY.UNDEFINED": 0, "SEVERITY.LOW": 1, "SEVERITY.MEDIUM": 0, "SEVERITY.HIGH": 1, "CONFIDENCE.UNDEFINED": 0, "CONFIDENCE.LOW": 0, "CONFIDENCE.MEDIUM": 0, "CONFIDENCE.HIGH": 2 }, "/data/uploads/user_1/project_1/files/python_example_3.py": { "loc": 28, "nosec": 0, "skipped_tests": 0, "SEVERITY.UNDEFINED": 0, "SEVERITY.LOW": 0, "SEVERITY.MEDIUM": 1, "SEVERITY.HIGH": 1, "CONFIDENCE.UNDEFINED": 0, "CONFIDENCE.LOW": 1, "CONFIDENCE.MEDIUM": 1, "CONFIDENCE.HIGH": 0 }, RESCALE – Public – Page 136 / 148 D5.5: Validation and Evaluation Report (first version) "/data/uploads/user_1/project_1/files/python_example_4.py": { "loc": 22, "nosec": 0, "skipped_tests": 0, "SEVERITY.UNDEFINED": 0, "SEVERITY.LOW": 3, "SEVERITY.MEDIUM": 1, "SEVERITY.HIGH": 2, "CONFIDENCE.UNDEFINED": 0, "CONFIDENCE.LOW": 0, "CONFIDENCE.MEDIUM": 1, "CONFIDENCE.HIGH": 5 }, "/data/uploads/user_1/project_1/files/python_example_5.py": { "loc": 19, "nosec": 0, "skipped_tests": 0, "SEVERITY.UNDEFINED": 0, "SEVERITY.LOW": 0, "SEVERITY.MEDIUM": 1, "SEVERITY.HIGH": 0, "CONFIDENCE.UNDEFINED": 0, "CONFIDENCE.LOW": 1, "CONFIDENCE.MEDIUM": 0, "CONFIDENCE.HIGH": 0 } } }, "results": [ { "message": { "text": "Consider possible security implications associated with the subprocess module." }, "level": "note", "locations": [ { "physicalLocation": { "region": { "snippet": { "text": "import subprocess\n" }, "endColumn": 18, "endLine": 1, "startColumn": 1, "startLine": 1 }, "artifactLocation": { "uri": "file:///data/uploads/user_1/project_1/files/ python_example_2.py" }, RESCALE – Public – Page 137 / 148 D5.5: Validation and Evaluation Report (first version) "physicalLocation": { "region": { "snippet": { "text": "SECRET_KEY = \"my_secret_key\"\n" }, "endColumn": 29, "endLine": 15, "startColumn": 14, "startLine": 15 }, "artifactLocation": { "uri": "file:///data/uploads/user_1/project_1/files/ python_example_4.py" }, "contextRegion": { "snippet": { "text": "# Vulnerability: Hardcoded secret key\nSECRET_KEY = \"my_secret_key\"\n\n" }, "endLine": 16, "startLine": 14 } } } ], "properties": { "issue_confidence": "MEDIUM", "issue_severity": "LOW" }, "ruleId": "B105", "ruleIndex": 6 }, { "message": { "text": "Pickle and modules that wrap it can be unsafe when used to deserialize untrusted data, possible security issue." }, "locations": [ { "physicalLocation": { "region": { "snippet": { "text": " return pickle.loads(data)\n" }, "endColumn": 30, "endLine": 19, "startColumn": 12, "startLine": 19 }, "artifactLocation": { RESCALE – Public – Page 144 / 148 D5.5: Validation and Evaluation Report (first version) "uri": "file:///data/uploads/user_1/project_1/files/ python_example_4.py" }, "contextRegion": { "snippet": { "text": "def insecure_deserialization(data):\n return pickle .loads(data)\n\n" }, "endLine": 20, "startLine": 18 } } } ], "properties": { "issue_confidence": "HIGH", "issue_severity": "MEDIUM" }, "ruleId": "B301", "ruleIndex": 7 }, { "message": { "text": "Call to requests without timeout" }, "locations": [ { "physicalLocation": { "region": { "snippet": { "text": " response = requests.post(\"http://insecure-server. com/api/data\", json=data)\n" }, "endColumn": 79, "endLine": 15, "startColumn": 16, "startLine": 15 }, "artifactLocation": { "uri": "file:///data/uploads/user_1/project_1/files/ python_example_5.py" }, "contextRegion": { "snippet": { "text": "def send_data(data):\n response = requests.post(\" http://insecure-server.com/api/data\", json=data)\n return response\n" }, "endLine": 16, "startLine": 14 } RESCALE – Public – Page 145 / 148 D5.5: Validation and Evaluation Report (first version) } } ], "properties": { "issue_confidence": "LOW", "issue_severity": "MEDIUM" }, "ruleId": "B113", "ruleIndex": 8 } ] } ] } RESCALE – Public – Page 146 / 148 D5.5: Validation and Evaluation Report (first version) References [1] Vasiliki (Vasia) Rentzios Atlidakis, Patrice Godefroid, and Marina Polishchuk. Restler: Stateful rest api fuzzing. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 748–758, 2019. doi:10.1109/ICSE.2019.00083. [2] Tom Christie. Starlette. https://www.starlette.io/, 2018. Accessed: 2025-03-25. [3] European Commission. NIS2 Directive: new rules on cybersecurity of network and information systems. https://digital-strategy.ec.europa.eu/en/policies/ nis2-directive. Accessed: 2025-04-06. [4] Mongo Express Contributors. Mongo express. https://github.com/mongo-express/ mongo-express, 2013. Accessed: 2025-03-25. [5] EUSurvey. Rescale usability survey, 2025. Accessed: 2025-03-12. URL: https://ec. europa.eu/eusurvey/runner/RESCALEUsabilitySurvey. [6] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, and et al. Jiang, Daxin. Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155, 2020. [7] Ethereum Foundation. Ethereum: Blockchain app platform. https://ethereum.org/, 2015. Accessed: 2025-03-25. [8] Hyperledger Foundation. Hyperledger besu. https://www.hyperledger.org/use/ besu, 2019. Accessed: 2025-03-25. [9] GitHub. Github advisory database · github. https://github.com/advisories. Accessed: 2025-03-31. [10] ´ Akos Hajdu, Matteo Marescotti, Thibault Suzanne, Ke Mao, Radu Grigore, Per Gustafsson, and Dino Distefano. Inferl: scalable and extensible erlang static analysis. In Proceedings of the 21st ACM SIGPLAN International Workshop on Erlang, Erlang 2022, page 33–39, New York, NY, USA, 2022. Association for Computing Machinery. doi:10.1145/3546186.3549929. [11] Kali Linux Project. Binwalk: Firmware Analysis Tool. https://www.kali.org/ tools/binwalk/, 2025. Accessed: 2025-04-03. [12] Let’s Encrypt. Let’s encrypt - free ssl/tls certificates. https://letsencrypt.org/, 2015. Accessed: 2025-03-25. [13] Inc. MongoDB. Mongodb: The developer data platform. https://www.mongodb.com/, 2009. Accessed: 2025-03-25. [14] NGINX, Inc. Nginx web server. https://www.nginx.com/, 2004. Accessed: 2025-0325. [15] OWASP Foundation. Cyclonedx bill of materials standard, 2024. Accessed: 2025-03-25. URL: https://cyclonedx.org/. RESCALE – Public – Page 147 / 148 D5.5: Validation and Evaluation Report (first version) [16] QEMU Project. QEMU: Open Source Processor Emulator. https://www.qemu.org/, 2025. Accessed: 2025-04-03. [17] Sebasti´ an Ram´ ırez. Fastapi. https://fastapi.tiangolo.com/, 2018. Accessed: 2025-03-25. [18] RESCALE Project. RESCALE Harbor Repository. https://harbor. rescale-project.eu/harbor/projects/3/repositories/save-me, 2025. [19] RESCALE Project. RESCALE Harbor Repository SASTER-cli Image. https://harbor.rescale-project.eu/harbor/projects/3/repositories/ saster-cli_image, 2025. [20] Sonatype. Sonatype oss index. https://ossindex.sonatype.org/. Accessed: 202503-31. RESCALE – Public – Page 148 / 148