D 7.3 Impact Measurement Matrix and Index

Deep-tech driven inclusive and socially-aware circular material joining & welding for a safe future Project Number: 101184218 Topic: ERASMUS-EDU-2024-PI-ALL-INNO-EDU-ENTERP Type of Action: ERASMUS-LS 7.3. Impact Assessment and Index Weldify Deliverables Template This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or European Education And Culture Executive Agency (EACEA). Neither the European Union nor the EACEA can be held responsible for them. Deliverable: Deliverable 7.3 - Impact measurement matrix and index Work Package: WP7 Due Date: 28.02.2025 Submission Date: 20.02.2025 Start Date of Project: 01.12.2025 Duration of Project: 36 months Organisation Responsible for Deliverable: EWF Version: 1.0 Status: Final Author name(s): Begum Canaslan Akyar Reviewer(s): Adelaide Almeida Nature: R – Report Dissemination level: PU – Public Document history Version Date Modified by (author/partner) Comments First version 01.02.2025 Begum Canaslan Akyar Final Version 25.02.2025 Begum Canaslan Akyar 2 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template Table of Contents 1. Abbreviations and Terminology 4 2. Executive Summary 5 3. Introduction 6 3.1. Objectives and goals.................................................................................................................................................6 3.2. Methodology....................................................................................................................................................................8 4. Impact Assessment Tool 8 4.1. State of the Art (SoA).................................................................................................................................................9 4.2. Dimensions....................................................................................................................................................................... 9 4.3. Indicators...........................................................................................................................................................................11 4.3.1. Definition of dimensions and indicators................................................................................................13 4.4. Dimensions and Indicators prioritization exercise...............................................................................15 5. Conclusion. 1 6. References 2 7. Annexes 1 7.1. Annex A: The first draft of the impact assessment matrix................................................................1 3 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template 1. Abbreviations and Terminology Abbreviations State of arts SoA Predictive Maintenance Education & Training System PreMETS The Millennium Development Goals MDGs The Organization for Economic Cooperation and Development OECD Key performance indicator KPI Terminology The Impact Measurement Matrix and Index The tool serves as a fundamental tool for tracking and optimizing performance, allowing stakeholders to make informed decisions based on reliable data. 4 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template 2. Executive Summary This document presents the Impact Measurement Matrix and Index for the Weldify project as part of Task 7.3. This report is a living document that will be iteratively reviewed and updated throughout the project’s lifecycle. The primary objective of this deliverable is to establish a structured framework of indicators to assess the impact of Weldify solutions over the medium and long term across multiple dimensions. The impact assessment results will be continuously monitored and adjusted to provide insights into project performance and outcomes. The document consists of two main sections. The first section introduces the significance of an impact assessment matrix, outlines its objectives, and describes the methodology used for identifying key impact dimensions and relevant indicators. The second section presents the state-of-the-art analysis, the selection and prioritization of impact dimensions and indicators, and the methodology for continuous evaluation and refinement. 5 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template 3. Introduction The Impact Measurement Matrix and Index is designed to be a dynamic framework that evolves alongside the Weldify project. As the project progresses, it is essential to establish a structured approach for evaluating its impact, ensuring that its contributions align with the intended objectives. This assessment serves as a fundamental tool for tracking and optimizing performance, allowing stakeholders to make informed decisions based on reliable data. The Weldify project focuses on advancing welding technologies and methodologies, making it crucial to define specific impact dimensions and performance indicators. These indicators will help monitor project progress across multiple areas, such as technological advancements, economic benefits, environmental sustainability, and social contributions. By systematically collecting and analysing data, the project team can assess how effectively Weldify is meeting its goals and identify areas for improvement. A well-structured impact measurement framework ensures that project outcomes are not only tracked but also optimized for long-term benefits. This document outlines the methodology used to define impact dimensions, establish relevant indicators, and develop a structured monitoring approach. By leveraging best practices from similar initiatives and incorporating stakeholder feedback, the Weldify impact assessment framework will provide valuable insights that contribute to the project’s success and wider industry adoption. 3.1. Objectives and goals The goal of an Impact measurement matrix and index is to provide a system to measure and monitor different indicators related to the project's exploitation performance, its achievements, and accountability. The assessment begins by redefining the scope and objectives of the developed solution, comprising its targeted group and the desired impact. Establishing clear objectives is important to achieve the main goal set. The objectives can be identified as follows: ● Identify and develop indicators: Establish dimensions and indicators to measure the effectiveness of the Weldify solution. ● Collect data: In order to evaluate the impact of the Weldify project, data must be continuously collected. Two types of data—qualitative and quantitative—will be gathered throughout the project to accurately interpret the impact of the final Weldify solution. The methodology should incorporate a combination of these assessment measures. 6 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template Quantitative data, such as numerical values and statistical analysis, will help identify trends across a large pool of Weldify adopters, offering measurable insights into project performance. Qualitative data, including surveys and interviews, will provide deeper insights into individual user experiences, helping to assess the acceptability and effectiveness of the Weldify solution. By combining both types of data, the assessment will offer a more comprehensive understanding of the solution’s impact across identified dimensions. To ensure effective data collection, it is essential to engage both primary stakeholders—direct users of the Weldify solution—and secondary stakeholders who interact with the solution indirectly. Soliciting feedback from these groups will help identify potential challenges, allowing the project team to implement targeted strategies and mitigation measures to overcome barriers efficiently. A comprehensive impact assessment requires the inclusion of diverse target groups to capture a holistic perspective on the project's effectiveness. Additionally, all data collection processes will be conducted in compliance with data protection regulations, ensuring that the privacy and security of Weldify users are safeguarded at all times. ● Evaluate changes and monitor impact: Analyzing the collected data will help identify and examine trends in key indicators, providing valuable insights into project effectiveness. This evaluation involves comparing baseline data with subsequent measurements to track progress over time. By continuously monitoring the impact, potential deviations can be detected early, allowing corrective actions to be implemented promptly. The evaluation process includes periodic reviews and data-driven analysis to ensure that Weldify solutions are meeting predefined objectives. Additionally, this step helps in assessing the sustainability and scalability of the project by identifying areas for improvement and optimizing resource allocation. Regular stakeholder feedback and engagement will further enhance the understanding of project impact, ensuring that the assessment framework remains relevant and responsive to emerging challenges. ● Adapt and refine the solution: Based on the feedback received and analysed from the evaluation of the collected data, any necessary modifications to the Weldify solution should be made and adopt countermeasures to reach a certain indicator. Adjusting implementation and exploitation strategies can maximize the solution’s impact. In the following section, the methodology adopted to create Weldify’s Impact Measurement Matrix and Indicators will be described in detail. 7 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template 3.2. Methodology While both the impact measurement tool and quality assessment are important tools for evaluating projects and programs, the two differ in their scope, purpose, focus, timeframe, and the stakeholders involved in the measurements of their impact. The impact measurement tool will be looking at the broader, long-term consequences of Weldify, whereas the quality assessment assesses the adherence to predefined quality standards and processes throughout the project's lifecycle from the quality indicators perspective. The data collection methods for impact measurement and quality assessment can overlap in some areas, but also retain significant differences based on their distinct purposes and focus. In developing the impact measurement framework for Weldify, we leveraged the methodology established in the PreMETS project, adapting it to the specific needs of Weldify so the methodology was directly built upon PreMETS’ structured approach. The impact measurement dimensions, and corresponding indicators were defined based on the project’s application, ensuring alignment with Weldify’s objectives. These dimensions reflect key areas where the project aims to create measurable impact, including social contributions, economic and governmental benefits. The selected indicators provide a structured means of evaluating the project's progress and effectiveness over time. To refine the impact measurement tool, a collaborative validation process was implemented. The initial draft of the tool was presented during the Weldify kick-off meeting, allowing consortium partners to review its structure and relevance. Following this presentation, the tool was circulated among the partnership for evaluation. Partners were asked to assess each item in terms of its relevance and provide feedback on how to improve the tool. Their input was systematically integrated, ensuring that the final framework is both comprehensive and reflective of Weldify partners’ insights. By utilizing this adapted methodology, the impact measurement framework for Weldify remains robust, ensuring that its dimensions and indicators effectively capture the project’s overall impact while maintaining a participatory approach to evaluation and refinement. 4. Impact Assessment Tool The impact assessment tool is a living document that will serve as the basis to start measuring and monitoring defined indicators that will be reiterated and extended based on the project’s evolution and proposed Weldify solution. 8 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template 4.1. State of the Art (SoA) The Weldify project builds on the state-of-the-art research conducted in the PreMETS project. PreMETS provided a comprehensive framework for impact assessment by identifying critical dimensions and indicators relevant to evaluating technological solutions. The key findings from PreMETS’ state-of-the-art analysis were instrumental in shaping Weldify’s impact measurement framework. By leveraging PreMETS’ findings, Weldify ensured that its impact assessment methodology is both robust and adaptable. Rather than duplicating a new state-of-the-art analysis, Weldify streamlined the existing PreMETS framework to suit the project’s specific needs. This approach allows for continuity in assessment methods while ensuring that Weldify benefits from proven best practices in impact evaluation. 4.2. Dimensions Social, economic, environmental, and cultural dimensions are often described as necessary elements to be included when conducting an impact assessment. These dimensions are crucial because they help assess the broader and more holistic effects of a project, program, or policy on various aspects of society and the environment. The Millennium Development Goals (MDGs) and the Research and Innovation (R&I) Horizon Frameworks should be mentioned as they are pivotal guides for shaping the objectives, methodologies, and impact assessments of research initiatives. The Millennium Development Goals (MDGs) established in 2000 by world leaders at the United Nations are probably the most famous and exponential example of an outlined shared vision for a better future, which included reducing poverty, hunger, and disease, improving maternal and child health, providing education for all, advancing gender equality, protecting the environment, and fostering global partnerships. The MDGs consisted of eight specific goals, that are similar from the concept of the Dimensions used in an impact assessment, were each supported by 60 indicators, aimed to address critical global challenges and measure and monitor the progress made towards those goals. MDGs, are pivotal in framing global development agendas and continue to provide invaluable insights into addressing pressing societal challenges. While this example is highly known, the concept of dimensions and indicators is adaptable to many different domains, depending on the need for what it needs to be assessed and monitored to advance towards the achievement of predefined targets. 9 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Table 2: final version of impact measurement matrix Dimensions Themes Nº Indicator Base Value Target Value Measureme nt Unit Measurement Method Region al Impact * Duration ** Target group* ** External Stakehold ers **** SOCIETAL Considers the impact that WELDIFY has on its learners, as well as other users considered as potential targets of interest Learning experien ce 1 CMW modules developed and integrated into university curricula 0 5 Number of modules Complete learning pack developed 2 CMW modules developed and integrated into VET curricula 0 5 Number of modules Complete learning pack developed 3 Number of train-the-trainers modules 0 5 Number of modules Complete learning pack developed 4 Number of trained learners on CMW 0 5000 Number of registrations Record/log keeping 5 Number of learners involved in blended work-based learning secondments 0 250 Number of certifications Record/log keeping 6 Degree of learners feeling empowered to be in charge of their own learning path 0 100 Number of registrations Record/log keeping Weldify Deliverables Template 7 Number of trained trainers in CMW 0 30% increase Subjective/p ersonal reflections Pre and post assessment Accessib ility 8 Number of learners from social groups at risk involved in the project 0 500 Number of certifications Record/log keeping 9 Make online education more inclusive 0 45% increase Subjective/p ersonal reflections Pre and post assessment Accepta bility 10 Number of companies engaged in the WELDIFY platform 0 100 Number of registrations Record/log keeping 11 Number of universities and VET centres that will uptake the WELDIFY platform 0 50 Number of institutions Record/log keeping 12 Number of modernised university and VET related CMW curricula 0 25 Number of institutions Record/log keeping Experien ce satisfacti on 13 Learner satisfaction with regards to the WELDIFY online platform 0 >85% satisfacti on 1-5 scale index Perceptions survey Local & Regional 14 Number of social media 0 100 000 Number of people's reactions Record/log keeping 2 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template Engage ment engagement/impressi ons 15 Number of website visitors 0 5000 Number of website views Record/log keeping 16 Number of people accessed from partners' countries 0 900 Number of open days participants Record/log keeping 17 Number of contacts 0 1.021.000 Number of contacts reached via online disseminatio n materials Record/log keeping ECONOMIC Considers whether industries adopting Weldify change their employability, income, operating costs, productivity and competitiven ess over time. Upskilling and employa bility 18 Learner digital competence increase 0 35% increase 1-5 point scale for DigComp Self-assessed DigComp survey 19 Learner green competence increase 0 30% increase 1-5 point scale for GreenComp Self-assessed GreenComp survey 20 Learner entrepreneurship competence increase 0 35% increase 1-5 point scale for EntreComp Self-assessed EntreComp survey (pre and post training) 21 Trainer competence increase with regards to online education delivery 0 25% increase 1-5 point scale for DigCompEd u Self-assessed DigComp survey 3 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template 22 Number of dropouts from university/VET 0 25% decrease Number of learners Comparing drop-outs in year 1 and in year 3 learners database 23 Make CMW workforce more productive (quant) 0 20% increase Productivity metrics Checking company reports 24 Learner digital competence increase (sample of 1000 learners) (quant) after preasessm ent 35% increase 1-5 point scale for DigComp Self-assessed DigComp survey (pre and post training) 25 Increase of employability Statistic al averag e 15-30% increase in 6-12 months Percentage Comparing yearly statistics GOVERNANCE Considers whether institutions were able to use Weldify to partially overcome potential differences in social capital, through their good governance Regional funds and investme nts in PM 26 Amount of funding obtained to sustain WELDIFY activities 0 500.000 Euro Combined budget from new projects Policy Making 27 Number of policy makers interested in using WELDIFY platform 0 20 Number of policy makers Record/log keeping 4 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template as well as cross-sectori al cooperation and partnerships. *Reginal Impact: Local/Regional, National, European **Duration: Short-term, Medium-term, Long-term ***Target group: Academic learners, VET learners, Current employees in CMW, Unemployed/redundant/social groups at risk, VET Trainers, VET training centres, Instructional designers and education providers ****External stakeholders: Policy makers, Industrial clusters/associations/chambers, Learner/educational associations/university unions, Skill monitoring/development associations:, EU's Digital Transformation Monitor, Associations and accelerators for EduTech innovations, Educational technology providers, EU's Micro-credential policy group, None 5 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 5. Conclusion. The development of the Impact Measurement Matrix for the Weldify project has established a comprehensive framework to evaluate the project's effectiveness and long-term impact. By leveraging insights from the PreMETS project and refining its methodology to align with Weldify’s specific goals, we have created a robust assessment tool that enables continuous monitoring and iterative improvements. The structured evaluation process has ensured that relevant dimensions, such as societal, economic, and governance impacts, are effectively measured. Through the involvement of consortium partners, a well-balanced set of indicators was identified, refined, and validated, contributing to a collaborative and transparent impact assessment approach. While the current version of the impact measurement matrix provides a solid foundation for evaluating Weldify’s success, it remains a living document. As the project evolves, new insights, stakeholder feedback, and industry developments may necessitate further refinements. Continuous engagement with consortium partners and project stakeholders will be crucial in ensuring that the impact measurement process remains relevant and aligned with the project’s overarching objectives. Moving forward, the next steps include ongoing data collection, periodic evaluations, and adaptive refinements based on real-world project outcomes. By maintaining an iterative and flexible approach, the Weldify project will continue to enhance its impact assessment framework, ensuring that its contributions to welding technology innovation and education are effectively measured and optimized for maximum societal and industrial benefit. Weldify Deliverables Template 6. References [1] Bührer, S.; Feidenheimer, A.; Walz, R.; Lindner, R.; Beckert, B.; Wallwaey, E. (2022): Concepts and methods to measure societal impacts – an overview (Discussion Papers Innovation Systems and Policy Analysis No. 74): Karlsruhe: Fraunhofer ISI. [2] European Commission, Directorate-General for Research and Innovation, Nixon, J., Study to support the monitoring and evaluation of the framework programme for research and innovation along key impact pathways – Indicator methodology and metadata handbook, Nixon, J.(editor), Publications Office of the European Union, 2022, https://data.europa.eu/doi/10.2777/44653 [3] Gouëdard P. (2021). Developing indicators to support the implementation of education policies (OECD Education Working Papers, No. 255). OECD Publishing. https://www.oecd-ilibrary.org/content/paper/b9f04dd0-en [4] Ilham, Z., Subramaniam, I., Jamaludin, A. A., Wan, W. A. A. Q. I., Halim-Lim, S. A., Ohgaki, H., Ishihara, K., Mansor, M. R. A. (2022). Analysing dimensions and indicators to design energy education framework in Malaysia using the analytic hierarchy process (AHP). Energy Reports, 8, 1013-1024. https://www.sciencedirect.com/science/article/pii/S2352484722013968 [5] SoPHIA D2.3. 2021. Deliverable D2.3 Final version of the SoPHIA Impact Assessment Model. https://sophiaplatform.eu/uploads/sophiaplatformeu/2021/11/04/aaccb1ae0b65909e6440d54 5882820c9.pdf (accessed 18 08 2023) [6] UNAIDS, H. D. (2010). An Introduction to Indicators. UNAIDS Monitoring and Evaluation Fundamentals. Geneva: UNAIDS. 2 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 7. Annexes 7.1. Annex A: The first draft of the impact assessment matrix Dimensions Themes Indicator Base Value Targe t Value Measureme nt Unit Measureme nt Method Measure ment method Regional Impact Durati on Targ et grou p External Stakehold ers Partner Organis ation Relevanc e for the project Shall we keep the item Comment s SOCIETAL Considers the impact that WELDIFY has on its learners, as well as other users considered as potential targets of interest Learning experienc e CMW modules develope d and integrate d into university curricula 0 5 Number of modules Complete learning pack developed Complet e learning pack develop ed CMW modules develope d and integrate d into VET curricula 0 5 Number of modules Complete learning pack developed Complet e learning pack develop ed Weldify Deliverables Template Number of train-the -trainers modules 0 5 Number of modules Complete learning pack developed Complet e learning pack develop ed Number of trained learners on CMW 0 5000 Number of registration s Record/log keeping Record/l og keeping Number of trained trainers in CMW 0 250 Number of certification s Record/log keeping Record/l og keeping Number of learners involved in blended work-bas 0 100 Number of registration s Record/log keeping Record/l og keeping 2 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218 Weldify Deliverables Template ed learning secondm ents Degree of learners feeling empower ed to be in charge of their own learning path 30% incre ase Subje ctive/ perso nal reflec tions Pre and post assessment Pre and post assessm ent (before and after the trainings ) Accesibilit y Number of learners from social groups at risk involved 0 500 Number of certification s Record/log keeping Record/l og keeping 3 This project has received funding from the European Union's Erasmus+ Programme under Grant Agreement No. 101184218