Safe and secure use of AI in research projects

Safe and secure use of AI in research projects Dr. Renat Shigapov About me •Background: Physics and Geophysics •2016-2019: Dr. rer. nat. at Karlsruhe Institute of Technology •2019-2021: Data scientist in BERD@BW (UB Mannheim) •2022-2024: Data scientist in BERD@NFDI (UB Mannheim) •2023-2025: Data science consultant in the research data center (UB Mannheim) •2024-2025: Lead data scientist in the research data center (UB Mannheim) •2025-now: Lead data scientist and project manager (UB Mannheim) •Expertise: data science, research data management, and project management. LinkedIn: https://www.linkedin.com/in/renat-shigapov GitHub: https://github.com/shigapov Agenda •09:00-09:30 INTRODUCTION •09:30-12:00 PART 1: Theory •Freedom of research under ethical, integrity, and governance (legal and regulatory) constraints •AI risks, AI safety, and AI security •Risk management •12:00-13:00 LUNCH •13:00-15:30 PART 2: Practice •Risk management framework adapted to the trinity of risks (ethical, integrity, and governance) •Hands-On: Use cases of AI and risk management across the research lifecycle. Plan & design, collect & create, analyze & collaborate, evaluate & archive, share & disseminate, access & reuse. •AI policies and checklists for research groups and research projects •15:30-16:00 SUMMARY and feedback Introduction This rectangle will separate different parts of this workshop What is AI? What is an AI system? •There exist many definitions. None of them are perfect. But when we talk about AI, we often mean AI systems. •An AI system is a special type of information technology (IT) system consisting of hardware, software, data (database, training and validation datasets), AI model(s), and networks, which can generate output data based on input data it receives and data it was trained on. Hardware Software Data Networks input Networks output AI model(s) Local AI system VS cloud AI system •Local AI system runs fully on a device you control, meaning you also have more control over its safety and security. •Cloud AI system runs on remote infrastructure managed by a provider. Using a cloud AI system means trusting a complex supply chain of hardware, software, data, models, and networks that you do not control. https://www.nvidia.com/en-us/products/workstations/dgx-spark Example (not product placement, really): üNVIDIA DGX SparkTM costs less than 4500€ üSupports models with up to 200b parameters What is an agentic AI system? •An Agentic AI system is an AI system that can set or interpret goals, plan actions, and carry out tasks with minimal human intervention or autonomously. To achieve these goals, it may interact with other IT systems via networks or execute code and communicate with software components locally on an IT system. •Agentic AI may use multiple models, including a router model that decides which model or tool to use next. Hardware Software Data Networks input Networks output AI router + AI model(s) What are AI model, model weights, and Open Source AI? From the Open Source AI Definition (https://opensource.org/ai/open-source-ai-definition) •“An AI model consists of the model architecture, model parameters (including weights) and inference code for running the model.” •“AI weights are the set of learned parameters that overlay the model architecture to produce an output from a given input.” •An Open Source AI is one made freely available with all necessary code (for data preprocessing; training, validation, and testing; inference; supporting libraries and tools), data (datasets, data card, technical report, and research paper) and model (architecture and parameters) under legal terms approved by the Open Source Initiative. Generative AI system Generative AI system is a special type of AI system, which was trained to generate content (text, image, audio, video, code, etc.). Here could be a beautiful figure generated by an AI system and illustrating what generative AI system is. But we can safe around 0.01-0.1 Wh of energy and 0.1-1 g of CO₂ and you imagine that beautiful figure yourself. It might remind you of But the stakes today are much higher... https://superintelligence-statement.org Safe and secure AI system: There are no zerorisk AI systems •The only fully safe and fully secure AI system is the one that is never used. •But modern research and everyday life rely increasingly on AI systems. •Today, we will identify AI risks and learn how to manage them. Safe and secure AI system: Reminder on complexity of the topic •Safety and security of AI system include safety and security of its components: hardware, software, data (including training, validation, testing, augmented, input, output, and database), models, and networks. This makes the topic very complex. •Safe and secure use of AI in research (projects) is shaped by ethical, integrity, and governance (legal and regulatory) requirements and expectations. They are complex. Hardware Software Data Networks input Networks output AI model(s) AI Safety and AI Security Lin, Zhiqiang, Huan Sun, and Ness Shroff. "AI Safety vs. AI Security: Demystifying the Distinction and Boundaries.” 2025, arXiv preprint, https://doi.org/10.48550/arXiv.2506.18932 AI Safety AI Security risks from accidental or unintended behaviors risks from intentional or adversarial actions by malicious actors Safety and security in context of research: The trinity of good research Research Ethics Research Integrity Research Governance Safety and Security Safety and Security Safety and Security Risk management Risk Management Framework: 1. Identifying Risks 2. Evaluating Risks by Estimating Likelihood and Impact 3. Deciding What to Do (Risk Treatment): Risk Mitigation, Transfer, Avoidance, or Acceptance. 4. Risk Communication and Monitoring We cannot treat risks unless we can identify them. We identify risks within research ethics, research integrity, and research governance. The general risks in AI safety and AI security will be also introduced. In Part 2 we do risk management across the entire research lifecycle. https://owaspai.org/goto/riskanalysis/ Defense in depth Defense in depth is a risk-mitigation approach that relies on multiple, overlapping, and complementary layers of safeguards, so that if one control fails, others remain effective. Key assumption: any single mitigation measure can fail. Dimensions of defense in research (projects) for an AI use case: •People: individual researchers, research teams, project partners, and participants •Infrastructure/Technology: hardware, software, data, models, networks, and the entire supply chain •Processes: research process including research ethics, research integrity, and research governance Risk management framework adapted to the trinity of risks in research (projects) Adapted from the Risk management framework: https://owaspai.org/goto/riskanalysis/ •Identifying Risks: Recognizing potential risks related to research ethics, research integrity, and research governance. •Evaluating Risks: Subjectively: low, medium, high. •Risk Treatment: Choosing an appropriate strategy to address the risk: Risk Mitigation, Transfer (to another party),Avoidance (eliminating the source),or Acceptance. •Risk Communication and Monitoring: Regularly sharing risk information with stakeholders. Creating a Risk Register, a list of risks and their attributes (e.g. severity, treatment plan, ownership, status, etc). Note: Any mitigation measure can fail. Apply defense in depth to all three dimensions (people, technology, and processes) in Risk Mitigation. Questions Before you use any AI system, ask yourself the following questions: •Is the hardware safe and secure? •Is the software safe and secure? •Is the data (including training, validation, testing, augmented, input, and output) safe and secure? •Is the model safe and secure? •Are the networks safe and secure? Cloud AI: Choose a compliant provider (check certifications and trust portals) + use case Local AI (self-built): You are responsible for safety, security, and use case Local AI with third-party open-weight models: Shared responsibility (see licenses) The trinity of good research Research integrity •Focus: character of researchers (good research practices of researchers) •Responsibility: researchers Research ethics •Focus: judgment on the ethical acceptability of research •Responsibility: research ethics committees with inputs from the public and research community Research governance •Focus: legal and policy requirements •Responsibility: research support officers with the skills and experience to address technical compliance Kolstoe, S. E., & Pugh, J. (2023). The trinity of good research: Distinguishing between research integrity, ethics, and governance. Accountability in Research, 31(8), 1222–1241. https://doi.org/10.1080/08989621.2023.2239712 To o m a ny va l u e s, p r i n c i p le s, l aws, re g u l a ti o n s . . . Researchers face variety of (partially contradicting) values, principles, codes of conduct, guidelines of good practices, ethics frameworks, laws, regulations, institutional policies, funder requirements, contractual obligations, disciplinary norms, open science mandates, data protection rules, security requirements, journal data and AI policies. It feels impossible to navigate. Pragmatic approach: Remember responsibilities •To conduct responsible high-quality research, researchers need supporting infrastructure. •Research organizations are responsible for providing infrastructure, legal certainty, ethical oversight, data protection, and secure environments. •Researchers are supported by ethics committees, ombudspersons, research data management teams, legal & data protection offices, IT security teams, Open Science offices, export control offices, etc. üUse these support structures early, not when problems appear. Pragmatic approach: Remember responsibilities Integrity How should we behave? Responsibility, honesty, rigor, transparency, FAIR & Open Science, following good research practices, avoiding questionable research practices & scientific misconduct. ACT responsibly. Ethics Should we do this project? Values, principles, social responsibility, impact on humans, animals, society, and nature. GET ethical approval from research ethics committees for your research project. Governance What must we comply with? Laws (GDPR, copyright, EU AI Act), policies, funding rules, contracts, licenses, and agreements. GET help from research support units for your research project. Note: expect to hear “on a caseby-base basis” and that’s fine. Note: Independently of ethical approval, your own ethical selfassessment of your research project can significantly improve its quality Pragmatic approach: Remember the goal Ethics, Integrity, and Governance are not constraints but foundations for excellent research. If we agree on that, compliance becomes a quality tool, not paperwork. Freedom of research Research Ethics Research ethics: Who defines that? •International Ethical Frameworks define the values and principles of ethics (e.g., UNESCO ethical frameworks) •Disciplinary Norms and Communities define what is ethically sensitive or ethically problematic in their own context. These norms guide ethical evaluation (e.g., the Declaration of Helsinki by the World Medical Association) •Ethics Committees or Institutional Review Boards check what is ethically permissible in concrete research projects •Funders may require ethical (self-)assessment and review (e.g., EU grants from three EU programs such as Horizon Europe, Digital Europe, and European Defence Fund) Main research ethics documents for using AI in research projects •UNESCO Recommendation on the Ethics of Artificial Intelligence (https://unesdoc.unesco.org/ark:/48223/pf0000381137) •OECD Principles on Artificial Intelligence (https://oecd.ai/en/ai-principles) •EU High-Level Expert Group on AI –Ethics Guidelines for Trustworthy AI (https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai) •EU Grants: How to complete your ethics self-assessment https://ec.europa.eu/info/funding-tenders/opportunities/docs/20212027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf Research ethics in Germany https://www.konsortswd.de/en/topics/best-practices-research-ethics UNESCO recommendation on the ethics of AI •A human rights approach to AI •4 core values •Respect, protection and promotion of human rights and fundamental freedoms and human dignity •Environment and ecosystem flourishing •Ensuring diversity and inclusiveness •Living in peaceful, just and interconnected societies •10 principles UNESCO. Recommendation on the ethics of artificial intelligence. United Nations Educational, Scientific and Cultural Organization, 2022. https://www.unesco.org/en/artificialintelligence/recommendation-ethics EU grants: The developed/used AI solutions must •ensure that people are aware they are interacting with an AI system and are informed (in a language and terms understandable by all) about its abilities, limitations, risks and benefits. The manner in which this is done must be described in the proposal. ⚠The manner in which information is provided should not depend on particular educational backgrounds, technical knowledge, or other skills which cannot be assumed of all people. •prevent possible limitations on human rights and freedoms (e.g. freedom of expression, access to information, freedom of movement etc.) •not be designed in a way that may lead to objectification, dehumanization, subordination, discrimination, stereotyping, coercion, manipulation of people or creation of attachment or addiction https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf EU grants: The developed/used AI solutions must •be able to demonstrate compliance with the principles of data minimization and privacy by design and by default when processing personal data. The principles of lawfulness, transparency and fairness of the data processing must be respected at all times. For more information, please consult the “Guidance on ethics and data protection in research projects” (https://ec.europa.eu/info/fundingtenders/opportunities/docs/2021-2027/horizon/guidance/ethics-and-dataprotection_he_en.pdf) •be designed in a way to avoid bias in both input data and algorithm design. The systems should be able to prevent potential discrimination, stigmatisation or any other adverse effects on the individual related to the use of the developed/deployed AI system/technique. The manner in which this is done must be described in your project proposal https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf EU grants: The developed/used AI solutions must •address the potential impact on the individual, society or the environment. An evaluation of the potential negative individual, societal and/or environmental impacts must be carried out and be included in the project proposal along with the measures to be set in place to mitigate any potential adverse effect. ⚠The ethics risk assessment and risk mitigation measures must cover the development, deployment and post-deployment phases. •not reduce the safety and wellbeing of the individuals. Whenever relevant, the safety of the developed/used systems must be demonstrated in the project proposal. •be developed in a way that enables human oversight (human-in-the-loop, human-onthe-loop, human-in-command), traceability and auditability. Whenever possible, explanation on how decisions are taken by the developed/used AI along with the logic behind it should be provided to the users. https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf Human oversight •A human-in-the-loop system requires active human involvement in decision-making during operation. The AI system cannot proceed or finalize decisions without human input, validation, or approval. •A human-on-the-loop system operates autonomously by default, but a human supervises, monitors, and can intervene or override decisions if necessary. •A human-in-command approach ensures that humans retain ultimate authority, responsibility, and accountability over the AI system, including its deployment, goals, and use, even if no human intervenes during operation. EU grants: Ethics issues checklist https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf EU grants: Ethics issues checklist https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf Safety Safety EU grants: Ethics issues checklist https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf Safety Security EU grants: Ethics issues checklist https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf Safety and security (dual use) EU grants: Ethics issues checklist https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf EU grants: Ethics issues checklist https://ec.europa.eu/info/funding-tenders/opportunities/docs/2021-2027/common/guidance/how-to-complete-your-ethics-self-assessment_en.pdf ⚠In case it is not possible to identify the potential risks related to the AI system/techniques at this stage, describe the procedure you intend to use to detect, assess and address potential ethics issues (or explain why such a procedure is not needed). Freedom of research Research Integrity Research integrity: Who defines that? •International, national, funders, and university Codes of Conduct for Research Integrity (e.g., ALLEA European Code of Conduct, Singapore Statement on Research Integrity, DFG’s "Guidelines for Safeguarding Good Research Practice”) •Research communities define domain-specific integrity standards •Local integrity offices, local ombudspersons, and Ombuds Committee for Research Integrity in Germany are points of contact for researchers to report and seek advice on issues related to research integrity •Publishers and journals create policies related to research integrity (data policy, AI policy, general guidelines for authors, etc.) •Committee on Publication Ethics issues guidelines for authors and reviewers Main research integrity documents for using AI in research General documents which do not specifically mention „AI“: •ALLEA European Code of Conduct for Research Integrity (https://allea.org/code-of-conduct) •Singapore Statement on Research Integrity (https://www.wcrif.org/statement) •DFG. (2025). Guidelines for Safeguarding Good Research Practice. Code of Conduct. https://doi.org/10.5281/zenodo.14281892 AI-specific documents: •Living guidelines on the responsible use of generative AI in research for researchers, research organizations, and research funding organizations (https://research-andinnovation.ec.europa.eu/document/download/2b6cf7e5-36ac-41cb-aab5-0d32050143dc_en) •AI policies and recommendations of research organizations (e.g., Helmholtz: https://www.helmholtz.de/assets/helmholtz_gemeinschaft/Downloads/Helmholtz_Recommendat ions_on_use_of_AI_Version_1.0.pdf) •AI policies of funders, journals, and publishers RI https://ukrio.org/wp-content/uploads/WhatisRI.png Graphical representation of research integrity based on the core areas described in The Concordat to Support Research Integrity 2019, created by the United Kingdom Research Integrity Office (UKRIO). Research misconduct in European Code of Conduct Research misconduct is traditionally defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results: •Fabrication is making up data or results and recording them as if they were real. •Falsification is manipulating research materials, equipment, images, or processes, or changing, omitting, or suppressing data or results without justification. •Plagiarism is using other people’s work or ideas without giving proper credit to the original source. https://allea.org/wp-content/uploads/2023/06/European-Code-of-Conduct-Revised-Edition-2023.pdf DFG and research misconduct: Scope §1 (2): These Rules of Procedure apply if the respondent is one of the following with regard to the allegation: 1. A grant applicant to the DFG 2. A grant recipient funded by the DFG, 3. Individuals with a high level of scientific responsibility in connection with funding proposals submitted by higher education institutions or non-university research institutions, 4. individuals reviewing a proposal for the DFG or 5. A member of a DFG committee or a committee supported by the DFG in administering funding instruments who participates in advisory, review, evaluation or decision-making procedures. DFG, “Rules of Procedure for Dealing with Scientific Misconduct”, https://www.dfg.de/resource/blob/339200/dfg-80-01-v0524-en.pdf DFG and research misconduct: All definitions §2: Scientific Misconduct (1) 1An individual pursuant to §1 (2) nos. 1-3 commits scientific misconduct if they do any of the following in particular, either intentionally or with gross negligence: 1. make misrepresentations (§ 3), 2. appropriate others’ research achievements without justification (§ 4), 3. interfere with others’ research (§ 5), 4. participate in the scientific misconduct of others by way of co-authorship (§ 6) or 5. neglect their supervisory duties (§ 7). 2Anyone who intentionally participates in the misconduct of others is also guilty of scientific misconduct (§8). (2) A person pursuant to §1 (2) nos. 4 and 5 commits scientific misconduct if they do any of the following, either intentionally or with gross negligence: 1. breach confidentiality (§ 9), 2. fail to disclose circumstances that give rise to the appearance of conflict of interest (§10) or 3. inadmissibly give unfair preferential treatment to others (§11). DFG, “Rules of Procedure for Dealing with Scientific Misconduct”, https://www.dfg.de/resource/blob/339200/dfg-80-01-v0524-en.pdf Reviewer Applicant or recipient DFG and research misconduct §3 Misrepresentations A misrepresentation is: 1. a fabrication of research-related data or results, 2. a falsification of research-related data or results, in particular by a) suppressing or eliminating data or results obtained in the course of the research process without disclosing this suppression or elimination, b) undertaking an undisclosed modification of a representation or illustration, 3. presenting an image and a statement corresponding to it in an incongruous manner, 4. making inaccurate statements in a grant proposal or within the scope of the reporting obligation, 5. claiming another person’s authorship or co-authorship without that person’s consent, unless consent was refused without sufficient research-related reason DFG, “Rules of Procedure for Dealing with Scientific Misconduct”, https://www.dfg.de/resource/blob/339200/dfg-80-01-v0524-en.pdf DFG and research misconduct: Appropriation §4 Inadmissible appropriation of others’ research achievements A person is deemed to have inadmissibly appropriated others’ research achievements if they: 1. use research content of which they are not the sole originator without sufficient reference to the source (plagiarism), 2. use research approaches, results or ideas of which they are not the sole originator without authorisation for their own purposes or for the purposes of third parties (theft of ideas), 3. disclose research data, approaches, ideas, hypotheses or theories to third parties without authorisation, unless the subject of the disclosure has already been duly published, 4. appropriate or accept authorship or co-authorship of an academic publication without justification, even though they did not make any genuine, identifiable contribution to the research content of the publication, 5. publish research data, approaches, ideas, hypotheses or theories without authorisation, if the subject matter has not yet been published by the authorised persons or with the latter’s consent. DFG, “Rules of Procedure for Dealing with Scientific Misconduct”, https://www.dfg.de/resource/blob/339200/dfg-80-01-v0524-en.pdf DFG and research misconduct: Co-author §6 Co-authorship 1A person pursuant to §1 (2) nos. 1-3 commits scientific misconduct if they are the co-author of a publication that contains misrepresentations within the meaning of § 3 or unjustifiably appropriated research achievements within the meaning of § 4. 2The misconduct of third parties pursuant to §§ 3 and 4 must have been intentional or grossly negligent, as must the person’s own coauthorship. DFG, “Rules of Procedure for Dealing with Scientific Misconduct”, https://www.dfg.de/resource/blob/339200/dfg-80-01-v0524-en.pdf Question: If a researcher commits research misconduct in a publication using AI, what are consequences for other co-authors? To o l: Res ea rc h I nt e gr ity R i s k I n d ex https://sites.aub.edu.lb/lmeho/ri2 Meho, L. I. (2025). Gaming the Metrics: Bibliometric Anomalies in Global University Rankings and the Research Integrity Risk Index (RI²). Scientometrics, 130, 6683– 6726. https://doi.org/10.1007/s11192-025-05480-2 To o l: Res ea rc h I nt e gr ity R i s k I n d ex https://sites.aub.edu.lb/lmeho/ri2 Living guidelines on the responsible use of generative AI in research for researchers 1. Remain ultimately responsible for scientific output. 2. Use generative AI transparently. 3. Pay particular attention to issues related to privacy, confidentiality and intellectual property rights when sharing sensitive or protected information with AI tools. 4. Respect applicable national, EU and international legislation. 5. Continuously learn how to use generative AI tools properly to maximise their benefits, including by undertaking training. 6. Refrain from using generative AI tools substantially in sensitive activities that could impact other researchers or organisations (for example peer review, evaluation of research proposals, etc). https://research-and-innovation.ec.europa.eu/document/download/2b6cf7e5-36ac-41cb-aab5-0d32050143dc_en Living guidelines on the responsible use of generative AI in research for research organizations The guidelines for research organizations include: “3. Reference or integrate these generative AI guidelines into their general research guidelines for good research practices and ethics. •Using these guidelines as a basis for discussion, research organisations openly consult their research staff and stakeholders on the use of generative AI and related policies. •Research organisations apply these guidelines whenever possible. If needed, they could be complemented with specific additional recommendations and/or exceptions that should be published for transparency.” https://research-and-innovation.ec.europa.eu/document/download/2b6cf7e5-36ac-41cb-aab5-0d32050143dc_en German FAQ on AI and research integrity •DE: Frisch, K. (2025). FAQ Künstliche Intelligenz und gute wissenschaftliche Praxis - Version 2. Zenodo. https://doi.org/10.5281/zenodo.17349995 •EN: Frisch, Katrin (2025). FAQ Artificial Intelligence and Research Integrity. Version 2. Zenodo. https://doi.org/10.5281/zenodo.17349995 •More resources by Dr. Katrin Frisch on research data and AI in context of research integrity are available at https://ombudsgremium.de/9806/research-data-andai/?lang=en https://ombudsgremium.de/13211/faq-kuenstliche-intelligenz-und-gute-wissenschaftliche-praxis AI policies of the major publishers and publishing associations Frisch, K. (2025). FAQ Künstliche Intelligenz und gute wissenschaftliche Praxis -Version 2. Zenodo. https://doi.org/10.5281/zenodo.17349995 Appropriate and transparent declaration of using AI The policies differ, but they share a common core: •name of AI application, including version, date of use, URL •what the AI application was used for and how it was used Tools: •Artificial Intelligence Disclosure (AID) – Statement Builder (https://aidframework.org) •AI Attribution Toolkit (https://aiattribution.github.io/interpret-attribution) Frisch, K. (2025). FAQ Künstliche Intelligenz und gute wissenschaftliche Praxis -Version 2. Zenodo. https://doi.org/10.5281/zenodo.17349995 To o l: A rt i fi c i a l I nt e l l i g e n c e D i s c lo s u re ( A I D ) https://aidframework.org To ol : A I Att r i bu ti on To ol k i t https://aiattribution.github.io/interpret-attribution To ol : GAIDeT Declaration Generator https://panbibliotekar.github.io/gaidet-declaration/index.htm https://doi.org/10.1080/08989621.2025.2544331l Can I use AI to generate images? From Frisch, K. (2025). FAQ Artificial Intelligence and Research Integrity: •“Concerning AI-generated images, journals often have very restrictive policies” •“The publishing house Frontiers is one of the few exceptions as of now, that explicitly allows the use of AI-generated images, provided that authors disclose its use” •“No guidelines yet exist for the use of AI-generated images in other research output, such as presentations and posters. Researchers should discuss the issue with their project group or peers.” •“AI-generated images must not be used to feign genuine research data or results.” •“AI-generated images that only serve illustrative purposes for use on presentation slides should be in line with research integrity. For graphics that visualise research processes (like diagrams or flow charts) the use of AI could be permissible, potentially requiring disclosure.” •“… it is the responsibility of the researchers involved to check the results for accuracy.” Can I use AI for peer review? From Frisch, K. (2025). FAQ Artificial Intelligence and Research Integrity: •“In existing editorial policies, the use of AI in peer review is either subject to severe restrictions or not permitted at all.” •“For reasons of confidentiality and data protection, uploading a submitted manuscript (or grant proposal) into a generative AI application is generally not allowed.” •“If editorial policies allow for a limited use of AI in peer review, it only applies to language post-processing (i.e. improving readability). Reviewers should check which requirements apply to them.” Comment: As we saw earlier in DFG “Rules of Procedure for Dealing with Scientific Misconduct”, breach of confidentiality by a reviewer is scientific misconduct. What should be taken into consideration when using AI in authorship teams? From Frisch, K. (2025). FAQ Artificial Intelligence and Research Integrity: •“it should be determined at the start of each project whether all authors agree on the use of AI tools and the extent of their use.” •“This is especially relevant in trans-and interdisciplinary teams, since there is a greater likelihood of conflicting views on aspects such as text production” •“Te am s c a n b en efi t f ro m g o od i nt er n a l d o cu m enta ti o n on t he u se of A I. In case of conflicts or breaches of research integrity, good documentation allows others to trace the work process and genesis of the research manuscript.” •“Authors publishing in teams may also consider following the suggestions of Hosseini et al. 2023 to additionally document the member of the team who made use of AI.” Hosseini, M., Resnik, D. B., & Holmes, K. (2023). The ethics of disclosing the use of artificial intelligence tools in writing scholarly manuscripts. Research Ethics, 19(4), 449-465. https://doi.org/10.1177/17470161231180449 Can I accidentally plagiarise other texts by using AI? •It is possible. •“In general, AI-generated text should only be used after extensive editing.” (Frisch, K. (2025). FAQ Artificial Intelligence and Research Integrity) https://doi.org/10.48550/arXiv.2505.12546 I am a reviewer/editor and I suspect a text or parts thereof has/have been AI-generated, but the authors have not disclosed it. What am I supposed to do? From Frisch, K. (2025). FAQ Artificial Intelligence and Research Integrity: •In case of strong markers: “Editorial teams should discuss how breaches of their AI policy should be handled. Some editorial policies have set out a basic procedure what to do in these cases” •In case of less tangible signs: “Editors and reviewers need to keep in mind that in such cases neither the presence of the aforementioned potential indicator words nor any detection tool can reliably determine the use of AI.” •Attention: “With regard to the DFG Code of Conduct, editors and reviewers should avoid making unfounded accusations that authors committed a breach against research integrity. Guideline 18 specifies that ‘[k]nowingly false or malicious allegations may themselves constitute misconduct‘.” To o l: D ec is io n t r e e f o r re sp ons ib le a p pl ica t i on of A I American Association for the Advancement of Science (AAAS). Decision Tree for the Responsible Application of Artificial Intelligence (v1.0): [Online]. (2023). https://www.aaas.org/sites/default/files/2023-08/AAAS%20Decision%20Tree.pdf Quiz Who do you believe is responsible for research integrity when AI systems are used? 1. Users of AI systems 2. Supervisors, PIs, and project managers 3. Co-authors 4. Project partners 5. Institutions and support teams 6. Developers and providers of AI systems 7. All of the above Freedom of research Research Governance Research governance: Who defines that? •Lawmakers •National: German Bundestag and Bundesrat (e.g., BDSG, Urheberrechtsgesetz) •European: European Parliament and Council (e.g, GDPR, EU AI Act, CDSM directive) •Regulator and government agencies (e.g., European Medicines Agency, European Chemicals Agency, and Data Protection Authorities) •Courts interpret how the law applies •International Treaties and Conventions (e.g., UNESCO conventions) •Institutional Governance Structures (policies, support structures, and processes) •Contracts (grant agreements, consortium agreements, NDAs, licenses, DSAs, etc.) •Mechanisms to ensure compliance (procedures and checklists) Main research governance documents for using AI in research projects •Data protection and privacy: GDPR (General Data Protection Regulation), BDSG (Bundesdatenschutzgesetz); see also Talus, Anu. Opinion 28/2024 on Certain Data Protection Aspects Related to the Processing of Personal Data in the Context of AI Models. The European Data Protection Board, 2024. https://www.edpb.europa.eu/system/files/2024-12/edpb_opinion_202428_aimodels_en.pdf] •Copyright law: EU copyright law consists of 13 directives and 2 regulations including InfoSoc Directive, CDSM or just DSM (Copyright in the Single Market) directive, Software directive, Database directive, etc.; see also European Union Intellectual Property Office, The development of generative artificial intelligence from a copyright perspective, European Union Intellectual Property Office, 2025, https://data.europa.eu/doi/10.2814/3893780 •AI-specific: EU AI Act https://artificialintelligenceact.eu •Dual use and security-relevant research: Regulation (EU) 2021/821 (EU Dual-Use Regulation), "Manual Export Control and Academia" by BAFA and "Recommendations for Handling SecurityRelevant Research" by DFG and Leopoldina •Contract law: software, data, and model licenses, consortium agreements, grant agreements, industry collaborations, cloud service terms, NDAs, data processing agreements (DPAs), etc. GDPR: Processing of special categories of personal data Art. 9 (1): “Processing of personal data revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, or trade union membership, and the processing of genetic data, biometric data for the purpose of uniquely identifying a natural person, data concerning health or data concerning a natural person’s sex life or sexual orientation shall be prohibited.” Art. 9 (2): A selected list of exceptions relevant in research context •Consent •Protection of vital interests of a natural person •Personal data which are manifestly made public by the data subject •Public interests •For archiving purposes in the public interest, scientific or historical research purposes or statistical purposes https://gdpr-info.eu/art-9-gdpr GDPR: Safeguards and derogations relating to processing for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes Art. 89 (1): •Processing for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes, shall be subject to appropriate safeguards […] for the rights and freedoms of the data subject. •Those safeguards shall ensure that technical and organisational measures are in place in particular in order to ensure respect for the principle of data minimisation. •Those measures may include pseudonymisation provided that those purposes can be fulfilled in that manner. https://gdpr-info.eu/art-89-gdpr Question: What are technical and organizational measures? Answer: See the next slide. GDPR: Security of processing Art. 32 (1) “the controller and the processor shall implement appropriate technical and organisational measures to ensure a level of security appropriate to the risk”: •the pseudonymisation and encryption of personal data; •the ability to ensure the ongoing confidentiality, integrity, availability and resilience of processing systems and services; •the ability to restore the availability and access to personal data in a timely manner in the event of a physical or technical incident; •a process for regularly testing, assessing and evaluating the effectiveness of technical and organisational measures for ensuring the security of the processing. https://gdpr-info.eu/art-32-gdpr Note: This might involve huge work for research support teams. GDPR: More important excerpts for research •Art. 44-50: Personal data may only be transferred outside of the European Economic Area in compliance with the conditions for such transfers laid down in Chapter 5 of the GDPR. The main types of transfer tools include standard data protection clauses (SCCs), binding corporate rules (BCRs), codes of conduct, certification mechanisms, and ad hoc contractual clauses. https://gdpr-info.eu/chapter-5 •Recital 156: “The processing of personal data for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes should be subject to appropriate safeguards for the rights and freedoms of the data subject“ https://gdpr-info.eu/recitals/no-156 GPDP-compliant international transfers of personal data GDPR: Data protection impact assessment (DPIA) •Art. 35 (1): “Where a type of processing in particular using new technologies, and taking into account the nature, scope, context and purposes of the processing, is likely to result in a high risk to the rights and freedoms of natural persons, the controller shall, prior to the processing, carry out an assessment of the impact of the envisaged processing operations on the protection of personal data.” https://gdpr-info.eu/art-35-gdpr •The purpose of the DPIA is to identify, assess and mitigate any risks to the rights and freedoms of affected (natural) persons that may result from data processing (“The Data Protection Impact Assessment according to Article 35 GDPR. A Practitioner's Manual.”, https://publica-rest.fraunhofer.de/server/api/core/bitstreams/e6b91341-71f4-409b-844603432231a0d0/content) GenAI is a new technology, right? GDPR: Data protection impact assessment (DPIA) Martin, N., Friedewald, M. et al.: The Data Protection Impact Assessment according to Article 35 GDPR. A Practitioner's Manual. Stuttgart: Fraunhofer Verlag 2020. https://publica-rest.fraunhofer.de/server/api/core/bitstreams/e6b91341-71f4-409b-8446-03432231a0d0/content Question Was a data protection impact assessment ever conducted for your research project if you processed personal data using AI? GDPR: What are risks as defined in the GDPR? •Recital 75: “The risk to the rights and freedoms of natural persons, of varying likelihood and severity, may result from personal data processing which could lead to physical, material or non-material damage” •„Non-material damages may be of a social, personal, and legal nature“ (see „The Data Protection Impact Assessment according to Article 35 GDPR. A Practitioner's Manual“): •Social disadvantages •Damage to privacy •Chilling effects (e.g., a state in which persons refrain from exercising their rights) •(Unjustified) interference with rights GDPR and AI •GDPR applies to all stages of the AI lifecycle if personal data is processed (including data collection, filtering, and processing; model training, fine-tuning, augmentation, validation, and inference; inputs and outputs of an AI system; data & model archiving) Hardware Software Data Networks input Networks output AI model(s) GDPR: EDPB opinion on AI models •The European Data Protection Board (EDPB) issued opinion on AI models: „GDPR principles support responsible AI“ (https://www.edpb.europa.eu/news/news/2024/edpbopinion-ai-models-gdpr-principles-support-responsible-ai_en) •“The EDPB considers that, for an AI model to be considered anonymous, using reasonable means, both (i) the likelihood of direct (including probabilistic) extraction of personal data regarding individuals whose personal data were used to train the model; as well as (ii) the likelihood of obtaining, intentionally or not, such personal data from queries, should be insignificant for any data subject. By default, supervisory authorities should consider that AI models are likely to require a thorough evaluation of the likelihood of identification to reach a conclusion on their possible anonymous nature. This likelihood should be assessed taking into account ‘all the means reasonably likely to be used’ by the controller or another person, and should also consider unintended (re)use or disclosure of the model” (https://www.edpb.europa.eu/system/files/2024-12/edpb_opinion_202428_aimodels_en.pdf) Inference residency in ChatGPT 02.12.2025: •„Inference residency for ChatGPT is currently available for the United States. It requires data residency in the U.S.“ •„We plan to expand supported inference residency regions over time and will update this article as new regions become available.“ https://help.openai.com/en/articles/9903489-data-residency-and-inference-residency-for-chatgpt Regional data residency via OpenAI API https://platform.openai.com/docs/guides/your-data ChatGPT and GDPR-Compliance The GDPR does not mandate data localization, but it outlines strict rules and requirements for processing data outside of the EEA, including adequacy decisions, standard contractual clauses, certifications, and binding corporate rules. https://trust.openai.com/?itemUid=45220873-6e51-4dbb-b1b1-37d66ee9ef95 ChatGPT and GDPR-Compliance https://ironcladapp.com/public-launch/63ffefa2bed6885f4536d0fe Quiz Who is responsible for GDPR compliance in a research project, if researchers use AI systems with personal data? 1. The AI system provider 2. The AI system developer 3. The researcher alone 4. The research institution as controller Quiz Which AI output can be personal data under GDPR? 1. A generated image 2. A prediction about an individual 3. A summary mentioning identifiable persons 4. All of the above Quiz When is a Data Protection Impact Assessment (DPIA) required? 1. Whenever processing is likely to result in a high risk to the rights and freedoms of individuals 2. A systematic and extensive evaluation of the personal aspects of an individual, including profiling 3. Processing of sensitive data on a large scale 4. All of the above Intellectual property rights and copyright The rights-based approach Intellectual property rights and copyright •Intellectual property rights (IPR) are legal rights that protect creations of the mind such as inventions, publications, datasets, software, designs, and trademarks. •Typical IPRs are copyright (publications, presentations, datasets [exception: facts are public domain], and software), patent rights, design rights, trademarks (names and logo), database rights, and trade secrets. •In EU countries, copyright protects your intellectual property until 70 years after your death or 70 years after the death of the last surviving author in the case of a work of joint authorship. •Outside of the EU, in any country which signed the Berne Convention, the duration of copyright protection can vary but it lasts until at least 50 years after the author's death. https://eur-lex.europa.eu/EN/legal-content/summary/copyright-and-related-rights-in-the-information-society.html Directive 2001/29/EC on the harmonisation of certain aspects of copyright and related rights in the information society (InfoSoc) •Art. 2-4: Authors and neighboring rightsholders have the reproduction right, the right of communication to the public and the distribution right (see Directive 2001/29/EC InfoSoc). •Art. 5 (1): There is a mandatory exception to the right of reproduction for certain temporary acts of reproduction which are an integral and essential part of a technological process (temporary copies), and which aim to enable a lawful use or a transmission in a network between third parties by an intermediary, of a work or other subject matter. https://eur-lex.europa.eu/EN/legal-content/summary/copyright-and-related-rights-in-the-information-society.html EUIPO’s copyright perspective on AI •No ‘one-size-fits all’ solution for copyright holders to protect their rights has emerged yet. •Instead, different approaches and solutions are developing for copyright holders to protect their rights, and for AI developers to respect their regulatory obligations. •On the one side, the rights reservation mechanisms for the INPUT phase (related to training AI models), whereby rightsholders can express their opt out from the ‘text and data mining’ (TDM)-exception. •On the other side, transparency measures exist for the OUTPUT phase that allow the indication and recognition of AI generated content. European Union Intellectual Property Office, The development of generative artificial intelligence from a copyright perspective, European Union Intellectual Property Office, 2025, https://data.europa.eu/doi/10.2814/3893780 436 pages! https://www.euipo.europa.eu/en/publications/genai-from-a-copyright-perspective-2025 Case Study: LAION v. Kneschke (Hamburg district court, 27 Sept 2024, 310 O 227/23) •Background: •Photographer Robert Kneschke sued LAION (https://laion.ai), a non-profit that creates AI training datasets using web-scraped publicly available images or the Common Crawl. •His allegation: LAION reproduced his image without permission while building its dataset https://laion.ai/blog/laion-5b . •Court’s Decision: LAION’s activity (dataset creation) is protected under Section 60d UrhG (TDM for scientific research) implementing Article 3 CDSM Directive. •Note: The shared dataset contains only metadata (e.g., URLs), not the images. •Preliminary conclusion: TDM exception seems to be applicable for dataset creation. Though in this case the dataset contained only metadata. But there are also model training and model inference in AI lifecycle. Case Study: GEMA v. OpenAI (Munich district court, 11.11.2025, 42 O 14139/24) •Background: •GEMA, the collective management organisation for musical copyrights, sued OpenAI, alleging ChatGPT was trained on copyrighted German song lyrics. •The lyrics allegedly reappeared (sometimes verbatim) in ChatGPT outputs. •Court‘s decision: Copyright infringement in both training and output. The court granted injunction, disclosure, and damages. •Takeaways: •GPT-4 and GPT-4o were shown to reproduce copyrighted lyrics (“memorization”). Memorization is reproduction. •No justification of training via TDM exception. •Trainer of the models and deployer of the AI systems OpenAI was liable. •Preliminary conclusion: Using copyrighted works for AI model training (for commercial purposes) is not permitted. Is “training an AI model” equal to “TDM”? •It is complicated. •If “yes”, then Art. 3 CDSM will allow training an AI model using copyrighted works for research purposes without any opt-out of rightsholders. •If “not”, Art. 3 and Art. 4 will be irrelevant. Then, to train an AI model with copyrighted works, one needs to find other lawful ways to make it. For example, via contracts with explicit permission to use copyrighted works for training. Dornis, T. W., & Stober, S. (2025). Generative AI Training and Copyright Law. https://doi.org/10.48550/arXiv.2502.15858 “While GenAI training shares some methodological overlaps with TDM, its objectives and outputs significantly diverge. The legal and conceptual frameworks governing TDM and fair use may not seamlessly extend to generative AI, particularly given its potential to compete with and replicate the expressive elements of copyrighted works.” Dornis, Tim W., The Training of Generative AI Is Not Text and Data Mining (2024). European Intellectual Property Review (E.I.P.R.), 2/2025, http://dx.doi.org/10.2139/ssrn.4993782 Another opinion on “Is AI model training TDM?” Leistner, Matthias and Antoine, Lucie, TDM and AI training in the European Union –from "LAION" to possible ways ahead? (March 14, 2025). GRUR International, volume 74, issue 11, 2025, pp. 1027 -1044 [10.1093/grurint/ikaf114], Available at SSRN: https://ssrn.com/abstract=5178237 or http://dx.doi.org/10.1093/grurint/ikaf114 The CDSM Directive alone is not sufficient to answer this. But when it is considered in context of Art. 53 EU AI Act, then: Okay, it’s complicated, but what shall we do with model training for research purposes? The permissibility is not yet fully clear. The opinions differ. But: •If you need copyrighted works for AI model training, get contractual permissions. •If you take the risk of interpreting “AI model training” as TDM and of using the TDM exception for research purposes, make sure that: •Requirements for the TDM exception are met (e.g., lawful access to copyrighted works) •You don’t train a model with a third-party infrastructure. •You train an open-weight model at a local secure hardware. •You don’t share or archive the trained model before the legal basis for this is clarified. •There is a need for case-by-case assessment and risk balancing. This slide contains among others the adapted messages from the talk "KI im Urheberrecht: Rechtsrahmen für Bibliothek und Wissenschaft" by Dr. Marion von Francken-Welz presented at UB Mannheim in April 2025. License agreements and contracts •A blanket reference in the contract to the term "artificial intelligence" is unsuitable. One should specify concrete user acts that are to be permitted or prohibited by the contract. •Contracts cannot effectively prohibit end users from using texts or images made available from databases as input data for AI for adaptation and transformation. •Contracts concluded from 1 March 2018 onwards cannot effectively restrict the use of copyrighted works by TDM for scientific purposes, including the creation of internal scientific AI systems. Making reproductions available to third parties without contractual permission is only permitted under the conditions set out in section 60d (4) UrhG. •Contracts can effectively make provision for the use of copyrighted works for TDM for nonscientific purposes. •Contracts can provide for measures that ensure the security and integrity of the networks and databases through appropriate security precautions and set guidelines for the copies made in the context of TDM. Agi, C., Beurskens, M., von Francken-Welz, M., Ludwig, J., Mittermaier, B., & Pampel, H. (2024). Arrangements on artificial intelligence in licence agreements. Zenodo. https://doi.org/10.5281/zenodo.13837688 RAG and copyright There is no clear reference to RAG as a form of TDM in the existing agreements between AI developers and rightsholders. •Static RAG may trigger more copyright-restricted acts compared to dynamic RAG. Reasons: •Locally hosted content often necessitates a longer retention of reproductions to enable ongoing reference, a requirement that may exceed the conditions of applicability of the CDSM Directive Article 4 TDM exception, as well as the (more strict) requirements for the applicability of the InfoSoc temporary reproduction exception. •By contrast, scraping the open internet for context references in dynamic RAG typically retains content only temporarily, aligning more closely with potential for application of either TDM or temporary reproduction exceptions. European Union Intellectual Property Office, The development of generative artificial intelligence from a copyright perspective, European Union Intellectual Property Office, 2025, https://data.europa.eu/doi/10.2814/3893780 More resources on copyright •EUIPO Copyright Knowledge Centre (https://www.euipo.europa.eu/en/copyrightknowledge-centre) •Künstliche Intelligenz im Verlagsbereich: Häufig gestellte Fragen zu generativer KI https://www.boersenverein.de/beratung-service/recht/kuenstliche-intelligenz •Brehm, E. (2022). Guidelines zum Text und Data Mining für Forschungszwecke in Deutschland. https://doi.org/10.34657/9388 Quiz Is it legal to use copyrighted works to create a dataset for training an AI model? 1. Yes, always 2. No, never 3. Only under specific legal conditions 4. It is complicated Use case: AI use for grant applications (DFG) https://www.dfg.de/resource/blob/167398/10-20-en.pdf Step 1: Identifying risks in the Plan & Design Stage 1. Ethical risks •Biased framing of research questions •Ethical blind spots introduced by AI suggestions •Exclusion or misrepresentation of vulnerable groups •Over-optimistic feasibility, impact, or benefit claims 2. Integrity risks •AI fabricates references or prior work •Proposal text appears polished but shallow •Use of AI without disclosure •Funders have different expectations on misconduct 3. Governance risks •Proposal assumes illegal data access or data reuse •Missing or incorrect assessment of GDPR relevance •Ignoring copyright and licensing constraints •No awareness of dual-use or export-control risks Hands-On Exercise (20 min): Plan & Design Risk management in Plan & Design stage •Pick: one AI use case •Identify: •Multiple ethical risks •Multiple integrity risks •Multiple governance risks •Rate severity: low, medium, and high •Choose multiple treatment strategies: Risk Mitigation, Transfer (to another party), Avoidance (eliminating the source), or Acceptance. In Risk mitigation apply defense in depth to all three dimensions (people, technology, and processes). •Create: a risk register for your AI use case. Research lifecycle Collect & Create stage: •Using AI to collect data (scraping, sensors, surveys, chatbots) •Generating data (text, images, and simulations) •Annotating and labeling data with AI assistance •Filtering, cleaning, or enriching datasets •Using AI in fieldwork or lab workflows •Using AI tools for collaborative data creation across partners Use case: Springer Nature policy on genAI images https://www.nature.com/nature-portfolio/editorial-policies/ai Use case: AI generated images https://doi.org/10.3389/fcell.2023.1339390 AI “beautification” Use case: AI generated images 262 https://doi.org/10.3389/fcell.2023.1339390 Step 1: Identifying risks in the Collect & Create Stage 1. Ethical risks •Collecting personal data without consent •Use of AI to collect data from vulnerable populations without safeguards •Environmental impact by energy-intensive generation •Biased data generation reinforcing stereotypes 2. Integrity risks •GenAI-generated images and visuals are not permitted by majority of journals. •AI-fabricated data and AI-manipulated raw data. •Poor documentation of AI-assisted data collection. •Synthetic data mixed with real data without disclosure. 3. Governance risks •GDPR violations (unlawful data collection) •Copyright violations (scraping protected content) •Breach of licenses or terms of service •Cross-border data transfer violations Hands-On Exercise (20 min): Collect & Create Risk management in Collect & Create stage •Pick: one AI use case •Identify: •Multiple ethical risks •Multiple integrity risks •Multiple governance risks •Rate severity: low, medium, and high •Choose multiple treatment strategies: Risk Mitigation, Transfer (to another party), Avoidance (eliminating the source), or Acceptance. In Risk mitigation apply defense in depth to all three dimensions (people, technology, and processes). •Create: a risk register for your AI use case. Research lifecycle Analyze & Collaborate stage: •Fundamental tricks in machine learning (contamination, cherrypicking, and misreporting) •AI-assisted data analysis and statistical interpretation •AI-generated or AI-assisted code •AI support for hypothesis testing and exploratory analysis •AI tools for collaborative writing, commenting, and versioning •Agentic AI tools interacting with repositories, notebooks, or pipelines Research lifecycle Evaluate & Archive stage: •AI-written peer reviews •AI-assisted manipulation of peer review process •Drafting or summarising peer reviews •Checking manuscripts for clarity, language, or structure •AI-assisted reproducibility checks •Generating metadata and documentation for archiving •Preparing replication packages •Organising datasets, code, and models for longterm storage Use case: Prompt injection in review process Gibney, Elizabeth. "Scientists hide messages in papers to game AI peer review." Nature 643.8073 (2025): 887-888. https://doi.org/10.1038/d41586-025-02172-y Use case: AI use in peer-review DFG: General Guidelines for Reviews https://www.dfg.de/resource/blob/167398/10-20-en.pdf Question: What’s about fully local AI System? Answer: Think about ethical questions with respect to potential harms to the authors even if you use local AI system. Step 1: Identifying risks in the Evaluate & Archive Stage 1. Ethical risks •Unfair peer review due to biased or shallow AIgenerated reviews •Undisclosed AI influence on evaluation decisions •Harm to other researchers through distorted assessments 2. Integrity risks •AI-written peer reviews •AI-assisted manipulation of peer review processes •Superficial validation masking errors or misconduct •Failure to archive complete and reproducible materials 3. Governance risks •Violation of journal or funder policies on AI use •Breach of confidentiality in peer review •GDPR violations when manuscripts or reviews contain personal data •Non-compliance with archiving and data-retention requirements Hands-On Exercise (20 min): Evaluate & Archive Risk management in Evaluate & Archive stage •Pick: one AI use case •Identify: •Multiple ethical risks •Multiple integrity risks •Multiple governance risks •Rate severity: low, medium, and high •Choose multiple treatment strategies: Risk Mitigation, Transfer (to another party), Avoidance (eliminating the source), or Acceptance. In Risk mitigation apply defense in depth to all three dimensions (people, technology, and processes). •Create: a risk register for your AI use case. Research lifecycle Share & Disseminate stage: •Drafting papers and reports •Language polishing and translation •Generating posts, summaries, and outreach material Opinions from Nature and Science https://doi.org/10.1038/d41586-025-03046-z https://doi.org/10.1126/science.zxxd90o https://doi.org/10.1126/science.z87syeh Use case: Low-quality papers https://doi.org/10.1038/d41586-025-02906-y Distribution of retraction reasons Kocyigit BF, Okyay RA, Seiil B, Qumar AB, Sumbul HE. Analysis of Retracted Publications on Artificial Intelligence: Trends, Ethical Concerns, and Scientific Integrity. J Korean Med Sci. 2025 Nov 17;40(44):e280. https://doi.org/10.3346/jkms.2025.40.e280 Use case: fake citations using fake papers Ibrahim, H., Liu, F., Zaki, Y. et al. Citation manipulation through citation mills and pre-print servers. Sci Rep 15, 5480 (2025). https://doi.org/10.1038/s41598-025-88709-7 AI policies and checklists for research groups and research projects How to? Mapping the trinity of risks and existing guidelines and recommendations on AI use in research Trinity of risks EU Guidelines on the responsible use of generative AI in research Helmholtz “Recommendations for the use of artificial intelligence” Ethical risks • Privacy, confidentiality, and IP rights • Privacy and confidentiality • Bias and prejudices due to training data Integrity risks • Responsibility for scientific output • Transparent AI use • Continuous AI literacy • Sensitive activities impacting others • (Scientific) information integrity Governance risks • National, EU & international legislation • Privacy, confidentiality, and IP rights • AI-related regulation • Copyright and IP rights • Privacy and confidentiality Why research groups and projects need their own AI policies and checklists Institutional AI policies and checklists are necessary but not sufficient. Research groups and projects may need context-specific rules that reflect their data, methods, risks, and responsibilities. Reasons: •AI risks are discipline-specific and use-case-specific •In international research projects even the regulations for research ethics, research integrity, and research governance may differ Purpose of AI Policies & Checklists AI policies and checklists help research groups and research projects: •prevent ethical, integrity, and governance breaches •clarify responsibilities •ensure transparency and reproducibility •support safe and secure AI use What is AI policy for a research group or project? An AI policy is: •A living document •A risk management tool and risk register •A shared agreement within the group/project •A bridge between ethics, integrity, and governance Scope, responsibilities, and minimal compliance •Which AI systems are allowed? •For which tasks? •Who is responsible for: •AI selection •risk management •documentation •incident reporting •Minimal compliance: GDPR, Copyright, EU AI Act, export control, cybersecurity, etc. PEOPLE Checklist (Ethics & Integrity First) Researchers & Team Members ☐ Basic AI literacy for all team members ☐ Awareness of ethical risks (harms to participants, society, and the environment) ☐ Understanding of research integrity rules for AI use ☐ Clear rules for disclosure of AI use ☐ No “shadow AI” or undocumented tool usage Collaboration & Culture ☐ AI use discussed openly in the team ☐ Agreement on what counts as acceptable AI assistance ☐ Clear expectations for students, HiWis, and PhDs ☐ Special care for interdisciplinary & international projects TECHNOLOGY Checklist (Safety & Security) AI Systems & Tools ☐ Approved AI tools list (local, cloud, or hybrid) ☐ Trust portals & compliance documentation checked (cloud and hybrid) ☐ Data residency & logging behavior understood ☐ No personal or sensitive data in public AI systems Security Controls ☐ Least-privilege access to data, code, networks, and credentials ☐ No unrestricted agentic AI access to local files ☐ Versioning of models, prompts, and outputs ☐ Monitoring for data leakage and misuse Open-Weight Models ☐ Risk assessment before downloading model weights ☐ No public release without governance review ☐ Vulnerability reporting plan in place PROCESSES Checklist (Defense in Depth) Ethics (Is it acceptable?) ☐ Could AI use distort interpretation or fairness? ☐ Could vulnerable groups be affected indirectly? ☐ Are societal or environmental impacts considered? Integrity (Is it good science?) ☐ AI use documented in methods sections ☐ Original sources preserved and cited ☐ Human judgment remains central ☐ Reproducibility ensured despite AI variability Governance (Is it compliant?) ☐ DPIA completed if required ☐ Licenses and terms respected ☐ Ethics committee informed if risk profile changes ☐ Archiving and sharing rules defined Last but not least •Rewrite your group or project AI policy as “10 simple rules for using AI” in your field. https://doi.org/10.1371/journal.pcbi.1011863 https://doi.org/10.1371/journal.pcbi.1013588 Summary Feedback