Interdisciplinary Colloquium on Digitalisation of Research: "Research Automation with Agentic LLMs" 04.09.2025 Arman Khalatyan

Research Automation with Agentic LLMs Arman Khalatyan1 Session moderator: Anna Jacyszyn2 Interdisciplinary Colloquium on Digitalisation of Research, FIZ Karlsruhe, 4 September 2025 (1) Leibniz Institute for Astrophysics Potsdam (AIP) (2) FIZ Karlsruhe - Leibniz Institute for Information Infrastructure Photos and recording Pixabay, ste_phania 2 Interdisciplinary Colloquium on Digitalisation of Research, Arman Khalatyan (AIP), FIZ Karlsruhe, 4 September 2025 www.youtube.com/@DiTraRe 3 Interdisciplinary Colloquium on Digitalisation of Research, Arman Khalatyan (AIP), FIZ Karlsruhe, 4 September 2025 Research Automation with Agentic LLMs DiTraRe Interdisciplinary Colloquium on Digitalisation of Research, 04.09.2025 Dr. Arman Khalatyan Researcher at eScience/Supercomputing/IT Leibniz-Institut für Astrophysik Potsdam (AIP), Germany 1 1 Scientific life (top to down) 2 Idea Collaborate Publish Reading Prototyping Reproduce results of others Share data Plots Data access Develop software Funding Hardware Infrastructure Scientific life (top to down) 3 Idea Collaborate Publish Reading Prototyping Reproduce results of others Share data Plots Data access Develop software Funding Hardware Infrastructure Scientific life (top to down) 4 Idea Collaborate Publish Reading Prototyping Reproduce results of others Share data Plots Data access Develop software Funding Hardware Infrastructure Scientific life (top to down) 5 Idea Collaborate Publish Reading Prototyping Reproduce results of others Share data Plots Data access Develop software Funding Hardware Infrastructure Scientific life (top to down) 6 Idea Collaborate Publish Reading Prototyping Reproduce results of others Share data Plots Data access Develop software Funding Hardware Infrastructure Warning Everything you see in this presentation about tools or LLMs may already be outdated by tonight! The LLM field is moving too fast! I need your total attention "Attention is All You Need" New keywords •AGI - Artificial General Intelligence comparable to human capabilities •ASI - Artificial Superintelligence, more capable than humans •… •Transformer – state of art NLP, with attention mechanisms •LLM – Large Language Model •GPT - Generative Pre-trained Transformer, is a special LLM •ChatGPT - is a LLM model created by OpenAI A new term added since 17 Jan 2025 •Reasoning - it involves processing information, making inferences, and generating coherent responses based on learned patterns. •Chain of thoughts – reasoning process “monitoring” Transformers with attention (by Google research team 2017) Transformers with attention mechanisms are the foundational architecture for Generative Pre-trained Transformers (GPTs). These models leverage self-attention to process input data, allowing them to weigh the importance of different words in a sequence and generate coherent text. Each layer of LLM is a transformer Context length Simple ANN How the chatgpt.com works? prompt How many stars in the sky? token vector representation tokenizer The number of sta rs visi ble in the night sky var ies depending on conditions such as lig ht pollution a nd atmospheric clarity. On a clea r, dark night, without light pollution, the human eye can see between 2,500 and 5,000 stars from a single location. However, the Milky Way galaxy, which is the galaxy we reside in, contains an estimated 100 to 400 billion stars. Beyond the Milky Way, the …. generated text reinforcement training, based on user data Textual etc. training data language model neural net Iterative token generation probabilistic choices Every user prompt is landing in the company database LLM market is HUGE (2024) The evolutionary tree of modern LLMs. Source "Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond" available on arXiv LLM sizing(aging very quickly) 1-100 TRILLION? New Era of … or or New paper from OpenAI team on Mathematical problems! Let’s Verify Step by Step trained a model to achieve a new stateof-the-art in mathematical problem solving by rewarding each correct step of reasoning (“process supervision”) instead of simply rewarding the correct final answer (“outcome supervision”). In addition to boosting performance relative to outcome supervision, process supervision also has an important alignment benefit: it directly trains the model to produce a chain-of-thought that is endorsed by humans https://openai.com/research/improving-mathematicalreasoning-with-process-supervision Strawberry is out from OpenAI:o1 “Thinking” Time quality 200$/m PhD level researcher?? O1 quite hyped One prompt based coding AI researchers on the way… Then this happened: Free DeepSeek-R1 Then this happened: Free DeepSeek-R1 •DeepSeek-R1 •Open weight: 671B – min >320GB VRAM, >500GB RAM •Reasoning: The model first uses chain-of-thought reasoning to think about the problem. Only once it finishes thinking it starts outputting the answer New keywords •RAG - Retrieval-Augmented Generation, combines retrieval of external information with generative models to enhance responses •Agentic LLM - Large Language Models designed to perform tasks autonomously •Agents - Systems or models capable of autonomous action and tool calling to achieve specified goals Why we cannot use it for own science? •It hallucinates a lot for the specific tasks (knowledge cut) •The answers are fragile, changes its mind, user can negate the answers easily Questions are remains: •How trustable are the answers? •No clear way to proof the results, we need accuracy. •What is the infrastructure behind the scenes? •No information on detail prompt ? Main Challenge: Prompts •Trained knowledge base is limited!!! I, Robot movie scene Note: a prompt engineer gets about +150-400K$/y salary Let’s create an appealing press release image. Pool of Agents Planner Validator User input: Show an example of REANA pipeline for Starhorse datasets from 2024, CMD data is extracted and plotted interactively Results: •Source code •REANA workflow •Data •Plots Knowledge base: Specific rules Planner Knowledge base: Data query Executor Knowledge base: Source Code Coder Knowledge base: Validations Testers Knowledge base: Papers Researcher Knowledge base: Infrastructure Executor LLM Agents Research Assistant LLM Coder Executor Testers Researcher Executor Coder Transferring spectroscopic stellar labels to 217 million Gaia DR3 XP stars with SHBoost, by Khalatyan, Anders, et al. (2024), aa5142724,arXiv:2407.06963 Input from paper Pool of Agents Planner Validator User input: Show an example of REANA pipeline for Starhorse datasets from 2024, CMD data is extracted and plotted interactively Results: •Source code •REANA workflow •Data •Plots Knowledge base: Specific rules Planner Knowledge base: Data query Executor Knowledge base: Source Code Coder Knowledge base: Validations Testers Knowledge base: Papers Researcher Knowledge base: Infrastructure Executor LLM Agents Research Assistant LLM Coder Executor Testers Researcher Executor Coder Enrico Tom Leonhard Strauss (BS student Nov 2024Feb 2025) Enhancing Data Workflows and Reproducibility with LLM Agents An experiment: Get a student, explore LLMs Prof A.Grüning (HOST, Stralsund) Tom Custom Chatbots for specific topic •You are a helpful assistant for giving the daily tips on python. every time when we ask something you should give a tips and tricks on python language for some of libraries in random order: pandas, matplotlib, numpy •Keep examples as shot as possible if user asking about other thems just answer: "I am an assistant for python tips“ •No guarantee answers will relate to other languages or code is 100% correct. An experiment: Get a student, explore LLMs Knowledge Base Supervisor True False LLM Simple models <70B Unable to answer questions on REANA without knowledgebase https://github.com/etlstrauss/bachelor-thesis-public An experiment: add Agents to LLMs Knowledge Base Supervisor True False LLM An experiment: Adding RAG and facts check Knowledge Base Supervisor True False LLM Final chat workflow Thank you for your attention Next Colloquium 6 Interdisciplinary Colloquium on Digitalisation of Research, Arman Khalatyan (AIP), FIZ Karlsruhe, 4 September 2025 DiTraRe Symposium ■2-3 December 2025 ■ZKM Karlsruhe ■Sessions on: ○Knowledge Representation and AI ○Law and Ethics in Digitalisation of Research ○Research Infrastructures ○Impact on Science and Society ■Call for posters https://www.ditrare.de/en/symposium-2025 7 Interdisciplinary Colloquium on Digitalisation of Research, Arman Khalatyan (AIP), FIZ Karlsruhe, 4 September 2025 www.ditrare.de/en Thank you for joining! Stay connected ■DiTraRe ○Website: www.ditrare.de/en ○Email: [email protected] ○LinkedIn: www.linkedin.com/company/ditrare ○Mastodon: social.kit.edu/@DiTraRe ○YouTube: www.youtube.com/@DiTraRe ○Zenodo: zenodo.org/communities/ditrare ■Discussion forum: www.ditrare.de/en/forum ■Newsletter: www.ditrare.de/en/newsletter 8