From demos to dependability: Practical evaluation strategies for AI applications
Abstract
The advent of Generative AI has transformed how we build AI models and software, but the speed we gain comes at a cost. Large language models (LLMs) let us add "intelligence" to an app in minutes, yet most prototypes stall before production because no one trusts their behaviour. We trace the shift from the traditional data-first, then model pipeline to today’s app-first, then API workflow and explain why it leaves teams blind, often without benchmark datasets or KPIs to judge performance.The talk presents ways Research Software Engineers can close this gap by making evaluation a first-class citizen. We survey three families of evaluation for AI applications: manual prototypes, data-gathering tools, and synthetic benchmark generation, giving an overview of the field. We do a deep dive into the latest research and reflect on lessons learned over the past few years. To ground the theory, we walk through concrete examples from our own projects.Attendees will leave with practical insights for matching evaluation strategy to project maturity, regulatory constraints, and available resources. Evaluation has become the critical bottleneck and RSEs are uniquely placed to turn it into an enabler for reproducible, sustainable science.A recording of this session is available on YouTube: https://youtu.be/dCHzTfT1zw8
Full text
From demos to dependability: Practical evaluation strategies for AI applications RSECon2025 Roman Wixinger Data Scientist (Ergon) Hannes Stählin Data Scientist (Ergon, ETH) 1
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 1. Introduction 2. Evaluation Strategies 3. Case Studies 4. Conclusion 2 From demos to dependability: Practical evaluation strategies for AI applications
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Introduction: What kind of AI systems do RSEs usually build? 3 Applied applications AI systems that are used to improve the operations of the research institute ▪Deep research tool for the university library ▪Course chatbot ▪Automated grading of multiple-choice tests ▪Use LLMs to reduce ambiguity in exams questions Research applications AI systems that are part of the research itself, either as subject of interest or as a tool ▪Computer Vision models for cell classification ▪Legal decision making with LLMs ▪Efficient quantization methods for foundation models
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Introduction: How are these AI systems usually developed? 4 Training Evaluation Release Training Data Model Evaluation Application Back in the days (Machine Learning)
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Introduction: How are these AI systems usually developed? 5 Training Inference Release Training Data Model Evaluation Application Classify cats and dogs: {“image_url”: “...”, “cat”, “Image_url”: “...”, “dog”} Evaluation metrics ● Accuracy ● Recall ● F1-Score ● Confusion matrix Back in the day (Machine Learning)
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Introduction: But today we have foundation models 6 Definition of the term “Foundation model” AI model that ▪is trained on broad data at scale ▪is designed for generality of outputs ▪and can be adapted to a wide range of distinctive tasks Bommasani et al. (2021), On the Opportunities and Risks of Foundation Models.
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Introduction: How are these AI systems usually developed? 7 Back in the day (Machine Learning) Many people today Training Inference Release API call Data Model Evaluation Application Model Application Evaluation Data
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Introduction: We need to bring back data and evaluation to AI Evaluation is crucial for ▪Making informed decisions about the use of AI ▪Build trust with stakeholders ▪Optimize performance ▪Produce reliable findings and systems … and a key part of the scientific method! 8
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 9 1. Introduction 2. Evaluation Strategies 3. Case Studies 4. Conclusion From demos to dependability: Practical evaluation strategies for AI applications
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 16 Manual testing Hand-crafted datasets Synthetic datasets Make it easy for users to gather data, which allows for repeated evaluation. Evaluation strategies
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 17 Answer: Synthetic data is data that was created artificially Examples: LLM-generated text or conversation Images and videos from game engines Images created from geometric shapes What is synthetic data?
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 18 Motivational example: Running LLMs for structured output on smartphones What if we want to use small LLMs on the smartphone? ▪Problem: Base model does not generate structured output (JSON) well enough ▪Idea: Fine-tune the model on synthetic data to make it follow the output format ▪Result: For specialized, domain-specific tasks, we can match the performance of larger models, producing correct JSON output
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 19 Example: Unnatural instructions What to do if human labels are expensive? Augment your existing data Honovich, O., Scialom, T., Levy, O., & Schick, T. (2022). Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor. arXiv preprint arXiv:2212.09689.
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 20 Applications in industry Optimize ML models for domain specific applications, for example execution on the smartphone Speech recognition for specific instructions Automated document processing
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 21 Evaluation strategies Manual testing Hand-crafted datasets Synthetic datasets Allows for the rapid generation of massive, fully labeled datasets. But might not represent the real world.
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Application of LLMs in research How can we evaluate legal reasoning with LLMs? Y. Fan et al. (2025), LEXAM: Benchmarking Legal Reasoning on 340 Law Exams
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 23 Example: Alternative Annotator Test vs Compare alignment with humans Calderon, N., Reichart, R., & Dror, R. (2025). The Alternative Annotator Test for LLM-as-a-Judge How can we know that using an LLM judge is valid?
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 24 Evaluation strategies Manual testing Hand-crafted datasets Synthetic datasets Great for fast, initial iterations, but does not really scale! Allows for the rapid generation of massive, fully labeled datasets. But might not represent the real world. Make it easy for users to gather data, which allows for repeated evaluation.
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] 25 Case Studies 1. LexCraft (University of Bern) 2. Precision OncoAssist (University Children's Hospital Zurich)
RSECon2025 / From demos to dependability: Practical evaluation strategies for AI applications Roman Wixinger, [email protected] Hannes Stählin, [email protected] Treating evaluation as a first class citizen, you gain: ●Trust ○Evaluation builds confidence with stakeholders. ●Performance ○Evaluation drives meaningful optimization. ●Reliability ○Evaluation ensures robust and reproducible systems. Testing and evaluation is a key part of good (research) software engineering! Key takeaways 56
From demos to dependability: Practical evaluation strategies for AI applications RSECon2025 Roman Wixinger Data Scientist (Ergon) Hannes Stählin Data Scientist (Ergon, ETH) 57 Thank you!