Evaluating AI: An Overview of the AI as Infrastructure (AIINFRA) Project

CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 1 Evaluating AI: An Overview of the AI as Infrastructure (AIINFRA) Project James Smithies Professor of Digital Humanities Director HASS Digital Research Hub The Australian National University 19 - 21 NOVEMBER 2025 Glen Berman PhD Candidate and Senior Research Officer Director HASS Digital Research Hub The Australian National University AI Meets Humanities: NTU-ANU-KCL Workshop November 19 – 21, 2025 Nanyang Technological University, Singapore CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C Overview 01 The AI as Infrastructure Project 02 HASTRICT Principles 03 ATLAS Darwin 3ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025 CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C 01 THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 3 •https://aiinfra.anu.edu.au/ •ANU HASS Digital Research Hub •ANU Library •Aotearoa / New Zealand Department of Internal Affairs •Australian Parliamentary Library •King’s College London •The National Library of Australia •UK History of Parliament project •UK National Archives •2024 – 2026, ANU Futures Scheme Partners A trans-national project focussed on assessing the value of Retrieval Augmented Generation (RAG) for historical research, using Hansard debates from Australia, New Zealand, United Kingdom as our content baseline. ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025 CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C 01 THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 4 •Identify and implement best practices for the design, development, evaluation, and maintenance of historical research tools enabled by Large Language Models (LLMs). •Develop high-level principles (the HASTRICT principles), an organisational maturity model, and associated evaluation framework to support the responsible implementation of AI tools into the GLAM and academic sector. •Develop a prototype technical ‘test harness’ for the evaluation of LLM Retrieval Augmented Generation (RAG) systems, and as a test bed for development of the other project outputs, using open-source tools. Goals ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025 CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C 5 •H: Human •A: Accountability •S: System targets •T: Transparent •R: Reproducible •I: Iterative •C: Contextually Informed Measures & Baselines •T: Tasks, workflows & use cases 04 HASTRICT PRINCIPLES James Smithies, Glen Berman, Karaitiana Taiuru, ‘Operationalisation of Responsible AI in the Research and Cultural Sectors’. In Building Digital Humanities, edited by Simon Burrows et al. Radboud University Press. Under review. Department of Industry Science and Resources, “Guidance for AI Adoption: Implementation”, Department of Industry Science and Resources,” October 21, 2025, https://www.industry.gov.au/publications/guidance-for-ai-adoption/guidance-ai-adoptionimplementation-practices. ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025 CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C 6 04 ATLAS DARWIN ATLAS: Analysis and Testing of Language Models for Archival Systems •https://github.com/AI-as-Infrastructure/aiinfra-atlas •https://github.com/AI-as-Infrastructure/aiinfra-atlasdarwin ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025 CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C 7 04 ATLAS DARWIN ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025 CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C 8 04 ATLAS DARWIN ATLAS: Analysis and Testing of Language Models for Archival Systems •https://app.phoenix.arize.com/ ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025 CHOOSE A SLIDE STYLE FROM 1. Home tab 2. New Slide 3. Choose layout 4. If layout goes awry, select Reset TEXT There are 5 levels of formatted text available. Please move between text levels using the increase/decrease button on the menu above. CHANGE SLIDE BACKGROUND 1. Design Menu 2. Format Background 3. Picture or Texture Fill 4. Find image REPLACE IMAGE IN SHAPE OR ON PAGE 1. Right click on image 2. Change Picture 3. Select your image OR 1. Delete the image 2. Click on icon 3. Find the new image TO REPOSITION IMAGE WITHIN SHAPE 1. Click on image 2. Format menu 3. Click on Crop button 4. To fit the whole image inside select FIT 5. To use only a portion select FILL then crop, move or resize image to show properly within shape. ADD/CHANGE FOOTER 1. Insert Menu 2. Header & Footer 3. Tick to activate/Untick to remove 4. Change text TEQSA PROVIDER ID: PRV12002 (AUSTRALIAN UNIVERSITY) CRICOS PROVIDER CODE: 00120C 9 04 ATLAS DARWIN ATLAS: Analysis and Testing of Language Models for Archival Systems ANU HASS DIGITAL RESEARCH HUB | EVALUATING AI: AN OVERVIEW OF THE AI AS INFRASTRUCTURE (AIINFRA) PROJECT 19 - 21 NOVEMBER 2025