Democratizing Scientific Machine Learning through Expert-Encoded Workflow Templates

Democratizing Scientific Machine Learning through Expert-Encoded Workflow Templates Andrii Kryvenko High Performance Research Computing Texas A&M University College Station, TX, USA andriikryvenk[email protected] Marinus Pennings High Performance Research Computing Texas A&M University College Station, TX, USA [email protected] Honggao Liu High Performance Research Computing Texas A&M University College Station, TX, USA [email protected] Abstract—The increasing complexity of AI/ML software stacks and diverse accelerator hardware on High-Performance Computing (HPC) systems creates a significant productivity barrier for domain scientists. Configuring optimized, reproducible AI workflows often requires specialized systems expertise that researchers may not possess. To tackle this, we introduce Drona, a workflow engine that encapsulates expert knowledge into reusable, domainspecific templates. Drona guides researchers through complex setups via a simple web interface, dynamically generating optimized and transparent workflows. We demonstrate Drona’s power and flexibility through an in-depth analysis of a Hugging Face template that automates the model inference and fine-tuning lifecycle, from proactive resource validation and distributed training configuration to seamless reproducibility. Drona lowers the barrier to entry for advanced AI/ML on HPC, improves scientific reproducibility, and empowers a wider community of researchers to leverage accelerated computing. Index Terms—Artificial Intelligence, Machine Learning, High Performance Computing, User Interfaces, Hugging Face I. INTRODUCTION The widespread adoption of artificial intelligence and machine learning in scientific research has introduced significant infrastructure complexity that can impede research productivity. Researchers must increasingly navigate sophisticated AI workflows across a diverse and rapidly evolving landscape of computational hardware. Modern High-Performance Computing (HPC) systems, such as the ACES cluster at Texas A&M University, feature diverse accelerator architectures including NVIDIA GPUs, Intel Datacenter MAX GPUs, Graphcore Intelligence Processing Units (IPUs), and NEC Vector Engines, each demanding its own distinct softwar stacks, optimization strategies, and configuration approaches to achieve optimal performance. The direct consequence of this technological diversity is a productivity barrier for domain scientists. While experts in their respective fields, they are increasingly burdened with acquiring deep systems-level knowledge that falls far outside their primary research focus. Configuring an optimized, multinode distributed training job, for instance, requires specialized expertise in launch commands, network interconnect settings, and hardware-specific libraries. This complexity not only slows the pace of discovery but also introduces a threat to scientific reproducibility, as the subtle yet essential configuration details for a successful job are often undocumented and difficult to share. To address these challenges, we present the Drona Workflow Engine, a framework that shifts the paradigm from procedural scripting to declarative, goal-oriented specification. Drona encapsulates complex systems knowledge into reusable, domainspecific templates, allowing researchers to define what they want to achieve, while the system handles how to execute it optimally on available hardware. While our prior work demonstrated the framework’s versatility for established scientific applications and educational tutorials [1], this paper extends the Drona methodology to the dynamic and complex domain of modern large-scale machine learning. Specifically, we introduce and analyze a new Hugging Face [2] template that automates the workflow for large language model finetuning and inference. This workflow guides researchers in selecting models, automatically configures multi-accelerator distributed training, and provides proactive validation to prevent common resource-related failures. Through this example, we demonstrate how Drona’s template-based approach can lower the barrier to entry for advanced AI/ML on HPC and improve scientific reproducibility. Moreover, our open-source implementation enables community-driven development of additional domain-specific templates across diverse research domains. II. DRONA WORKFLOW ENGINE Drona is a workflow engine that translates high-level researcher goals into ready-to-run computational jobs using self-contained environment templates that encapsulate domainspecific expertise. The engine, publicly available on GitHub [3], operates through three core components: a declarative user interface, a dynamic generation engine, and built-in reproducibility mechanisms, as illustrated in Figure 1. A. Environment Templates User interface is driven by a schema.json file within each template, which declaratively defines a dynamic web form. This schema supports rich features, such as rendering custom HTML, creating conditionally visible elements, and populating values by executing shell scripts on the fly. schema.json utils.py map.json Fig. 1. The Drona Workflow Engine architecture demonstrating the complete workflow from researcher input to executable job generation. Researchers interact with a dynamic web form (left) whose structure is defined by a declarative schema.json specification. The job submission history below the form enables one-click reproduction of previous experiments. When researchers submit the form, the map.json file processes their inputs and invokes functions from utils.py to generate optimized job scripts and configurations. The system presents a comprehensive job preview (right) with editable tabs for all generated files, including intelligent warnings (shown in yellow) when resource validation detects potential issues. B. Dynamic Job Generation Engine Upon form submission, a map.json specification orchestrates the generation process, linking UI inputs to job parameters and invoking functions from expert-authored Python scripts (utils.py). This design enables complex, contextaware logic—such as dynamically injecting IPU-specific configurations when Graphcore accelerators are selected. The system then renders template files (e.g., job.sh,run.py) and presents them in an editable preview interface for transparency and final adjustments. C. Reproducibility by Design Every job launched through Drona has its complete context, including all provided inputs and the final generated files, stored in a persistent job history. From this history, a researcher can inspect past runs and launch precise replicas of any previous experiment with a single click. Moreover, the selfcontained nature of Drona templates facilitates broad sharing and collaboration among researchers. D. Implementation Details Drona Workflow Engine is implemented as a Flask/React WebApp and can be used standalone, e.g., on a researcher’s local machine, or as part of an external web-based platform. The ACES cluster utilizes the Open OnDemand (OOD) [4] web-based platform, enabling users to access and manage the cluster through a web browser. Consequently, Drona is encapsulated as a Phusion Passenger app to work within the OOD framework. Additionally, Drona can seamlessly integrate with other frameworks that manage computational workloads. One such example is TAPIS [5], where Drona can act as a frontend, allowing the researcher to specify domain-specific information in the Drona form. Drona then generates TAPIS code for system, application, and application arguments files, and submits the job through TAPIS. III. A WORKFLOW FOR HUGGING FACE INFERENCE AND FINE-TUNING To demonstrate how the Drona architecture translates system-level expertise into a practical, researcher-focused tool, this section analyzes a Drona environment for Hugging Face [2]. This environment streamlines the entire lifecycle of using transformer models on an HPC cluster. By capturing a researcher’s high-level goals through a guided web interface, the Drona engine generates a fully configured, optimized, and executable job package for inference and fine-tuning of language models. The following subsections detail the key features that enable this intelligent workflow, using a finetuning job as a running example. A. Dynamic Model Selection and Validation The researcher’s first step is to select a model. To ensure access to the latest advancements, the environment leverages the Hugging Face Hub API to dynamically populate a searchable dropdown menu. After the researcher selects the TinyLlama/TinyLlama-1.1B-Chat-v1.0 model [6], the system immediately provides additional information about the model such as parameter count and VRAM usage of the model as shown in Figure 2. The VRAM estimation is calculated using established methodologies for analyzing transformer memory requirements, as detailed by EleutherAI [7]. For example, a full fine-tuning would require 7.71 GB of VRAM. In contrast, by using LoRA optimizations [8], the same task requires only 2.00 GB. This immediate, data-driven feedback guides the user toward a viable training strategy, preventing job failures due to resource exhaustion. Fig. 2. Upon selecting the TinyLlama/TinyLlama-1.1B-Chat-v1.0 model, Drona presents a VRAM estimation table. This proactive validation shows that LoRA fine-tuning at float16 requires 2.00 GB, a safe choice, while a full fine-tune would demand 7.71 GB, guiding the user’s hardware and training strategy selection. B. Flexible Data Integration Next, the researcher selects a dataset for the chosen model. The environment is designed for flexibility, supporting data ingestion from the Hugging Face Hub, a local file upload, or direct text input. In this case, the researcher chooses the flwrlabs/shakespeare dataset [9]. To align the data with the model’s requirements, the interface prompts them to map the dataset’s columns to the expected inputs. As seen in Figure 3, the researcher maps the ‘x‘ and ‘y‘ columns, a simple but powerful feature that makes the workflow adaptable to custom diverse data schemas. Fig. 3. After selecting a model, the user chooses the flwrlabs/shakespeare dataset and maps its ‘x‘ (input) and ‘y‘ (target) columns to the model’s training requirements, demonstrating the environment’s data schema flexibility. C. Automated Hardware and Distributed Training Configuration With the model and data defined, the researcher configures the computational resources. As shown in Figure 4, they select NVIDIA H100 GPUs and request a multi-node, multiGPU job by setting ‘Number of GPUs/node‘ to ‘2‘ and ‘Number of Nodes‘ to ‘2‘. In response, the Drona engine will generate a fully configured job script that handles the complexities of a multi-node Distributed Data Parallel (DDP) launch using PyTorch [10]. The researcher could just as easily have selected ”Graphcore IPU” from the accelerator menu, and for compatible models Drona would have generated a completely different, specialized script utilizing the Hugging Face Optimum library [11], which provides tools to run models on targeted hardware with maximum efficiency. Fig. 4. The researcher configures a multi-node, multi-GPU training job by requesting 2 H100 GPUs on 2 nodes. Drona uses these inputs to automatically generate a script for Distributed Data Parallel training. Fig. 5. The interface for hyperparameters. The researcher sets basic parameters like learning rate and epochs in simple fields, while using the advanced JSON editor for more specific arguments like the learning rate scheduler type. D. Flexible Hyperparameter Tuning Finally, the researcher specifies the fine-tuning parameters. The interface provides a two-tiered approach for both ease of use and expert control. As seen in Figure 5, the researcher uses the simple input fields to set a ‘Learning Rate‘ of ‘0.0002‘, ‘3‘ training epochs, and a ‘Batch Size‘ of ‘4‘. For more granular control, they then use the integrated JSON editor to specify advanced arguments, such as setting the ‘lr scheduler type‘ to ‘cosine‘. This empowers researchers to experiment quickly while retaining the full control needed for advanced tuning. Fig. 6. Drona’s Job Preview screen. All generated files for the fine-tuning workflow are presented in editable tabs (left), while the live log from the cluster execution is streamed in real-time (right), confirming the job’s progress and successful completion. E. Job Preview and Reproducibility Mechanism Upon the submission of the form, the researcher has a chance to inspect the generated scripts and modify them if needed before launching the job as shown in Figure 6. Warnings, errors, and notes dependent on the chosen settings can be displayed to the researcher at this stage as well as shown in the Figure 1. As detailed in the section II-C, when the job is launched, all of the inputs and generated scripts are saved to a persistent job history, which allows the researcher to re-launch precise replicas of their experiments in the future. IV. SUMMARY AND FUTURE WORK This paper has demonstrated, through an analysis of a new environment for Hugging Face workflows, that encapsulating expert knowledge in domain-specific templates can significantly lower the barrier to entry for complex, accelerated computing. Researchers can focus on performing their actual research instead of dealing with HPC specifics Future work will include investigating possibilities for a dedicated AI/ML dashboard. The ACES cluster at Texas A&M University is part of the NAIRR Pilot initiative [12], and a dedicated AI/ML dashboard will be beneficial for its researchers. In addition to the Hugging Face environment, the dashboard will provide further functionality to support AI/ML research needs, including collaboration tools, AI/ML training environments such as IPU training, and AI tools like AlphaFold [13]. The latter two are already available as Drona environments. REFERENCES [1] M. Pennings, A. Kryvenko, and H. Liu, “Generating scientific workflows with drona environments,” in Practice and Experience in Advanced Research Computing (PEARC ’25), 2025, forthcoming. [2] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., “Huggingface’s transformers: State-of-the-art natural language processing,” arXiv preprint arXiv:1910.03771, 2019. [3] Texas A&M High Performance Research Computing, “Drona workflow engine github repository,” 2025. [Online]. Available: https://github.com/ tamu-edu/dor-hprc-drona-composer [4] D. Hudak, D. Johnson, A. Chalker, J. Nicklas, E. Franz, T. Dockendorf, and B. L. McMichael, “Open ondemand: A web-based client portal for hpc centers,” Journal of Open Source Software, vol. 3, no. 25, p. 622, 2018. [Online]. Available: https://doi.org/10.21105/joss.00622 [5] J. Stubbs, R. Cardone, M. Packard, A. Jamthe, S. Padhy, S. Terry, J. Looney, J. Meiring, S. Black, M. Dahan, S. Cleveland, and G. Jacobs, “Tapis: An api platform for reproducible, distributed computational research,” in Advances in Information and Communication, K. Arai, Ed. Cham: Springer International Publishing, 2021, pp. 878–900. [6] P. Zhang, G. Zeng, T. Wang, and W. Lu, “Tinyllama: An open-source small language model,” arXiv preprint arXiv:2401.02385, 2024. [7] EleutherAI, “A guide to transformer math,” https://blog.eleuther.ai/ transformer-math/, 2023. [8] Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang et al., “Lora: Low-rank adaptation of large language models.” [9] S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Koneˇ cn` y, H. B. McMahan, V. Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,” arXiv preprint arXiv:1812.01097, 2018. [10] A. Paszke, “Pytorch: An imperative style, high-performance deep learning library,” arXiv preprint arXiv:1912.01703, 2019. [11] Hugging Face, “Optimum,” https://huggingface.co/docs/optimum, 2024. [12] NAIRR Pilot, “Advancing us innovation in artificial intelligence,” https: //nairrpilot.org/, 2024. [13] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. ˇ Z´ ıdek, A. Potapenko et al., “Highly accurate protein structure prediction with alphafold,” nature, vol. 596, no. 7873, pp. 583–589, 2021.