Expanding Computational Capabilities of the useGalaxy.ca Science Gateway

Expanding Computational Capabilities of the useGalaxy.ca Science Gateway Enis Afgan Department of Biology Johns Hopkins University Baltimore, MD, USA [email protected] Nuwan Goonasekera Melbourne Bioinformatics University of Melbourne Melbourne, VIC, Australia [email protected] Keith Suderman Department of Biology Johns Hopkins University Baltimore, MD, USA [email protected] Jonathan Laperle Centre de calcul scientifique Universit´ e de Sherbrooke Sherbrooke, QC, Canada [email protected] Charles Coulombe Centre de calcul scientifique Universit´ e de Sherbrooke Sherbrooke, QC, Canada [email protected] Carol Gauthier Centre de calcul scientifique Universit´ e de Sherbrooke Sherbrooke, QC, Canada [email protected] Pierre- ´ Etienne Jacques Centre de calcul scientifique and Department of Biology Universit´ e de Sherbrooke Sherbrooke, QC, Canada [email protected] Abstract—We report the integration of Alliance Cloud Connect Pilot (ACCP) resources with the useGalaxy.ca service, enhancing its capacity to support scientific research. This capability enables researchers to harness distributed cloud resources seamlessly and expands the scope of compute resources available on this science gateway. The implementation paves a path for other public Galaxy servers globally and underscores the potential for decentralized, user-driven resource scaling in Galaxy-based science gateways. Index Terms—cloud, bursting, Galaxy I. INTRODUCTION Science gateways offer user-friendly platforms for computational analyses. They are exemplified by the Galaxy project’s useGalaxy.* services that offer global access to fully managed resources, thousands of domain-specific tools, integrate with dozens of data resources, are linked to training materials, and support an active community that counts tens of thousands [2]. useGalaxy.ca, as part of this broader network, represents the Canadian national rendition and aims to primarily support Canadian researchers with an accessible, web-based interface for data analysis. Despite being vast in terms of the resources these services offer, useGalaxy.* gateways have one thing in common: they rely on nationally available shared resources. In turn, it is important to ensure that these resources are used fairly among all users. However, this can limit scalability and customization for any single user. To alleviate this restriction, we have integrated cloud-based compute power directly into the useGalaxy.ca service. This integration represents a critical advancement by enabling scalable and customizable computation for useGalaxy.ca and can easily be used with other useGalaxy.* science gateways. The documentation and code are available from a GitHub repository available at https://github.com/galaxyproject/galaxy-k8s-boot/. II. BACKGROUND The Alliance Cloud Connect Pilot (ACCP) project from the Digital Research Alliance of Canada (DRAC) provides a multi-cloud environment to support Canadian research through flexible and scalable compute resources. As part of the pilot, researchers were able to access commercial clouds such as Amazon, Azure, and Google Cloud through a centralized portal. This setup enabled both managed and user-controlled deployments of scientific software such as Galaxy and Jupyter, streamlining access to advanced computational infrastructure. The pilot aims to experiment with reducing technical barriers and facilitating rapid deployment of cloud native research environments on behalf of domain researchers. Although access to cloud resources supported during the pilot is no longer available, the underlying principles and capabilities described in this paper remain available for deployment on alternative resources. This applies to any Galaxy deployment, not just the one managed by the authors. III. ARCHITECTURE AND IMPLEMENTATION To make use of ACCP resources, we integrated the useGalaxy.ca service with the ACCP portal through two complementary modes of service: useGalaxy Now (mode 1) and Launch-your-Own (mode 2). The relevant software stack for both modes is depicted in Fig. 1. Mode 1 allowed a user to connect their own compute resources to the otherwise managed Galaxy service, expanding the available compute resources for their jobs. This mode was used by a user launching a virtual machine (VM) via the ACCP portal. Once launched, the VM was automatically connected to useGalaxy.ca. This was performed via a relay service that we devloped as part of this work. The service obtains user credentials from the ACCP portal and updates user preferences in Galaxy. Although this could also be done from the ACCP portal, implementing the relay service was deemed a more secure option because the ACCP portal did not have to store a Galaxy administrator key required to update user preferences. Instead, it stored an API key for the relay service, and the relay service stored the Galaxy admin key. Being a light-weight, dedicated service that is a more secure approach for handling necessary credentials. Once connected, useGalaxy.ca was automatically configured to send jobs to this VM, but only from the user that connected the VM - no other user was allowed to send jobs to that machine. Behind the scenes, this mode operated by installing Pulsar, a remote job runner for Galaxy, on the VM and passing the necessary API access key and size of the VM to Galaxy via the relay service. Once that information was received, Galaxy injected that information into the user’s preferences and configured the Total Perspective Vortex (TPV) metascheduler [3] to route jobs from that user to the given VM. The user continued to use Galaxy as usual, but their jobs were routed to the dedicated VM. This allowed them to skip the job queue or any quota restrictions on the number of simultaneous jobs a user can have running. In turn, they benefited from expanded compute capacity and seamless access to Galaxy. Once the user completed their analysis, they deleted the VM through the ACCP portal. The Launch-your-Own mode enabled users to provision dedicated Galaxy instances running entirely on one of the supported clouds (AWS, Azure, and Google Cloud). These standalone instances supported use cases where researchers needed to install custom tools, work with private data that should not be uploaded to a public service, or access large compute and storage resources. These instances were also shareable among a group of collaborators. The deployment process leveraged Kubernetes and Helm for reproducibility and robustness, with automated setup taking only minutes. A user started by launching Galaxy via the ACCP portal. Once a VM started, an Ansible role was automatically run to set up Kubernetes, Helm, a web proxy, a shared file system, and to install the Galaxy Helm chart [4]. Users were presented with an IP address where they could log into the system. The user account information was managed via the ACCP portal, so additional users could easily be added. The authenticated users were treated as administrators of the Galaxy servers, allowing them to install additional tools and perform other administrative functions. Once the need for the server diminishes, the users deleted the instance via the ACCP portal. IV. ENABLING DECENTRALIZED RESOURCE EXPANSION IN GALAXY SERVICES Although the ACCP was decommissioned, the successful integration of these resources was a demonstration of a blueprint for decentralized, scalable resource expansion across the global useGalaxy network. Traditionally, useGalaxy nodes have operated within fixed, centrally managed infrastructures. The ACCP-backed model demonstrates how public servers can augment their capabilities with dynamic, user-provisioned resources while retaining a familiar interface and consistent user experience. This approach offers a sustainable path to support growing user demand without relying solely on naFig. 1. Stack diagram of the components used to deploy Galaxy on ACCP resources. Core layers are reused, helping minimize code redundancy. All the components are portable and robust across cloud providers. tional infrastructure investments. By allowing researchers to attach their own commercial or community cloud allocations, it distributes both the cost and management overhead while enabling advanced scientific workflows. V. LIMITATIONS AND FUTURE ENHANCEMENTS Both the useGalaxy Now and Launch-your-Own modes come with unique tradeoffs, summarized in Table I. Although useGalaxy Now provides immediate access to Galaxy through the useGalaxy.ca platform, bottlenecks could arise due to data transfer latency between usegalaxy.ca and the remote VM. In scenarios where data transfer duration exceeds computational runtime, such as with lightweight jobs (e.g., text parsing), the approach becomes suboptimal. Conversely, high-compute, long-duration workflows (e.g., genome assembly) often justify the transfer overhead. This must also be weighed against queue wait times on public infrastructure, which a private VM will reduce or eliminate. Future enhancements include integrating profiling-based automation to recommend the optimal subset of tools, parameters, and data size thresholds suitable for offloading to cloud resources. Such a decision engine could be based on workload metadata, past runtimes, and job characteristics. Similarly, while Launch-your-Own empowers users to spin up fully isolated Galaxy instances with full administrative control, support for custom tools, and enhanced data privacy, it requires users to administer their environment, which can be a barrier for non-expert users. Provisioning latency (ranging from 5 to 20+ minutes) and lack of direct administrator TABLE I GALAXY DEPLOYMENT MODES IN ACCP – LIMITATIONS AND FUTURE ENHANCEMENTS Mode Limitations / Tradeoffs Proposed Enhancements useGalaxy Now •Performance degrades if data transfer exceeds compute time. •Inefficient for short, light jobs. •Requires data transfer to a remote Pulsar node. •Intelligent job routing based on workload size and duration. •Profiling-based automation to suggest remote execution. Launch-your-Own •Requires users to manage their own Galaxy instance. •Provisioning latency (5–20+ mins). •No direct system administrator support. •Automated lifecycle tooling (monitoring, upgrades, maintenance). •Seamless SSO and IAM integration. Common to Both Modes •No autoscaling currently implemented. •Lack of cost visibility or control mechanisms. •Enable autoscaling using Kubernetes orchestration. •Integrate cost monitoring, alerting, and budgeting tools. •Transition to a shared-responsibility model with centralized management. support, unlike a managed service, present additional challenges. To address these, tooling for automated provisioning, monitoring, and teardown are needed to reduce operational burden and provide a ”set-and-forget” deployment experience. At present, neither mode supports autoscaling or automated cost control. This limits operational efficiency and resource optimization. These capabilities are particularly crucial for enabling dynamic right-sizing and budget/energy-conscious computing in multi-cloud environments. Additional enhancements include GPU/AI workload integration, and sharedresponsibility models for security and maintenance. Such an architecture would facilitate seamless transitions between managed and self-managed configurations, depending on user needs. VI. BROADER IMPLICATIONS FOR SCIENCE GATEWAYS The integration of ACCP resources into useGalaxy.ca exemplifies a broader trend in science gateways toward hybrid resource models [1]. This development is aligned with the principles of community-driven resource sharing and customizable environments for diverse scientific disciplines. By combining centralized oversight with decentralized resource provisioning, science gateways like Galaxy can remain responsive to emerging research needs while fostering innovation and scalability. Moreover, this model is highly portable and can be replicated across other science gateways seeking to bridge the gap between static, centrally managed resources and the dynamic requirements of modern research. The use of Kubernetes and Helm further encourages cross-platform consistency and resilience, essential traits for science gateways operating in complex, multi-cloud environments. VII. CONCLUSION The addition of ACCP support to useGalaxy.ca service represents a notable advance in the evolution of useGalaxy gateways. By integrating flexible, scalable cloud resources into the existing Galaxy framework, it provided researchers with new levels of control and capacity. This capability charts a path forward for decentralized, user-driven resource scaling across the global useGalaxy.* network and the broader science gateway community. Although access to the pilot’s resources has ended, the core capabilities remain available, and we are exploring options to connect to alternative, more stable resource providers. ACKNOWLEDGMENT This work was in part supported by the Digital Research Alliance of Canada, Universit´ e de Sherbrooke, Johns Hopkins University, and University of Melbourne. REFERENCES [1] Prasad Calyam, Nancy Wilkins-Diehr, Mark Miller, Emre H Brookes, Ritu Arora, Amit Chourasia, Douglas M Jennewein, Viswanath Nandigam, M Drew LaMar, Sean B Cleveland, et al. Measuring success for a future vision: Defining impact in science gateways/virtual research environments. Concurrency and Computation: Practice and Experience, 33(19):e6099, 2021. [2] The Galaxy Community. The Galaxy platform for accessible, reproducible, and collaborative data analyses: 2024 update. Nucleic Acids Research, 05 2024. [3] Nuwan Goonasekera, Catherine Bromhead, Simon Gladman, Nate Coraor, Bjorn Gruning, and Enis Afgan. Right-sizing compute resource allocations for bioinformatics tools with total perspective vortex, 2023. [4] Nuwan Goonasekera, Alexandru Mahmoud, Keith Suderman, and Enis Afgan. Galaxy Helm chart: a standardized method for deploying production Galaxy servers. Bioinformatics, 40(8):btae486, 08 2024.