A Science Gateway as a Hub for Island Resilience

A Science Gateway as a Hub for Island Resilience 1st Sean B. Cleveland University of Hawai‘i -System Information Technology Services Honolulu, USA [email protected] 2nd Jared McLean University of Hawai‘i -System Information Technology Services Honolulu, USA [email protected] 3th Cheryle Heu University of Hawai‘i at M¯ anoa Water Resource Research Center Honolulu, USA [email protected] 4th Derek Ford University of Hawai‘i at M¯ anoa Water Resource Research Center Honolulu, USA [email protected] 5th Jennifer Geis University of Hawai‘i -System Information Technology Services Honolulu, USA [email protected] 6th Matthew P. Lucas University of Hawai‘i at M¯ anoa Water Resource Research Center Honolulu, USA [email protected] 7th Keri Kodama University of Hawai‘i at M¯ anoa Water Resource Research Center Honolulu, USA [email protected] 8th Ryan Longman Pacific Islands Climate Adaptation Center Water Resource Research Center Honolulu, USA [email protected] 9th Thomas W. Giambelluca University of Hawai‘i at M¯ anoa Water Resource Research Center Honolulu, USA [email protected] Abstract The Change Hawai‘i (Change-HI) project is fundamentally addressing the existential threats of increasing temperatures and frequency of extreme weather events in Hawai‘i by integrating data and environmental dynamics research to foster statewide resilience, enhance decision science, and support workforce development in critical fields. A cornerstone of this initiative is the Hawai‘i Climate Data Portal (HCDP), which operates as a vital science gateway and data hub [?] The primary objective of the HCDP is to bring together climate data for the state of Hawai‘i and serve as a robust resource to monitor, visualize and communicate environmental change [1]. Its critical role is highlighted by its extensive provision of climate data and its Application Programming Interface (API), which is instrumental in the development and functionality of various decision support tools tailored for various stakeholders across the state. This paper details the role of the HCDP as a platform that supports the development of actionable climate science outcomes for Hawai‘i. I. INTRODUCTION The Hawaiian Islands face profound and unique vulnerabilities to warming temperatures, extreme weather events, and environmental dynamics. These existential threats necessitate a concerted effort to enhance resilience across the state. The Change Hawai‘i (Change-HI)) project has been established to confront these challenges by leveraging the power of data science and climate science. Central to Change-HI’s strategy is the Hawai‘i Climate Data Portal (HCDP), conceived as a comprehensive science gateway and data hub [1]. The HCDP is designed to serve as an indispensable resource for monitoring, visualizing, and communicating climate-related changes. The importance of the HCDP as a data hub is underscored by its ability to provide a wide array of climate data and, critically, its Application Programming Interface (API) [2]. This programmatic access is fundamental, enabling the creation and functionality of various decision support tools that are specifically tailored to the diverse needs of stakeholders throughout the Hawaiian Islands. This paper expands on the architecture, capabilities, and societal impact of the HCDP, demonstrating its pivotal role in supporting tools that translate complex climate science into actionable information for stakeholders. II. BACKGROUND A. The Change(HI) Project Overview The Change(HI) project is driven by a clear vision: to develop actionable data and climate science for stakeholders to effectively address the existential threats related to extreme weather events and environmental dynamics in Hawai‘i. Its mission is to ’harness the data revolution’ to build statewide resilience, foster economic development, and expand educational opportunities in climate science and data analytics. The project pursues three overarching goals: 1) to enhance fundamental understanding of Hawai‘i’s current and future climate, 2) to integrate technology and data science into research to produce actionable climate science outcomes and improve communication with state stakeholders, and 3) to provide education and training in data science for workforce readiness. These efforts are contextualized by the unique needs of the state of Hawai‘i and the broader USaffiliated Pacific region, which face climate-related economic, social, and health precarity. B. The Hawai‘i Climate Data Portal (HCDP) The HCDP serves as the leading platform for accessing climate data and analytics in Hawai‘i. It hosts a broad spectrum of data, including extensive historical climate data and near-real-time (NRT) datasets [3]. Significant NRT additions in Year 3 of the project include daily relative humidity, daily rainfall, daily fire ignition probability, daily normalized difference vegetation index (NDVI), a measure of vegetation health by analysing ”greenness” via satellite imagery, and standardized precipitation index (SPI), a meteorological drought index, over several different time scales. These datasets greatly enhance the portal’s capabilities. These datasets complement existing NRT monthly rainfall and daily and monthly air temperature datasets and are crucial for various analyses, such as rainfall frequency analysis, baseline climatologies, and decision-support tools. Additionally, the ignition probability and SPI NRT workflows rely on the rainfall, air temperature, relative humidity, and NDVI datasets, enabling more advanced workflows within the HCDP environment. The portal also provides legacy (1920–1990) and contemporary (1990–Current) climatology maps for rainfall and temperature, enabling users to compare historical trends with current conditions. Data sources for the HCDP include the Meteorological Assimilation Data Ingest System (MADIS), Hydrometeorological Automated Data System (HADS), National Weather Service (NWS), and local station networks like the Hawai‘i Mesonet. The HCDP’s utility is evident in its substantial usage statistics, reporting over 3 million API requests and over 110,000 thousands of visitors that accessed over 58 million data files. The portal is continuously improved through enhanced data acquisition workflows, a robust backend database, and advanced anomaly detection methods that utilize AI/ML methods. Collaborative efforts are also advancing gap-filling methods and leveraging deep learning approaches for spatial interpolation of sparse climate observations across Hawai‘i’s complex terrain. These technical advancements continue to improve the HCDP and keep it relevant as a data hub. 1) Data Size and Frequency: The HCDP hosts several NRT and static datasets including daily and monthly precipitation [3], daily and monthly air temperature [4], daily relative humidity, daily fire ignition probability, daily NDVI, SPI, future rainfall and air temperature projections, and averaged climatologies. The NRT daily rainfall dataset covers 1990 to the present day and is continuously updated. Fire ignition probability maps extend back to 2002. The climate data aggregation and mapping workflows are designed to execute every day to collect information from thousands of stations from multiple providers and generate NRT daily map products and ensure transient data is captured before it becomes unavailable from source providers. This daily execution leverages Jetstream2 [5] for executing containerized data aggregation, climate analysis, and product workflows. The Hawai‘i MesoNet project manages climatological sensor stations across that state. The raw sensor station data is recorded in a 5 minute interval and collected by a management software called Loggernet, from the logger manufacturer Campbell Scientific, every 15 minutes. The data is then ingested into a MesoNet database as well as daily timestep files which are uploaded to the gateways storage system. This data is processed daily along with sensor station data from sensor networks managed by other organizations. The values go through a quality assurance and quality control (QA/QC) process and are gap filled, attempting to fill missing values, before being used to produce the maps generated by the HCDP NRT workflows. C. The Cyberinfrastructure Backbone of the HCDP The HCDP’s robust functionality relies heavily on the Tapis framework, an open-source API platform developed at the Texas Advanced Computing Center (TACC) [6]. Tapis is designed to facilitate reproducible, distributed computational research by providing a unified, simple-to-use API for managing data and executing codes across a wide range of remote systems, from high-performance computing (HPC) clusters to cloud resources like the National Science Foundation’s Jetstream 2 [5]. Funded by the National Science Foundation (NSF) since 2019, Tapis delivers production-grade capabilities for securely executing workflows across geographically distributed providers, storing and retrieving streaming/sensor data with temporal and spatial indexing, and leveraging containerized codes for portability. It also enhances reproducibility through provenance tracking and manages data access via a fine-grained permissions model. The University of Hawai‘i operates a hybrid Tapis deployment, utilizing both local Tapis API services and those hosted at TACC that support multiple science gateways [7] from microbiome [8] to groundwater recharge [9]. Tapis’s core APIs include the Files API for remote storage management, the Apps and Jobs APIs for software execution, the Actors API for low-latency executables, the Meta API for scalable document storage, and the Streams API for sensor data [10]. The platform leverages container technology, primarily Docker, to enhance portability and reproducibility for both user computations and Tapis’s internal operations. This distributed microservice architecture and robust security kernel combined with custom data serving APIs provide the HCDP with the necessary infrastructure for its complex data processing and dissemination tasks. The HCDP leverages Tapis to store and retrieve processed station data and provide access to the data files managed by the HCDP, enabling data sharing with the wider community. While Tapis offers a robust platform of services, the HCDP team has developed additional custom APIs to address some needed features such as: •Custom Data Export and Packaging: The Tapis Files API for remote storage management had limitations. Directly using it would require extensive client-side logic for file paths, which is cumbersome for large datasets and would necessitate application modifications for any storage changes. To overcome this, the HCDP’s own API was developed to index stored files based on their dataset attributes and dates, simplifying retrieval and allowing the generation of custom zipped export packages. For large packages, the API generates a zip file and emails a download link, managed by the Tapis file API, to the user once it is completed, a functionality that is not fully available through existing Tapis files services alone. •Raster data support: The HCDP’s public-facing web application uses Angular and Leaflet to provide an interactive map interface for visualizing GeoTiff raster/gridded data. It efficiently retrieves data by querying the HCDP raster API endpoint by data type (e.g. temperature, precipitation, etc.), aggregation type (monthly or daily), and date. •Virtual station time-series: The HCDP also offers the ability to extract a time-series from a single raster grid cell (250Mx250M) near a provided latitude and longitude from a temporal series of GeoTiff files. This essentially established a virtual climate station for the particular variable of interest in monthly and daily aggregations for locations that previously had no time series resolution of data available due to the lack of a physical sensor station. 1) Data storage and management: The HCDP and the underlying Tapis framework utilize several databases: HCDP Backend Database: . •Tapis Metadata Service: This service uses MongoDB as its backend to store metadata in JSON documents along with geospatial indexing. This is crucial for identifying and retrieving data for the HCDP’s interactive map and other applications. •Tapis Security Kernel: This core security component leverages Postgres for general database needs and HashiCorp’s Vault as a secrets store to manage passwords and keys securely, ensuring fine-grained permissions and distributed security management. •File Storage: For raw data, derived data products and raster datasets the HCDP leverages the University of Hawai‘i’s KoaStore lustre filesystem. KoaStore provides high performance scale out storage that can be leveraged on the local HPC system or the regional Hawai‘i Jetstream2 VMs hosting HCDP services. The Tapis Files API is integrated into this storage system to provide easy access to public data hosted on the system. As of this publication the system has 4.9TB of raw, intermediate, and production data and has served over 1.9TB of data requested from API users via 3 million API calls. III. DISCUSSION: HCDP AS A HUB FOR INNOVATION AND RESILIENCE The HCDP’s design and implementation exemplify a modern science gateway, effectively translating complex science into actionable information for a diverse range of stakeholders. Its core functionality as an innovation and resilience hub is primarily driven by its robust API and the sophisticated tools it supports. A. API as a Core Enabler A fundamental aspect of the HCDP’s role as a science gateway is its API (Application Programming Interface). This API grants users programmatic access to the hosted climate data, allowing them to retrieve and download it. Such programmable access is crucial as it enables external applications and tools to directly integrate and utilize the HCDP’s datasets and real-time data streams. The API is leveraged to develop higher-level products and support decision-makers. This open access via a programmatic interface is a hallmark of a robust science gateway, fostering data-driven research and application development beyond the core project team. The HCDP’s underlying data processing pipeline also benefits from the API, particularly for managing large volumes of data and generating custom export packages, which were functionalities not fully available through existing Tapis services alone. B. Decision Support Tools The HCDP’s functionality and API are instrumental in supporting the creation and operation of several decision support tools, each designed to serve specific stakeholder groups by transforming complex climate data into actionable insights. •Hawai‘i Rangeland Information Portal (H-RIP): This portal specifically targets ranchers, providing rancher-relevant climate information by utilizing live rainfall and temperature data streamed from the HCDP. Co-developed with rancher feedback, H-RIP demonstrates how HCDP data can be repackaged into a tailored dashboard format to meet the specific needs of a key industry, aiding in rangeland management decisions. H-RIP includes features such as island selection, site submission, and displays statewide conditions, including an El Ni˜ no-Southern Oscillation (ENSO) gauge. In addition, the portal provides site-specific conditions, historically based rainfall outlooks, and forage production estimates. •Rainfall Atlas of Hawai‘i: The HCDP builds upon the legacy of the original Rainfall Atlas of Hawai‘i [11] website, which provided historical mean monthly and annual rainfall maps for the 1978-2007 [12], [13]base period, as well as month-year and trend maps (1920-2012). The HCDP now integrates these legacy climatologies as data products and has developed and hosts new contemporary climatologies (1991-2020) for rainfall and temperature, based on continuously updated NRT data. •Hawai‘i Climate Portfolio tool: This tool allows users to obtain site-specific climate portfolios detailing climate conditions within a user-defined geospatial area, directly leveraging data from the HCDP. Results are emailed to the user, highlighting the HCDP API’s flexibility in extracting relevant climate data for specific geographic locations and presenting it in a user-friendly format for planning and analysis. C. Example Use Cases The HCDP hosts a wide array of climate data, including extensive historical climate data and near-real-time (NRT) datasets, which are fundamental for decision-support tools. Here are examples for the two uses cases: 1) Climate Data Use Case for Planning and Analysis: A city planner, environmental consultant or researcher may need to understand long-term climate trends and current conditions to inform urban planning, infrastructure development, or environmental impact assessments. •User Action: A user, such as a city planner, could access the HCDP through its web portal. They may navigate to the Climate Maps section, where the HCDP hosts both legacy (1978–2007) and contemporary (1991–2020) climate maps for rainfall and temperature. They could also be interested in future projections of rainfall and temperature for different climate scenarios (RCP 4.5 & 8.5). •How the HCDP Ecosystem Addresses the Need: The planner can visually compare how rainfall or temperature patterns have shifted between the older ”Rainfall Atlas of Hawai‘i” data (1978-2007 base period) and the new contemporary climatologies (1991-2020). This allows them to identify significant climate shifts relevant for designing resilient infrastructure or developing water management strategies. For more granular, site-specific analysis, the user could utilize the Hawai‘i Climate Portfolio tool, which directly leverages HCDP data to provide customized climate portfolios for a user-defined area, with results emailed directly to them. 2) Rangeland Information Use Case for Ranchers: A rancher needs up-to-date and historical climate information specifically tailored for rangeland management, such as making informed decisions about grazing rotations, water resource allocation, or assessing fire risks. •User Action: The rancher would access the Hawai‘i Rangeland Information Portal (H-RIP). On the H-RIP homepage, they would select their site of interest on their ranch. •How HCDP Ecosystem Addresses the Need: H-RIP provides rancher-relevant climate information by utilizing live rainfall and temperature data streamed from the HCDP. For example, a rancher could monitor current rainfall conditions to decide if pastures are dry enough to require moving cattle or if supplemental water is needed. The portal also displays statewide conditions, including an El Ni˜ no-Southern Oscillation (ENSO) gauge, and leverages HCDP’s extensive historical records to provide context on broader environmental patterns that might influence rainfall and forage production in the coming months, which is vital for long-term strategic planning of ranch operations. This co-developed tool demonstrates how HCDP data is repackaged into a tailored, actionable dashboard for a key stakeholder group. D. Stakeholder Engagement The HCDP actively engages with its diverse stakeholder base through the HCDP User Group (HUG) and various public outreach activities. These engagements, including meetings, workshops, invited talks, media mentions, and community events, serve as platforms for sharing HCDP developments and, crucially, for gathering feedback to improve the portal and its supported tools. HUG participants represent a wide array of organizations, from state and federal agencies to agricultural interests and planning commissions. This direct interaction ensures that the data and tools developed are relevant and responsive to the actual needs of the people and organizations utilizing them, fostering a collaborative approach to adaptation and resilience. IV. CONCLUSION The Hawai‘i Climate Data Portal (HCDP) serves as a critical science gateway, data hub, and collaborative ecosystem for both the Change Hawai‘i (Change-HI) initiative and the broader state of Hawai‘i. Through its rigorous collection, processing, and hosting of a wide array of real-time and historical climate data, HCDP provides seamless access via robust APIs. This accessibility has been instrumental in powering the development of customized decision-support tools—effectively translating complex climate science into actionable insights for diverse stakeholders. By bridging data and decision-making, HCDP plays a vital role in advancing the vision of a resilient Hawai‘i. ACKNOWLEDGMENT This work was made possible by grant funding from the National Science Foundation (NSF) OIA-2149133 RII Hawai‘i EPSCoR: Change Hawai‘i Project; Tapis: Next Generation Software for Distributed Research NSF #1931439, #1931575; NSF OAC-2201428 CC* Compute: Koa - A High Performance and Flexible Research Computing Resource; NSF OAC-2232862 CC* Data Storage: KoaStore: A High Performance and Flexible Research Storage Resource; Jetstream2 at Indiana University through allocation ASC190026 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296 REFERENCES [1] R. J. Longman, M. P. Lucas, J. Mclean, S. B. Cleveland, K. Kodama, A. G. Frazier, K. Kamelamela, A. Schriber, M. Dodge, G. Jacobs, and T. W. Giambelluca, “The Hawai‘i Climate Data Portal (HCDP),” Bulletin of the American Meteorological Society, vol. -1, no. aop, Apr. 2024. [Online]. Available: https://journals.ametsoc.org/view/journals/bams/aop/BAMS-D-23-0188.1/BAMS-D-23-0188.1.xml [2] J. McLean, S. B. Cleveland, M. Dodge II, M. P. Lucas, R. J. Longman, T. W. Giambelluca, and G. A. Jacobs, “Building a portal for climate data — mapping automation, visualization, and dissemination,” Concurrency and Computation: Practice and Experience, vol. 33. [3] M. P. Lucas, R. J. Longman, T. W. Giambelluca, A. G. Frazier, J. Mclean, S. B. Cleveland, Y.-F. Huang, and J. Lee, “Optimizing automated kriging to improve spatial interpolation of monthly rainfall over complex terrain,” Journal of Hydrometeorology, vol. 23, no. 4, pp. 561 – 572, 2022. [Online]. Available: https://journals.ametsoc.org/view/journals/hydr/23/4/JHM-D-21-0171.1.xml [4] K. M. Kodama, E. Kourkchi, R. J. Longman, M. P. Lucas, S. M. Bateni, Y.-F. Huang, A. Kagawa-Viviani, J. Mclean, S. B. Cleveland, and T. W. Giambelluca, “Mapping daily air temperature over the hawaiian islands from 1990 to 2021 via an optimized piecewise linear regression technique,” Earth and Space Science, vol. 11, no. 1, p. e2023EA002851, 2024, e2023EA002851 2023EA002851. [Online]. Available: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2023EA002851 [5] D. Y. Hancock, J. Fischer, J. M. Lowe, W. Snapp-Childs, M. Pierce, S. Marru, J. E. Coulter, M. Vaughn, B. Beck, N. Merchant, E. Skidmore, and G. Jacobs, “Jetstream2: Accelerating cloud computing via jetstream,” in Practice and Experience in Advanced Research Computing, ser. PEARC ’21. New York, NY, USA: Association for Computing Machinery, 2021. [Online]. Available: https://doi.org/10.1145/3437359.3465565 [6] J. Stubbs, R. Cardone, M. Packard, A. Jamthe, S. Padhy, S. Terry, J. Looney, J. Meiring, S. Black, M. Dahan, S. Cleveland, and G. Jacobs, “Tapis: An api platform for reproducible, distributed computational research,” 2020, submitted. [7] S. B. Cleveland, R. Dooley, D. Perry, J. Stubbs, J. M. Fonner, and G. A. Jacobs, “Building science gateway infrastructure in the middle of the pacific and beyond,” in Proceedings of the Practice and Experience on Advanced Research Computing. ACM, Jul. 2018. [Online]. Available: https://doi.org/10.1145/3219104.3219151 [8] S. Cleveland, C. Arisdakessian, C. Nelson, M. Belcaid, K. Frank, and G. Jacobs, “The c-mundefinediki gateway: A modern science platform for analyzing microbiome data,” in Practice and Experience in Advanced Research Computing 2022: Revolutionary: Computing, Connections, You, ser. PEARC ’22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491418.3530291 [9] J. McLean, S. Cleveland, K. Rotzoll, S. Izuka, J. Leigh, G. Jacobs, and R. Theriot, “The ‘ike wai hawai‘i groundwater recharge tool.” Proceedings of Gateways 2019, 2019. [10] S. B. Cleveland, A. Jamthe, S. Padhy, J. Stubbs, M. Packard, J. Looney, S. Terry, R. Cardone, M. Dahan, and G. A. Jacobs, “Tapis api development with python: Best practices in scientific rest api implementation: Experience implementing a distributed stream api,” in Practice and Experience in Advanced Research Computing (PEARC ’20), 2020, pp. 1–10, provided as source . [11] T. W. Giambelluca, Q. Chen, A. G. Frazier, J. P. Price, Y.-L. Chen, P.-S. Chu, J. K. Eischeid, and D. M. Delparte, “Online rainfall atlas of hawai'i,” Bull. Amer. Meteor. Soc., vol. 94, pp. 313–316, 2013, cited in 361. [Online]. Available: https://www.soest.hawaii.edu/rainfall/ [12] A. G. Frazier, T. W. Giambelluca, H. F. Diaz, and H. L. Needham, “Comparison of geostatistical approaches to spatially interpolate month-year rainfall for the hawaiian islands,” International Journal of Climatology, vol. 36, no. 3, pp. 1459–1470, 2016. [Online]. Available: https://rmets.onlinelibrary.wiley.com/doi/abs/10.1002/joc.4437 [13] R. J. Longman, T. W. Giambelluca, M. A. Nullet, A. G. Frazier, K. Kodama, S. D. Crausbay, P. D. Krushelnycky, S. Cordell, M. Clark, A. Newman, and J. R. Arnold, “Compilation of climate data from heterogeneous networks across the hawaiian islands,” Scientific Data, vol. 5, no. 180012, February 2018.