OPeNDAP in the Cloud: Metadata Driving Performant, Interoperable, and Sustainable Geospatial Workflows
Abstract
Earth science data is growing rapidly, with NASA’s archives projected to exceed 400 petabytes, much of it stored in the cloud—even though many datasets were never designed for cloud-native access. Instead of rewriting massive files into new formats, OPeNDAP uses DMR++, a sidecar metadata file that maps a dataset’s internal structure and enables efficient server-side subsetting, web streaming without full downloads, and cloud-native virtualization through tools like VirtualiZarr and Kerchunk.
Full text
OPeNDAP in the Cloud: Metadata Driving Performant, Interoperable, And Sustainable Geospatial Workflows Miguel A. Jimenez-Urias ([email protected]) & James Gallagher ([email protected]) OPeNDAP Inc, Butte, Montana USA I. Motivation II. Background IV. Looking Ahead V. Summary & Conclusions •Cloud OPeNDAP server (Hyrax) produces DMR++, a metadata side car file that enables cloudperformant access to data and metadata. •DMR++ translation can enable interoperable, sustainable workflows. •Parsers are implemented in Python by VirtualiZarr and Pydap. •Virtualization can be exploited to form virtual datacubes, most appropriate for Level 3 or Level 4 data. •Level 2 data often has variable length dimensions, and attempting to create a datacube for such collections can be extremely challenging. •DMR++ exist available for many Level 2 collections, including HDF4 and HDF4-EOS collections. Streaming and de-serializing dap4 responses via HTTP can be very performant! VI. References [1] www.earthdata.nasa.gov [2] https://opendap.github.io/dap4-specification/DAP4.html. [3] https://opendap.github.io/DMRpp-wiki/DMRpp.html. [4] https://virtualizarr.readthedocs.io/en/stable/ [5] https://pydap.github.io/pydap/en/intro.html. [6] Tempo L2 NO2 tropospheric and stratospheric columns V04 DOI: 10.5067/IS-40e/TEMPO/NO2_L2.003 [7] CALIPSO Lidar Level 2 1 km Cloud Layer, V5-00: DOI:10.5067/CALIOP/CALIPSO/CAL_LID_L2_01kmCLay-Standard-V5-00 VII. Acknowledgements Poster ID: IN21D-0361 III. Results: Metadata-driven workflows Poster ID: IN21D-0361 •Distributed translator over http get requests relies on ”agreement” about what data is. APIs that implement DAP protocol What is OPeNDAP? Cloud OPeNDAP? Interoperability for Cloud OPeNDAP? •Works with geospatial, self-describing, chunked, hierarchical data formats •Server-client model that implements DAP protocols (see Fig. 1) •Enables data-proximate subsetting, reducing data transfer, times, and download cost. •Configured to scale with demand. •Superchunk algorithms for contiguous small chunks of data in non “cloud optimized” data format (HDF5, HDF4). •Uses sidecar metadata files DMR++ for efficient server-side subsetting. Fig.3 Anatomy of a DMR++ file: OPeNDAP’s generated metadata reference file that enables performant, cloud workflows What is a DMR++? Sidecar metadata reference file that declares internal structure within files. For example: Attributes: description, user-defined Variables: Name, dimension, type, size Containers: Groups, Sequences, Structures Metadata interoperability Language agnostic Chunk Manifests: Compression, bytes, offsets, ChunkPositionInArray, Endianness, deflateLevel, file location OPeNDAP in Action Query NASA’s Common Metadata Repository (CMR) Earthdata Login (EDL) authentication Filter (variables names, dimension slices, …) DMR++ can be translated to zarr metadata Widely used to create virtual DataCubes •Works best for Level 3 / Level 4 data ML-accelerated server-side subsetting •Growing data archives in formats not designed for cloud access •Cloud data migration •Heterogeneous geospatial data not fitted for Virtualization Houston, we have a Big Data Problem! Fig.1 DAP protocols and data they can represent. [2] Fig.2 Data in S3 and the reference metadata file that can enable performant cloud workflows. [3] Fig.5 Benchmarks for two different “In Cloud” collections from NASA’s ASDC: TEMPO Near Real Time L2 NO2 [6], and Calipso’s Lidar Level 2 [7]. For both collections, data was downloaded via http (non-cloud access). In all tests, the data for each collection is spread across N=100 granules. STEPS: 1. 2. 3. 4. Stream data with Pydap! [5] Fig.6 Example DMR++ for a Level 2 (swath) data) Data archives ~ 128 Pb in 2024 Data archives projected to surpass 400 Pb by 2028 Cloud-performant Fig.4 Snippet of code showing how to stream data locally. The method “stream_parallel” lies within Pydap. It streams a serialized DAP4 response into a local file as NetCDF4. source: [1] Server-side subsetting (DAP4) only works by index value (e.g. [0:1:100]) •Does not support subsetting by coordinate value. •None scalable approach (e.g. IceSAT2) DMR++ contains information about the coordinates. •Knowledge of swaths trajectory ML (supervised) training on real data to predict “hyperlab slice” given desired coordinate value subsetting for rest of collection. •Coordinate endpoints of swath •For level 2 data, better to use OPeNDAP directly •DAP4 reponse is streamable! Virtualization Approach: OPeNDAP development has been supported by NASA. Benchmarks for this poster were performed in Johns Hopkins University’s Sciserver, an effort by JHU’s IDIES.