Data Infrastructures Paradigms

DataFlow White paper Data Infrastructures Paradigms Bram Ton1, ID and Deepak Tunuguntla1, ID 1Ambient Intelligence group, Saxion University of Applied Sciences, The Netherlands Published 10 June 2025 DOI 10.5281/zenodo.15585260 Abstract In the era of Industry 4.0, data has become a critical driver of innovation and competitiveness in manufacturing. The up-take of technologies such as IoT, AI, and robotics has led to an unprecedented explosion of data, offering transformative opportunities for process optimization, product quality enhancement, predictive maintenance, and customisation. However, realizing these benefits requires effective data collection, storage, and processing, which necessitates robust and adaptable IT infrastructures. This white paper provides a comprehensive overview of four popular data infrastructure paradigms: data lakes, data fabrics, data meshes, and data spaces. Each of these paradigms have their unique focus points and application domains. Metadata serves as a crucial component, enhancing the findability, accessibility, interoperability, and reusability of data. The paper also explores critical building blocks of modern data infrastructures within manufacturing environments, such as OPC UA, brokers, and data catalogues, highlighting their role in enabling effective data sharing, management, and governance. 1Introduction Industry 4.0 represents the fourth industrial revolution, characterized by the integration of advanced digital technologies into manufacturing and industrial processes [1]. This digital integration is an enabling factor to to collect vast amounts of data about indus‐ trial processes. This wealth of data offers transformative opportunities: processes can be optimized, product quality enhanced, maintenance predicted and planned, and cus‐ tomization options expanded –just to name a few. However, before these benefits can be realized, organizations must address a fundamental challenge: how to effectively col‐ lect, store, and process these data. This requires the design and deployment of robust and adaptable data architectures. The journey to selecting the right data architecture can be daunting, as the landscape is populated with numerous paradigms. This white paper aims to provide a concise yet comprehensive overview of four popular paradigms within the domain of data infras‐ tructures: data lakes, data fabrics, data meshes, and data spaces. To be fair, this is not the first work to attempt to summarise this vast landscape. Others have also attempted to navigate the jungle of different data architectures [2]. Among the four paradigms discussed, data lakes, data fabrics, and data meshes are three paradigms particularly focused on enabling intra‐business data solutions. These ap‐ proaches emphasize scalability, interoperability, and the ability to manage large, hetero‐ geneous datasets across organizational boundaries. The fourth paradigm, data spaces, on the other hand focuses on inter‐business data sharing. An essential component common to all these paradigms is metadata. Metadata serves as the backbone of any modern data infrastructure by enhancing the findability, acces‐ sibility, interoperability, and reusability of data. Without a well‐structured approach to metadata management, organizations risk underutilizing their data assets, thereby limiting the value they can derive from their investments. In addition to exploring these paradigms, this white paper also delves into the critical building blocks of modern data infrastructures within manufacturing environments, such as OPC UA, brokers, and data catalogues, which are discussed in Section 3. Copyright © 2025 B. Ton et al., released under a Creative Commons Attribution 4.0 International license. Correspondence should be addressed to Bram Ton ([email protected]) The authors have declared that no competing interests exist. Ton et al. 2025 1 Data Infrastructures Paradigms 1.1 DataFlow project This white paper originates from one of the key questions of the DataFlow project. The consortium of this project consists of several small and medium‐sized enterprises (SMEs) within the region of Twente, The Netherlands, and is co‐funded by the TechForFuture centre of expertise. The DataFlow project addresses the challenge of leveraging data to enhance industrial processes, enabling the transition of SMEs towards the era of In‐ dustry 4.0. Data is a critical resource for improving existing processes, such as qual‐ ity control through automated computer vision and optimizing maintenance schedules. Moreover, data can pave the way for innovative business models, like transitioning to as‐a‐service practices. However, data only holds value when it is usable and compre‐ hensible, highlighting a significant barrier to adopting data‐driven approaches in the industry. DataFlow tackles the challenge of adoption and integration of data into industrial busi‐ nesses from a methodological, technological, and practical view. The project aims to enhance the data readiness of the involved partners by providing methods to organize data‐driven work, best practices for tools and frameworks to implement these methods, and learning from real‐world cases. These cases involve industrial partners addressing problems through prototyping and validation experiments to refine the project’s meth‐ ods and best practices. Additionally, laboratory experiments at Saxion will complement these industrial cases, offering more controlled environments to further develop and improve applied methods and best practices. Ultimately, DataFlow aims to empower SMEs to develop successful data‐driven inno‐ vations. The goal is to integrate these innovations with existing physical products and support industrial partners of the project with robust business cases, ensuring that data‐ driven approaches are not only implemented but also sustainable and beneficial for in‐ dustrial growth. 2Paradigms This section provides an overview of four distinct paradigms in data management and architecture: data lakes, data meshes, data fabric, and data spaces. Each approach ad‐ dresses unique challenges related to handling, sharing, and governing data, emerging as responses to the increasing complexity and volume of data in the modern era. 2.1 Data lake A data lake is a centralised repository designed for storing diverse data in its raw format, offering scalability for big data analytics [3]. Also the term monolithic data architecture is sometimes used to describe a data lake. Data lakes often operate on a schema-on-read principle, where the data structure is defined when the data is queried rather than when it is stored. This approach provides flexibility and supports various data types, including structured, semi‐structured, and unstructured data. A formal definition of a data lake is provided below. A data lake is a scalable storage and analysis system for data of any type, retained in their native format and used mainly by data specialists (statisti‐ cians, data scientists or analysts) for knowledge extraction. (Sawadogo and Darmont, 2020) Effective metadata management is crucial to prevent data lakes from becoming unman‐ ageable ‘data swamps’. Section 3.3 goes into more detail about effective metadata man‐ agement solutions. Ton et al. 2025 2 Data Infrastructures Paradigms 2.2 Data mesh The data mesh represents a decentralised, socio‐technical approach to data manage‐ ment, contrasting with the centralised model of data lakes by adopting a domain‐oriented data ownership approach. This decentralisation aims to enhance data quality by using domain expertise and treating data as a product, rather than a by‐product. A formal definition of a data mesh is: Data mesh, at its core, is founded in decentralisation and distribution of re‐ sponsibility to people who are closest to the data in order to support contin‐ uous change and scalability. (Dehghani, 2020) The data mesh has four key principles that guide its implementation, each of these core principles is briefly described below. Domain‐oriented data ownership means that individual business domains own the data they produce and use their domain expertise to improve data quality. Data as a product means that data is treated as a valuable product with end‐to‐end re‐ sponsibility, ensuring that it is discoverable, addressable, understandable, trust‐ worthy, accessible, interoperable, valuable, and secure. Self‐serve data platform provides the necessary infrastructure and high‐level abstrac‐ tions to enable domains to work autonomously, without needing to duplicate tech‐ nical efforts. Federated computational governance Despite giving individual teams a lot of freedom, there still remains certain data governance aspects which should be managed on a business wide scale. This is were the computational governance part comes into play, this principle states that business wide governance should be enforced in a automated way. 2.3 Data fabric This seems to be the most loosely defined concept of the four and is really focused on providing a unified entry point for all types of data sources. A formal definition is given below: The data fabric concept is concerned with the more effective integration of heterogenous and isolated data sources so that data provision in organiza‐ tions can be improved. (Blohm et al.,2024) 2.4 Data space A data space is a concept that enables decentralised data sharing between industry part‐ ners, ensuring that data remains at its source and avoiding the need for centralised storage. The International Data Spaces (IDS) initiative has developed a standardised architecture for secure and trusted data exchange across diverse platforms and across company boundaries. A formal definition of the IDS is given below. International Data Spaces (IDS) enable the sovereign and self‐determined exchange of data via a standardized connection across company boundaries. (Pettenpohl, Spiekermann, and Both,2022) Some common architectural components of data spaces are: Connectors are software components that act as a secure interface between a data provider’s internal systems and the data space, ensuring data sovereignty and controlled ac‐ cess Ton et al. 2025 3 Data Infrastructures Paradigms Brokers facilitate the discovery of data sources and their associated usage policies, act‐ ing as a directory service for the data space App Store enables the development and distribution of software and data services that can be used within the data space Identity Provider manages the identities of participants and ensures secure access to data by validating and authenticating users and connectors Clearing House supervises and records data exchange transactions, ensuring that data exchange is compliant and providing a mechanism for reversing transactions if necessary Vocabulary Provider offers a standardised set of vocabularies, ontologies, and metadata elements for describing data, enabling interoperability across the data space 2.5 Summary The main take‐away message here is that meta‐data is paramount when dealing with any data infrastructure paradigm. 3Components Several components commonly present in data architectures are discussed below. First, the Open Platform Communications Unified Architecture (OPC UA) standard is discussed. This standard plays a crucial role towards interoperability within industrial automatisa‐ tion. Secondly, it briefly presents the crucial role of data brokers within the ecosystem. And lastly, it details the pivotal role of data catalogues within the data ecosystem. 3.1 OPC Unified Architecture Within industrial automation, effective communication is crucial for operations. A widely adopted standard for sharing information is OPC UA [6], which is supported by a vast ma‐ jority of vendors, ensuring broad compatibility and integration across diverse systems. OPC UA provides two fundamental concepts to enable interoperability: it defines both how systems communicate and what is communicated. This standard facilitates data exchange through various features, including alarms, historical data access, addressing, and security. A notable extension of OPC UA is the OPC UA Field eXchange (UAFX), which enhances communication capabilities at the field level. UAFX ensures efficient and reliable data transmission between field devices, further strengthening the interconnectedness and responsiveness of industrial automation systems. An alternative to OPC UA is Web Object Oriented Protocol for Software and Automation (Woopsa) 1. Though, it does not have an extensive proliferation as OPC UA and it seems not very actively developed any more. 3.2 Brokers Data brokers form the backbone of the majority of the data collection initiatives. They are considered middleware as they sit between the locations where data is generated and where data is stored. Two popular protocols are Advanced Message Queuing Protocol (AMQP) and Message Queuing Telemetry Transport (MQTT). OPC UA has mappings for both these protocols, which means that messages from these protocols can be embedded within OPC UA messages. 1https://woopsa.org Ton et al. 2025 4 Data Infrastructures Paradigms 3.3 Data catalogues Cataloguing data is essential for governance, findability, and data lineage. Each of these elements is detailed in the sections below. Findability — Findability in data architectures refers to the ease with which users can lo‐ cate and access relevant data within an organization’s data ecosystem. Effective find‐ ability is crucial for enhancing data utilization and driving informed decision‐making, as it allows data consumers to quickly discover the data they need without extensive searching. Implementing robust metadata management and search capabilities within data catalogues significantly improves findability, making data assets more accessible and valuable. Without proper findability, there is a risk of creating data swamps. Data governance — Data governance involves the policies, procedures, and standards that ensure data is managed consistently, securely, and in compliance with relevant regula‐ tions. It is essential for maintaining data quality, integrity, and security, thereby foster‐ ing trust in the data and mitigating risks associated with data misuse. Data catalogues play a pivotal role in data governance by providing a centralized repository for metadata, enabling better control and oversight of data assets across the organization. Data lineage — Data lineage tracks the journey of data from its origin to its current state, capturing all transformations and movements along the way. It provides visibility into data flows, aiding in troubleshooting, impact analysis, and compliance verification. By offering a clear view of data lineage, data catalogues help organizations understand the provenance of their data, ensuring transparency and accountability in data man‐ agement processes. Open-source catalogues — There is a wide variety of data cataloguing frameworks available for enterprises. A small selection of open‐source data catalogues are listed below as an example. • Amundsen (https://www.amundsen.io) • Apache ATLAS (https://atlas.apache.org) • DataHub (https://datahub.com) • Marquez (https://marquezproject.ai) To highlight a few cherry picked capabilities of data catalogues: Apache ATLAS has ex‐ tensive built‐in governance capabilities. Marquez offers detailed visualisations of data lineage, showing how data is transformed during its lifetime. DataHub and Amundsen provide a social element to their data catalogues, showing which users are using which data and what queries are the most popular. Acknowledgements The DataFlow project is co‐funded by TechForFuture. The template for this document has been copied from the ReScience C project. Ton et al. 2025 5 Data Infrastructures Paradigms Declaration of the use of Generative AI During the preparation of this work the author(s) used ChatGPT and mistral.ai in order to improve the readability and flow of the text. After using these tools/services, the au‐ thor(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication. References 1. L. D. Xu, E. L. Xu, and L. Li. “Industry 4.0: State of the Art and Future Trends.” In: International Journal of Production Research 56.8 (Apr. 18, 2018), pp. 2941–2962. DOI: 10.1080/00207543.2018.1444806. 2. T. Priebe, S. Neumaier, and S. Markus. “Finding Your Way Through the Jungle of Big Data Architectures.” In: 2021 IEEE International Conference on Big Data (Big Data). 2021 IEEE International Conference on Big Data (Big Data). Orlando, FL, USA: IEEE, Dec. 15, 2021, pp. 5994–5996. DOI: 10.1109/BigData52589.2021.9671862. (Visited on 01/21/2025). 3. P. Sawadogo and J. Darmont. “On Data Lake Architectures and Metadata Management.” In: Journal of Intelligent Information Systems 56.1 (2020), pp. 97–120. DOI: 10.1007/s10844-020-00608-7. 4. I. Blohm, F. Wortmann, C. Legner, and F. Köbler. “Data Products, Data Mesh, and Data Fabric: New Paradigm(s) for Data and Analytics?” In: Business & Information Systems Engineering 66.5 (Oct. 2024), pp. 643–652. DOI: 10.1007/s12599-024-00876-5. 5. H. Pettenpohl, M. Spiekermann, and J. R. Both. “International Data Spaces in a Nutshell.” In: Designing Data Spaces. Ed. by B. Otto, M. Ten Hompel, and S. Wrobel. Cham: Springer International Publishing, 2022, pp. 29– 40. DOI: 10.1007/978-3-030-93975-5_3. 6. W. Mahnke and S.-H. Leitner. “OPC Unified Architecture: The Future Standard for Communication and Information Modeling in Automation.” In: ABB Review 3 (2009), pp. 56–61. Ton et al. 2025 6