Data Mesh describes the way companies manage and use their data. As an advanced data architecture concept, a data mesh aims to overcome the challenges of centralised data structures and create a decentralised, agile data landscape. It enables the connection of data owners, data producers and data consumers to improve information exchange and make data-driven processes more efficient. In doing so, a data mesh views data as valuable products that are managed independently by the respective domain experts and made available to other teams. But how exactly does this concept work, what principles underlie it and what are the advantages and disadvantages of implementing it? This article will provide a comprehensive insight into the world of Data Mesh and illuminate how companies can benefit from this groundbreaking data architecture.

What is a data mesh?

Data Mesh describes a concept for the Data architecture in companies, which aims to improve the Decentralise data management and improve data-driven processes. The aim is to connect the data owner, the data producer and the data consumer. According to its founder Zhamak Dehghani, the data mesh concept should primarily address those challenges where centralised and monolithic data structures reach their limits. This applies above all to the organisation and accessibility of the data. With the data mesh approach Data as products and the consumers of this data should be treated as customers. The principle of viewing data as products aims to solve the problems of the Data quality and legacy data silos, also known as "dark data". Dark data is the information that organisations collect, process and store as part of their regular business activities, but generally do not use for other purposes.

What are the 4 principles of the concept?

The Data Mesh concept is based on the following 4 principles:

1. Domain ownershipThe data is organised in a data mesh in so-called domains, each of which corresponds to a specific business area in a company. The teams within these domains are responsible for the management, quality assurance and release of their data themselves through domain experts. This creates decentralised data ownership, which increases agility and flexibility.
2. Data as a product: Data Mesh treats data as products, which are created, maintained and made available to internal or external users by the aforementioned domain experts according to defined roles. This means that data producers and data consumers work directly together, similar to a product development team, for example.
3. Self-service data platformsThe concept promotes the creation of so-called "self-service platforms" that enable data-owning teams to share their data using standardised APIs and interfaces easily shared and accessible. This facilitates collaboration between teams and reduces dependency on centralised Data platforms reduced. In addition, this approach supports data integration, quality assurance and the ability to analyse the data.
4. Federated computational governanceData Mesh promotes a decentralised Data governance structure in which each domain team has authority over its own data and data products, ensuring that privacy, security and compliance are maintained without restricting the autonomy of the data-owning teams. However, there are also certain overarching governance policies and standards that are set by a central body or data-focused community.

What are the advantages and disadvantages of a data mesh architecture?

advantages:

• Scalability or cost efficiency: The distributed architecture of a data mesh relies on cloud data platforms and streaming pipelines for real-time data collection, rather than batch data processing. Cloud storage offers a cost advantage as data teams assemble resources as needed and only pay for the storage they use. The flexibility allows additional computing power to be added as needed.
• Data quality: The teams' responsibility for their data leads to higher data quality as they have specific domain knowledge.
• Democratisation of data: By simplifying self-service applications from multiple data sources, data mesh architectures facilitate access to data beyond technical resources such as data scientists, data engineers and developers. This domain-oriented design reduces data silos and operational bottlenecks, enabling faster decision-making and allowing technical users to better utilise their skills.
• Reduction of technical debt: Centralised data infrastructures often create so-called technical debt due to complexity and the need for maintenance collaboration. By distributing the data pipeline by domain ownership, data teams can better respond to the needs of their data consumers and reduce the burden on the storage system.
• Interoperability: Data Mesh models promote the standardisation of data fields across domains, facilitating interoperability. This consistency enables easy data linkages and the development of applications that better meet business needs.
• Security and complianceData mesh architectures support stronger governance practices by enforcing data standards and access controls for sensitive data. This ensures compliance with government regulations and enables data audits.

Cloud computing enables companies to use their IT resources more flexibly and cost-efficiently. The cloud architecture and infrastructure play a central role in this. Find out which aspects you need to pay attention to in our blog post:

Cloud Computing - Architecture & Infrastructure: Compactly explained

Disadvantages and challenges:

• ComplexityThe decentralised data management of a data mesh can lead to increased complexity, especially if not enough attention is paid to data integration, data protection and security. The integration of different domains, data sources and pipelines can be complex and may require extensive changes to existing data processes.
• Increased governance challenges: With data mesh, data responsibilities are spread across different domain teams. This can complicate governance and data quality as control and responsibility for data is split between teams. It can be difficult to establish consistent standards and policies across different teams, leading to inconsistencies, ambiguities in data interpretation and possibly security breaches.
• Overhead through coordination and communication: Because data mesh relies on decentralised data responsibilities, the individual domain teams need to cooperate and communicate more to efficiently develop and manage data products and pipelines. This increased coordination effort can lead to additional overhead and lost time, especially when teams are spread across different locations or time zones.
• Cultural changesThe introduction of a data mesh requires a cultural change in a company, as it means a shift from centralised decision-making to more autonomy of the teams. In addition, the shift from a centralised approach to a decentralised data mesh usually also involves implementation costs as well as time.

In a data-driven world, Data Fabric transcends traditional boundaries. Learn how optimised data flow opens up new business opportunities.

Data Fabric: Optimising the data ecosystem for businesses

What is the difference to Data Fabric?

Data Mesh and Data Fabric describe two approaches to data architecture, but they have different emphases. While data mesh focuses on decentralised data management and the autonomy of data-owning teams, and it aims to view data as products and promotes self-service capability, a Data Fabric on the other hand, an integrated data approach that seamlessly connects a company's various data stores, data sources and data processing technologies. It Emphasises the uniformity and consistency of data accesses and transformations and strives for central data control to provide a unified view of the data.

In relation to Data security With a data mesh, the responsibility lies with the individual teams, whereas a data fabric is a centralised Data security enables. Data Mesh emphasises team ownership of data governance, while Data Fabric can embrace centralised data governance. Data Mesh is suitable for complex and scaling data landscapes, while Data Fabric is designed to facilitate the end-to-end connection and processing of large amounts of data across disparate systems.

Despite the different focus of data mesh and data fabric, the two approaches can be combined to develop an end-to-end data strategy and generate benefits from both approaches. One possibility is to implement a data fabric as the basic data infrastructure on which the data mesh concept is based. This provides a unified view of the data, enables data integration across different systems and supports the scalability of the data infrastructure. Thus, the teams in the data mesh have a solid foundation to access high-quality and integrated data and do not need to worry about the technical aspects of data integration. An alternative approach is to implement parts of the data mesh into the data fabric strategy. In concrete terms, this means that the responsibility for the data is not only distributed to central units, but also to the individual teams in the data fabric. Each team becomes a so-called "data product owner" for the data it manages. This approach reinforces decentralised responsibility and collaboration as defined by the data mesh concept. At the same time, the data fabric ensures the infrastructure so that data integration, data quality and data governance are consistent and efficient across all teams.

Data Mesh or Data Fabric: Learn which approach is right for your organisation to ensure effective data management and integration.

Data Mesh vs. Data Fabric: A Comparison of Data Management Concepts

What is the difference to the Data Lake?

Similar to a data fabric, a Data Lake an approach to data architecture that differs from a data fabric or a data mesh, but also has some similarities. A Data Lake is a central storagewhich ingests a large amount of unstructured and structured data from various sources. It provides a cost-effective way to store data before it is analysed or loaded into other systems. In a data lake, data can be easily merged and analysed, making it a valuable tool for Big Data analyses.

In contrast, a data mesh is decentralised because it distributes responsibility for the data among the data-owning teams in the domains. Each team is responsible for managing its own data and makes it available to other teams via standardised interfaces. This achieves tighter integration between the business units and the data itself, which increases agility and flexibility.

Although a data mesh and a data lake (as well as a data fabric) are different approaches, they can be combined in some situations. For example, a data lake could serve as a foundation on which the principles of data mesh or data fabric are applied to enable decentralised data responsibility or a unified data infrastructure. Alternatively, a data lake could act as a central data source serving different domains. Even within a data mesh, individual teams and domains can generate their own data lakes to organise their data.

Discover the advantages of a Data Lake: The central storage solution for large amounts of data from various sources, enabling effective analysis and use.

Basics, use cases and benefits of a data lake: Everything companies need to know about data lakes

What data mesh solutions exist?

There are various solutions and tools that support companies in successfully using a data mesh:

• Amazon Web Services (AWS): AWS provides several tools and services that can help implement a data mesh. These include Amazon S3 for storing data, Amazon Glue for data integration and transformation, and Amazon Athena for querying data.
• Microsoft AzureAzure also offers a range of tools to support data mesh architectures. These include, for example, Azure Data Factory for data integration and transformation and Azure Synapse Analytics for querying data. Microsoft also provides Azure Data Lake Storage, a service for storing data. It supports the integration of Azure services and third-party tools to ensure seamless data movement and processing.
• IBM: With IBM Data Fabric on Cloud Pak for Data, IBM delivers an integrated data and AI platform that provides tools for data storage, integration and analytics, creating true self-service of enterprise-level data products.
• TalendTalend describes a provider of data integration and data quality solutions, which with its Data Catalog provides support for data mesh architectures. With this tool it is possible to create a data mesh and to share and manage the data, among other things.

How do I implement a data mesh in my company?

The introduction of a data mesh requires careful planning and a step-by-step implementation. The following is the Standard flow of the implementation process of a data mesh described in a company:

1. Define data strategy or identify data domainsThe first step should be to define clear goals and strategies for the data landscape. For this, it is also useful, among other things, to identify and describe domain experts and the exact areas of responsibility.
2. Organisational changesA change in data architecture is always accompanied by a change in the culture of cooperation in terms of decentralised data responsibility. For this reason, employees should be trained in their new roles and responsibilities.
3. Technological implementation: The technical implementation takes place within the framework of the implementation of the self-service platform, which enables the individual teams to independently create and manage their data products in the data architecture of the data mesh.
4. Promoting federated data governance and security measuresWhen making the transition (especially from a centralised data architecture), it is important to note that a data mesh requires federated governance, where responsibility for data management is shared between the different data domains. This means that each team is responsible for the quality of and access to its own data. This mindset should be encouraged accordingly.
5. Monitoring and evaluation: By monitoring and evaluating the benefits of the data mesh, structures and processes can be adapted and optimised.

Conclusion

A data mesh is a decentralised data architecture concept that aims to improve data management. It connects data owners, producers and consumers by viewing data as products and promoting self-service. Due to the advantages of a data mesh, such as good scalability, democratisation of data, reduction of technical debt or interoperability, this decentralised data architecture can bring great benefits to companies. In combinations with similar approaches such as a data fabric or a data lake, companies can improve their data management, promote collaboration between teams and benefit from the advantages of a decentralised data architecture.