For the last couple of years, working at Mesh-AI, we’ve worked closely with a number of organisations embarking on a journey to transform their data, hoping to achieve more value with analytics and AI.
The emergence of data mesh has been a catalyst for that, setting the foundation for what future data-driven organisations could look like. As for any new major idea, data mesh generated a lot of interest, opinions, debates and, let’s admit it, hype.
Let’s remind ourselves what data mesh is and why it’s so important. A data mesh approach allows your organisation to develop highly decentralised data architecture and a federated operating model around it, making access to essential data more seamless whatever the use case.
Data mesh allows you to streamline practices and empower business domains to be better data owners, so you can foster a data-driven approach organisation-wide.
Tangibly, federated models of data governance support compliance and regulatory requirements without compromising on innovation, while ensuring data is trustworthy, high in quality and accessible to all areas of the business.
Practically speaking, it’s fair to say that the adoption of data mesh hasn’t been a walk in the park for many organisations. And how could it be? Most large and complex organisations have been operating in functional silos and have been treating their data as a technical byproduct and a liability for a number of decades now. These types of environments that would benefit the most from adopting data mesh are, paradoxically, where data mesh adoption is going to be the hardest.
What Does Successful Data Mesh Look Like?
We’ve seen great success in implementing data mesh at some of the world's most highly-regulated enterprise organisations across multiple industries, transforming how they view and value data. We’ve helped a global financial institution reduce the risk of tens of millions in fines, by complying with new regulation and strengthening operational resilience, by democratising their data. We’ve helped a global energy supplier save millions of pounds in operational efficiencies by using their operational data to predictively maintain and better operate physical energy infrastructure.
However, to understand what a realistic adoption of data mesh looks like from the inside, it’s important to understand that this is far from being primarily a technology-based, cookie cutter exercise. Data mesh is not just a technology upgrade to an existing data platform, even through the acquisition of top-end new tools.
In addition to the obvious technical uplift that most organisations are happy to embrace, the change needs to go much deeper. To generate new value from data, we need a truly modern approach to data platforms that enables quick and scalable innovation. We need to elevate data to a main source of value through product thinking.
We need to flip the success criteria of data producers on its head, by focusing on measurable business value derived from data (and pivot away from the focus on data ingestion and storage as a measure of success). All this adds up to a deep transformation where IT and the rest of the business need to be fully aligned to deliver if data, analytics and AI are to deliver tangible outcomes.
What Lessons Have We Learned From This?
Reassuringly, we have found from our engagement with a broad range of enterprise organisations that most have identified and understood the core ideas behind data mesh and the value that a modern approach to data can deliver. Many of these enterprises have not necessarily identified the implications, cost and complexity associated with such a deep transformation of the way data is treated and the way IT interacts with the rest of the business. This usually manifests itself through constant friction between different parts of the business and a lack of alignment on the approach and priorities.
For a data mesh approach to work, it requires changes to the inherent structure of the business itself:
However, the department, be it technology or centralised IT, that understands the necessity and the value of a data mesh approach – because they’ve been the organisation’s data custodians for years – is traditionally seen as a service provider to the organisation, not always a value creator. This means they’re not in a position to instigate or drive this transformation at scale and without the collaboration of other executives.
Given these challenges behind full scape adoption, we have seen a number of technology-centric variations emerge. It is understandable that ideas evolve and people tend to look for practical approaches but I still have my doubts about the long term impact and viability of any approach that doesn’t address organisational challenges at their root cause.
These technology-centric approaches have fitted in two common categories. Firstly, data fabric, which we can oversimplify as ‘an integrated layer of data and connecting processes.’ This is the concept that data needs to be interconnected in a well defined layer, rather than all over the place.
The second tech-centric approach we have observed involves the localised application of product thinking to centralised data models. As an example, this could take the form of making certain data more accessible to the particular users who need to consume it, but still within a centralised framework such as a data warehouse model.
I’m all for increased investment in modern data infrastructure and for the introduction of product thinking to data as it can instigate a transition towards a more modern approach. However, neither approach tackles the broader challenges that organisations aspiring to be data-driven need to overcome. These tech-centric approaches suffer from unclear definitions and are often pitted as an alternative to a data mesh approach, when in fact they’re simply tactics of data-driven transformation.
How Enterprises Are Benefitting From Data Mesh
Embracing product thinking and extending it to data is a great idea. We should apply product thinking to data (through data products) but also to any analytical capability we build to benefit the broader system.
There is no alternative to unifying IT and the business if we want to produce value from data through sophisticated capabilities that go beyond operational reporting. This should be through investment in data-driven valuable outcomes that allows to shape a modern operating model, rather than big bang transformation.
To enable any of that we need a modern, scalable, self-service (data) platform. A modern platform should be multi-tenanted by design, allowing teams to experiment quickly and build fit-for-purpose solutions, as opposed to shoehorning new use cases into pre-existing and jaded data warehouse models.
Finally, the desire to do more and go quickly will require organisations to allow their teams to operate in a more distributed and autonomous way. This decentralisation does not equate to chaos. If a federated operating model is put in place allowing teams to agree on what matters (e.g. data interoperability) while maintaining their autonomy when it comes to what is not important for other parts of the organisation (e.g. implementation details). A key aspect of data mesh is a non centralised approach to data (non centralised as a technology but also as an operating model). Data is naturally decentralised. We should embrace this rather than fight it.
The Enterprise Data Leader's Handbook is an essential resource for those in charge of data within enterprise businesses, get your copy here.
Here's five ways to overcome complex data challenges.
Find out how data mesh accelerates delivery by reducing waste.
