It’s not exactly a secret that the lack of access to high quality data is holding back enterprises from achieving their data-driven goals, such as AI adoption. In our recent State of AI in the Enterprise report, data quality was the top cited barrier to AI adoption. We’ve become so accustomed to this state of affairs that we don’t even notice a simple paradox: while these organisations are constantly generating massive amounts of data they are unable to access their own data, for reasons that are clearly not technology related.
I’m yet to encounter a CIO, CTO or CDO lamenting “if only we upgrade our data platform, all our problems will be resolved.” People know the problem is not technical yet curiously, on the ground, the go-to remedy for poor quality technical data is often technology-centric. New platforms and data hoarding are the typical answer to data problems!
During our work at Mesh-AI we are constantly faced with this problem and we are on the hook to make a tangible impact and quickly. We always approach this by addressing the root cause of data quality issues and we often have to approach it in innovative ways to circumvent the constraints of the specific situation we’re dealing with.
I would like to share here some of the real world examples of the work our brilliant engineers and consultants have done and some of the learnings we’ve earned from the coal face. You can find these insights and more on our Engineering & Technology Medium site.
Principal Data Consultant Dan Ward gives his insight on the importance of data contracts to extracting the right value from data products.
"Have you ever built flat pack furniture before? If you opened the box and got the parts, but no instructions, would you know if all the right parts were there? When you start trying to fit pieces together, how would you know if you were building it correctly? Instructions are to flat pack furniture as Data Contracts are to Data Products; they provide infallible information of the thing they are describing and, if anything is awry, then it is the flat pack kit (Data Product) you have received that is wrong, not the instructions (Data Contract). This creates a guarantee of services which, in turn, generates trust in the product."
The popularity of event-driven data pipelines is the subject of this piece from Eric Papaluca, Principal Data Consultant. He explores some of the challenges to this approach, its impact on regulatory requirements and technical debugging, and how to implement it.
"As organisations modernise their data estates, a trend is to move towards event-driven data pipelines for their myriad of benefits. However, like all things in life — there are drawbacks and pitfalls to any approach and in the cases of event-driven pipelines, these can largely be attributed to a lack of observability over the system state, resulting in the follow-on effects in terms of technical debugging and most importantly, regulatory requirements."
Senior Data Consultant Dan Harrington explores how to solve data quality issues–the greatest technical barrier to AI adoption–to generative meaningful insights from data and harness it to its full ability.
"In this article, I will be exploring one of the main blockers that prevent teams from getting from Raw data to the utopia of deployed ML models giving accurate and timely insights: Data Quality. Further to this, I will be demonstrating how one of the most revolutionary technologies of our generation can massively reduce the headache surrounding this divisive subject and accelerate your path from data discovery to business value."
In this two part series, Lead Data Analyst Tom Warburton discusses how to assess the quality of data and how to solve some of the biggest challenges presented by a highly regulated environment. You can read part two here.
"Data quality increasingly plays an important part in ensuring businesses remain compliant. This is a result of an ever-increasing dependency on data than ever before, with more data being stored, processed and used within applications."
For more insights from our engineers and information on our technical expertise, check out our Engineering & Technology blog.
Find out how a federated approach to data governance has helped one of the world's leading energy firms.
Our insights from two years of implementing data mesh at some of the world's most highly regulated enterprises.
