In the pursuit of ensuring clean and on-demand data provisioning, businesses are not hesitating from experimenting with change. Unlike previously, they are more prepared for revamping their data management verticals. So be it moving on to advanced fabrics or migrating from monolithic lakes to data mesh, a lot is happening and waiting to happen in the quarters ahead. However, this came with no surprise as IDC predicted the propelling growth in 2019. As per their report, the global data sphere could easily grow to 175 zettabytes by 2025, says Yash Mehta, an IoT and big data science specialist.
Briefly, enterprise architecture allows you to collect data from different sources and then further provide it for diverse use cases. Although for start-up applications (businesses), a monolithic architecture is a foolproof way to get started as it’s simple, has a single -backlog, vendor, solution, and team, it can be quite challenging in case you decide to scale up.
Moreover, managing several data products and their dependencies across different domains is another complexity plaguing many mesh architecture platforms.We discuss more such complexities that start-ups should know before proceeding.
1. Successful decentralisation of ownership to all domains
If we look at different domains, each one of them has different quality and governance requirements. These must be taken into account every time as these pipelines and data products work as shared communities. This is important to ensure federated data governance.
- Domain duplicity
A common issue in data mesh architecture is the multiple domains is the data duplicity. Therefore, when a domain is repurposed to serve a business, redundancy is often a byproduct. This potentially influences different parameters like data management costs and resource utilisation.
2: Inhibit cross-domain analytics
To define an enterprise-level wide data model, you must be able to consolidate different data products and bring them out to your customers in a single central location. Although a data mesh answers several challenges that arise because of data lakes, it usually fails to address cross-functional analytics which is one of the main benefits of a data lake. This is precisely why a start-up organisation must always make sure that they are ready to address such issues else they stand a chance to close the doors to analytical innovation
3: Convince domains to cooperate
It is a well-known fact that a data mesh creates a lot of extra work for the domains which previously came in the form of consumed reports. Now you need to convince the domains that all this extra work is worth the effort. When they are on board you still need to coordinate all serious releases with them.
You can also face some delays if you are looking to improve the platform while the domains are testing new applications. This can stall the process by months and so implementing a release calendar or creating a guide for platform-domain team cooperation.
Domain management also depends upon the choice of Mesh architecture you pick. K2View, for example, addresses the ‘domain independence confederacy’ issue by striking the right balance with the central data teams. Here, the centralised teams associate with domain teams to produce products. These teams create APIs for every data consumer; govern and control the access rights followed by consistent monitoring.
Their platform works because it integrates data from a multitude of sources into target products thereby ensuring secured distribution among all the domains. This helps in provisioning centralised data modelling, governance, and cataloguing for analytical as well as operational workloads.
4: Ensuring technical expertise
For start-ups, this is a major challenge. While enterprises have the infrastructure and the knowledge base for persistent upskilling & reskilling, new companies should accelerate their recruitment goals.
This becomes even more important if they strategise to delegate the ownership at the domain level.
This means they can hire new talent and train themselves which sometimes can get too much. Ignoring such issues will not cause any problem immediately but issues generally start popping up in time as the performance decreases.
Until now no tools have been able to solve these challenges even by implementing abstraction layers that hide technical aspects. Understanding data engineering is thus quite crucial and is a skill that should never be underestimated.
- Need for data integration expertise
Data pipelining at the domain level requires expertise across complicated data integration and modelling of multiple disparate source systems. The idea is to avoid dealing with underlying source systems during data management in the virtual layer.
5: Slow-to-adopt process
The data mesh has proven to be quite an ambitious architectural change because it calls for a huge upfront investment. The move to decentralising data management calls for considerate change management when it comes to centralised data management protocols.
Also, let us not forget that popular large-scale data infrastructure overhauls are time-consuming and decentralise all data operations as well. This is why organisations must plan and prepare in advance before opting for a data mesh.
Finally,
6: Selecting the right data mesh platform
Start-ups must build a strong foundation in data management. Likewise, they need to start with the most appropriate platform. Ideally, the mesh architecture should enable the teams to elevate their data analytics outcomes. As already explained above, the right platform must support cross-functional domain collaboration between the centralised teams and different domains. Given the rapid adoption, we are not too far from data mesh 2.0. Therefore, companies should handpick their data mesh platforms carefully. While we are at it, IBM Data Mesh, K2view, Trustgrid, Oracle, and Talend are a few of the names.
- Time for data mesh 2.0
The need for more data is a compelling driver for data-driven organisations to embrace advanced automation. Start-ups have an advantage here. They can directly start with 2.0, which effectively addresses the gaps in the traditional workflows. Which data mesh platform are you using?.
The author is Yash Mehta, an IoT and big data science specialist.
Comment on this article below or via Twitter: @VanillaPlus OR @jcvplus
