DATA MESH | A NEW WAY OF APPROACHING DATA ENGINEERING

Thought provoking session on distributed analytics architecture. Thanks

Love this! Hello to the new future of data management :)

Excellent presentation! I would definitely agree that maintaining a single data lake at scale for a large organization is definitely a big challenge. I believe that some organizations built data lakes to avoid data duplication and also to build a “single source of truth”. How does this architecture address those problems ?

Very useful and thought provoking, thank you for sharing.

Low coupling and high cohesion ... again :) Very nice presentation !!!

Very nice
I just think it would be great if there were examples of how all of this connects to real world products. I mean, data products are commonly only legacy business products like a ERP or CRM, or MES system. What are the changes that should be made in those products so they can adapt to the domain perspective? How it would impact the current user journey?
When I consider this I either conclude that we still have lots of big gaps to cover or I conclude that I didn't understand the concept

Very interesting presentation.. the only risk I foresee is that when data products consume each other instead of from a centralized lake, the errors and bugs will spread out into the network virtually unihibited. In centralized data pipelines, it is still possible to just purge and recreate the data product from immutable logs. But in this case, it will be extremely difficult to prevent the spread and later correct. If one data product goes down, it will impact the whole network. This is a problem with microservices architecture and this will be a problem in Data Mesh as well.

Game changer. In reality, people do build data products as a result of that friction in the centralized approach . They just don’t call it that way. However, I loved the phrases interoperable and trustworthy. There needs to be a well defined pattern around bringing consistency to inter-dataproduct interactions.

Fantastic! Working for a steel company that is just dipping its toes into the Big Data platform field, aspects of optimzing resources with respect to data engineering et al. are paramount moving forward.

Persians are really smart yet friendly and nice. Zhamak is not an exception obviously.

data mesh ~ data analog of "microservices" architecture for a mesh of services

Absolutely, crisp and clear explaination by just bringing on the abstract of the concept. My perspective, though this is going to be a paradigm shift from lakehouse to data mesh, what is absolutely requried is to understand the traps which the Arch. Strategy team should be careful about.

Lots of those problems have already been solved a decade or more ago. Most data project failures today (which she did not mention but Gartner ranges at around 70-80% percente) do not stem from lack of technology.

Data mesh + Better Value Sooner Safer Happier for the win :-)

Complimentary to any warehouse lake, lake house, or mesh... or whatever new concept of data storage governance and security model is thought up in the future

Our AI/ML effortlessly harmonizes and contextualizes all data across all silos even third-party or unstructured content like handwriting, images, etc

If you want to achieve mesh at any scale, you'll want to talk to me.

Star models are built for technologies? No, these are optimized around business processes and are more end user understandable. So what about the integration of business keys if you move the DWH to the source systems?

Great job Zhamak! Was very good information regarding data mesh. Easy to understand and good ideas.

Fantastic, thought provoking and great guidance.

She can store data in my mesh.

Data mesh? Another hype? We have gone from centralized to decentralized data. From centralized analytics to self service analytics? There is no silver bullet solution. Data mesh is mostly a cultural change and less technology. And no matter what solution. The weakest link in the multiple data product teams determines the success. Data Governance is a challenge and with cross domain products teams are dependable on each other. So what problem are we really trying to solve we are already not able to solve in monolytical data silos? It doesn’t matter if you ingest data, cleanse and create data product decentralized or centralized. We are still talking about creating data processes that need to be developed, released and maintained. Will it go faster? Will it add more quality? Will we be more flexible? Or are we creating chaos because we are not able to manage the multiple data product teams that will do what ever they want eventually because they don’t want to be dependable on other teams? So centralize where you have to and decentralize where not. Within 5 years from now we are talking about the Data Labyrinth because we are all lost in translation.

I don’t get it. Data mesh seems stupid to me