Data as Code — Cutting Things Smaller

A very different approach to designing analytics systems without doing any data meshy things.

From Mono DAGs to Service-level DAGs for more value extracted from data.

(This was originally part of my newsletter as a “Thoughtful Friday edition”)

Domain-Driven Design and microservices have changed the way software engineers work. And I think, they can be used to multiply the productivity of data teams as well. But I also think, they have to be used slightly differently because the common centralized data team is in a very different situation than the common (decentralized) software dev team.

This article is about exploring this very different way of working with your central data system. It offers a simple iterative way, just like microservices, to start with the default monolithic one pot to capture it all. It then lets you slowly break out pieces once you realize the time is ready. But you’ll have to throw away some of your biases because, just as in the microservices world, you will end up with duplicate data and duplicate work, by intention.

I’m trying to do something different than the data mesh approach here, the data mesh approach would try to get your data team to just “own one domain”. Whereas this article is about acknowledging that the data team simply has a lot of domains from other teams inside their…