14 thoughts on the economics of the open-source data space and how to become the next MongoDB or Databricks — The data space is booming, with companies like mongoDB (valued at 18 billion USD), databricks (30 billion), or Confluent, and many others. The startup space is overflowing with money and lots of founders want a share of the pie. But in my opinion, the data space is set up to…

I don’t like to dwell on the past. And a lot of times, it is essential to make mistakes yourself. And yet, I find myself recommending many practices to new data engineers. Because I felt the pain personally, and it wasn’t fun. I spent two months early on my job…

A simple explanation of what vector databases & embeddings are and why they are so hot right now! — Gazillions of dollars are being poured into vector databases like Pinecone and Weaviate. But why? Both companies were founded already five years ago. To understand, you need to understand the essence of three things, vectors embeddings vector databases Luckily, I got a story ready for you that makes it easy…

Why data engineers can make their lives easier by providing fresher data — Google “Real-time data is underrated,” and you’ll likely end up with an article by Hunter Walk explaining why it is overrated and the true data that’s underrated is historical data.

Immutable raw areas, no transformations, no flattening, and no dedups before finishing your excavations — This post is an updated version of the original version of the Meltano blog. ELT is becoming the default choice for data architectures, yet many best practices focus primarily on “T”: the transformations. But data quality is determined for transformation and beyond in the extract and load phase. As the…

Build your data company faster, manage your data products better, and invest smarter by using data-specific mental models — “In Munger’s view, it is better to be worldly wise than to spend lots of time working with a single model that is precisely wrong. A multiple-model approach that is only approximately right will produce a far better outcome in anything that involves people or a social system.” (Tren Griffin…

5 Reasons to switch your career to data product management. — I’ve noticed a shift. It happened to me when I entered my role as data PM, it is happening all around the tech industry. It is happening in traditional industries, and it comes fast. Data product management, emphasis on data, is becoming a thing. In job titles, web content, discussions…

7 Lessons to build lean, simple, and agile data stacks, not fat ones. — Data architecture, it’s that time of the year again. For some reason, whenever it is dark outside, so are the thoughts about your own data architecture, outdated, too complex, something is missing. But it doesn’t have to be that way. The solution is not to hire dedicated data architects that…

What Writing a PhD Thesis In Singularity Theory Has Taught Me About Data Engineering — In August 2017 I started writing my phD thesis in a field called singularity theory. Not the fun AI kind of theory, but the mathematical version about “when unexpected things happen”. And so it happened, that while I was still sitting more in my office than actually writing, a friend…