Reading is a superpower, knowledge a person has accumulated over years you get to ingest in minutes.
But it also comes with dangers, the danger of ingesting crap, to spend hours filtering through content that isn’t good at all.
And yet, from time to time, you stumble over unique articles. My feeling then usually is, “Why didn’t I know this before? Why has no one told me about this article?”
There are only a few articles that changed my perspective on data engineering as a discipline, and I do remember all of them pretty well. All of them contain a small, unique lesson that’s unique, thought-provoking, and true, at least to me.
Here are the 8 that stand out, that got me just that feeling, the 8 I think you need to read. So let me tell you why:
The Rise of the Data Engineer
I joined Facebook in 2011 as a business intelligence engineer. By the time I left in 2013, I was a data engineer.
You might know this article; it’s been all over the internet for quite some time. It is already six years old, and it already has some patina.
But the lesson I walked away with does not: No data engineering job description won’t change in the next year.
That’s right, if you didn’t notice, in 2015, there were no data engineers. The discipline was basically invented around 2015–2017, going away from the database admin-like role it was before.
In 2018-now, multiple streams of change rolled over to data engineers. Suddenly, learning Python was a must. Learning heavy SQL and knowing your way around dbt became necessary, and collaborating with data scientists and machine learners became necessary.
Every year, the job description changes to a degree that makes it a completely new job within 1–2 years, with new skills and new challenges.