Surviving the Rocky Road of Data Engineering
When you wake up one day and decide to be a Data Engineer, the obvious first step is to learn your SQL and your Python, like a good student (and yes, you should do this). Perhaps the adventurous ones out there will try their hand at data modelling (some of you will get good at […]
Why Deadlines Make This Data Engineer Tick (Here’s Why)
Deadlines are all about helping yourself. Deadlines are excuse-busters. Deadlines keep you on point and in focus. Well, that’s how I see them, anyway. Not everyone will agree, but I love deadlines. Both for work and for my personal side projects. They keep me in check. They keep me honest. With any project, there’s one thing you […]
The 3 Biggest Realisations I’ve Had As Data Engineer
Self-Belief, Self-Improvement, and Self-Education One of my favourite proverbs is a Haitian one about overcoming obstacles: Beyond mountains, more mountains. If you work in data engineering, I think you can agree that you can easily apply that wisdom to a data engineer’s journey: when you think you’ve grasped one thing or solved one problem, something […]
10 Years Worth of Valuable Advice From a Data Engineer (Navigate the DataWorld)
I remember the first time I ever typed “SELECT * FROM” I was 18 and had no idea what I was doing. I clearly remember thinking, “Star” what? I also remember the first database I ever dropped – accidentally (it was in development if you were wondering). I also remember restoring it — sharpish. I […]
Harnessing the Full Potential of BigQuery Table Functions
I come from SQL Server land, it was a great place to live. It’s an even better place to live if you know what you are doing (and I like to think I knew what I was doing — from time to time). I tell you this because, for the better part of my career, […]
5 Quiet Signs You’re Destined to Be a Great Data Engineer
What you do today affects what you build tomorrow. I recently came across this quote, and it made me realize there are many talented people out there who grind it out day in day out. Who show up and get the job done. Who always find the answers no matter what. Who figure things out. […]
Breaking Free from the Data Engineering Learning Loop
Seasoned data engineers know something you don’t, but I’m going to let you in on the secret. You learn more by doing. That’s it, that’s the secret. But the trouble is, there is a lot of bad advice out there. Advice that says you just need to follow this 3-step plan, do this “step by step”, […]
Airflow in Less Than 5 Minutes: Getting Started with Astro CLI
Airflow has been my go-to orchestration tool for a few years. I say that like I had a choice, but really, it’s been the tool of choice for all the companies I’ve worked for, and it’s served them well. I’ve worked on self-managed setups and seen the joys of Managed Airflow in AWS. I’ve also […]
3 Lessons for Junior Data Engineers
It’s been a few years since I began my Data Engineering journey. There have been a lot of ups and downs along the way. Here are 3 things I wish I’d known from day one: 1 — Leverage your strengths to gain an edge We are all different, and we all have our own strengths […]
Why Data Engineers Shouldn’t Be Afraid to Say “I Don’t Know”
The three words that will serve you well are: “I don’t know.” We can’t know everything. We just can’t, it’s not possible. The sooner you come to peace with this fact the better. This has never been more true in the field of Data Engineering. The data industry is exploding. Every other day, some new […]