When you wake up one day and decide to be a Data Engineer, the obvious first step is to learn your SQL and your Python, like a good student (and yes, you should do this).
Perhaps the adventurous ones out there will try their hand at data modelling (some of you will get good at it too, but most won’t). There will be trials with Docker, dabbling with unit testing, and messing around with Airflow or the like.
Most of you will sign up for tutorials (the good ones and the bad ones). You will hit all the courses you can get your little hands on. Some will drop money they don’t have (hundreds of dollars of it too) on boot camps (the get-rich-quick schemes of the data engineering world).
Some will finish those tutorials and courses. Others will start them and never finish them. They will lie piled as a constant reminder of everything you don’t know.
Some will tackle LeetCode problems, while others will take on side projects. A few will spend hours reading and never doing. Google will become your friend, and ChatGPT, an unlikely partner. A select few out there make this whole thing look easy, while you grind it out day in and day out.
The route you take to get “good” differs from person to person. The road is a rocky one, not one smooth path to success. Rarely do things ever go smoothly. Surprises lay around every corner, ready to jump out at every turn. You need to keep your guard up, keep your wits about you, and prepare for the unexpected.
Here are a few things you need to come to terms with:
- You are going to mess up (sooner or later).
- You are going to fail (we are human, and mistakes happen).
- You will want to give up (don’t do that, ever).
A mixture of reasons can cause the above to happen.
Impatience or arrogance: You move too fast, cut corners to get the project done, or worse, you think you know it all and dive headfirst into tasks, only to get a rude awakening in the form of a failed pipeline or project. Slow down and think things through.
Inexperience or insecurity: Imposter syndrome, anyone? You might feel like you’re not good enough to tackle complex data tasks. Inexperience can also lead to a fear of making mistakes, which can bring on anxiety and before you know it you’re on the carousel that never stops to let you off. Remember that everyone starts somewhere, and learning from your mistakes is the best way to learn. Keep going.
Knowledge or lack of interest: You cannot know it all in Data Engineering. Having gaps in your knowledge can happen (actually it will happen), and it can also lead to confusion and errors in your work. I’ve also seen Data Engineers lose their passion for the field; it’s hard to stay motivated and engaged in your projects, especially if all you want to do is run off and play with the “new” shiny tools. The best thing you can do is to continuously learn and find the aspects of data engineering that genuinely captivate your interest. Work on side projects and do what you have to do to stay motivated. For me, helping others brings out the most joy. Find what works for you.
This will happen:
- You will break things over and over again.
- You will get stuck on errors that need to be investigated and understood.
- You will most definitely over-engineer simple tasks, take on too much, forget to test, miss deadlines, and projects will get changed halfway through.
- You will neglect those all-important soft skills and get lost occasionally.
There will be moments when, if you are not losing interest in a project, you are losing your mind about it. At times, it will be confusing and frustrating. Welcome to Data Engineering!
This is why we love it. This is why we spend hours and hours learning and improving — trying this and trying that. It’s all part of the process of becoming a great data engineer. Wherever you are on your journey, keep going, have patience, and keep the faith. It will all work out in the end.
🎯 Thanks for reading! If you liked it, please give me a follow and don’t forget to subscribe to get my latest articles.
Want to get in touch? feel free to find me on LinkedIn.
