Seasoned data engineers know something you don’t, but I’m going to let you in on the secret.
You learn more by doing. That’s it, that’s the secret.
But the trouble is, there is a lot of bad advice out there. Advice that says you just need to follow this 3-step plan, do this “step by step”, have a positive attitude, and a $500+ course (which, you guessed it, happens to be theirs) — it’s rampant.
You are also up against the marketing teams of the world selling you this and that course, all wrapped and ribboned as the silver bullet that’s going to give you the skills you need to land that dream job.
Unfortunately, out of sheer desperation, many fall victim to these sales tactics and end up out of pocket and out of luck, asking themselves why they can’t be like them.
You’re never going to be like them, be like you. Leverage your skills to gain an edge.
The Trap
The biggest trap for data engineers, especially for those starting out, is thinking you need to know it all. It goes something like this:
You think to yourself, “I need to know Spark.” You sign up for a Spark course watch the 40 hours of content, get your little certificate of completion, and you don’t really have a grasp on any of it. Why? Because you didn’t get hands-on. “But, I followed along,” I hear you say — that’s not doing. That’s just copying someone else’s thought process. Sure, some of it’s going to stick, but not all of it. “Oh, maybe if I learn Databricks, it will make more sense.” Then, you sign up for a Databricks course. Do you see the pattern? Around and around it goes.
You don’t need to learn or know more to start building out projects. You run the risk of acquiring all this knowledge but never applying it.
Reality Check
In picking up the skills to be a data engineer, there are no shortcuts, and if you think you’ve found one. You haven’t. Simple as that.
If you were to run a root cause analysis on why this cycle isn’t working, you would quickly realize that this method of learning is not sustainable. Why? Your expectations are way off. When you’ve wrapped up your courses (although, let’s be honest most people never finish them), you quickly realize how much you don’t know and how much there is still to learn. You lose faith and give up.
The scale of the task overwhelms you.
The Harsh Truth
You’re supposed to make mistakes. You are supposed to fail. Failure is just part of the process toward succeeding.
The struggle is part of the journey. The time you spend figuring things out, researching your errors (yes, there will be errors, lots of them), building out a project, or picking apart a problem is not wasted time. That, my friend, is you succeeding and building up your knowledge.
The truth is that you’ll learn more by building your own projects. Only then will that knowledge take hold and stick with you.
I can guarantee the greats out there don’t jump from tutorial to tutorial. They learn by grabbing what they need and applying the knowledge. Early on in my career — I can’t recall exactly when — I attended a lecture on Database design. The last slide of the presentation had a comment that changed the way I learn once and forever:
Experience = 70% hands-on + 30% theory
That has been my philosophy ever since. Tutorials have their place, and I’m sure you could pick up a thing or two, but you need to start applying what you learn. Start experimenting. Find a data source that interests you and play. The internet is full of resources to help you — and guess what? they are free. Take what is useful and try it yourself.
If you keep at it and have a stubborn determination, things will get better. The sense of accomplishment you will have after building and applying what you learn will be just as good if not better than getting a certificate of completion. Remember, the scariest moment is always just before you start. After that, things can only get better.
Thanks for reading! If you want to get in touch, feel free to find me on LinkedIn.
