. Part of me began this journey as a result of information engineering is among the hottest and highest-paying careers proper now. I’m not going to fake that wasn’t an element.
However there’s extra to it than that.
I’ve been studying information analytics for some time now. SQL, Energy BI, Python (Pandas, NumPy, slightly Polars), information cleansing, EDA. You identify it, I’ve been within the weeds with it. And I genuinely get pleasure from it. However someplace alongside the way in which, I began getting inquisitive about what occurs earlier than the info lands on my desk. How does it transfer? Who builds these pipelines? What does the infrastructure behind all of this truly appear like?
That curiosity planted a seed.
Then AI began making numerous what I do quicker and simpler. Which is nice. However it additionally made me assume: if AI can deal with the evaluation, what’s my edge? What can I construct and perceive that goes deeper? I work as an IT System Analyst at a startup, and whereas I benefit from the work, I spotted I wasn’t difficult myself the way in which I wished to. I used to be prepared for extra.
The ultimate push got here from a video by Knowledge With Baraa, the place he laid out a whole data engineering roadmap. One thing about seeing it structured and damaged down made it really feel actual and doable. So right here I’m.
I’m studying information engineering in public. And this text is the start of that journey.
Additionally, simply leaving a disclaimer that I’m not affiliated with Knowledge with Baraa. I’m simply sharing my private journey. Hope it helps.
Why Knowledge Engineering Particularly
I need to spend a second right here as a result of I feel this query deserves an actual reply.
Knowledge analytics taught me the best way to work with information after it arrives. Clear it, discover it, visualize it, draw insights from it. That skillset is genuinely invaluable. However the extra I realized, the extra I stored bumping into the identical wall. The information I used to be working with had already been formed and moved by another person. Somebody had constructed the pipeline that introduced it to me. Somebody had determined the way it was saved, the way it was structured, how usually it refreshed.
I wished to be that individual.
Knowledge engineering sits upstream from analytics. It’s about constructing the techniques that make evaluation attainable within the first place. Knowledge pipelines, storage structure, workflow orchestration, large-scale information processing. These are the foundations every part else is constructed on. And actually, that type of infrastructure work appeals to me in a manner that pure evaluation now not does.
There’s additionally a sensible argument. Knowledge engineering roles persistently rank among the many highest paying within the information business. As AI instruments get higher at automating the analytical layer, the demand for individuals who can construct and keep dependable information infrastructure is just going to develop. I’d relatively be constructing the pipes than simply utilizing them.
And yet one more factor. The startup I work at doesn’t use any of the instruments I’m about to be taught. Which implies each hour I put into that is totally self-directed. No group to be taught from, no work initiatives to use it on. Simply me, the web, and no matter I can construct alone. That’s a problem I’m selecting on objective.
Why I’m Doing This in Public
Writing about what I be taught is one thing I already consider in deeply. It forces you to really perceive one thing earlier than you clarify it. It retains you accountable. And over time, it builds one thing {that a} resume alone by no means may.
However I’ll be trustworthy about my fears too, as a result of I feel that’s the purpose of doing this publicly.
I’ve shiny object syndrome. There, I mentioned it. I’ve explored graphic design, animation, writing, advertising and marketing, and IT earlier than touchdown in information. There’s all the time one thing new and thrilling pulling my consideration. Knowledge engineering may simply get changed by the subsequent flashy factor in my feed if I’m not intentional about it.
Consistency is one other one. I work a 9-5 the place I barely contact the instruments I’ll be studying. There’s no pure reinforcement at work, no colleague I can bounce Airflow questions off of. I’m constructing this totally alone time, outdoors of my job tasks.
And steadiness. Three to 4 hours a day is the aim. Some days that may really feel straightforward. Different days it’s going to really feel unattainable.
Publishing this journey is my accountability system. If I’m going quiet, you’ll know I slipped. And I’d relatively not slip.
What I’m Beginning With
I’m not ranging from zero, which helps. I have already got newbie to intermediate SQL information from my information analytics work, primary Python fundamentals, and a few hands-on expertise with Pandas. That offers me a basis to construct on relatively than rebuild from scratch.
Right here’s the total studying stack, roughly within the order I’ll be tackling it.
1. SQL: Going Deeper Than Analytics
I do know SQL. However analytics SQL and engineering SQL are totally different animals. I’ll be going deeper into question optimization, indexing, working with very massive datasets, and writing SQL that’s constructed for efficiency relatively than simply exploration. For those who’ve solely ever used SQL to drag and filter information, there’s a complete different layer beneath price understanding.
Why it’s first: Every little thing in information engineering finally touches SQL. Getting sharp right here earlier than layering in additional advanced instruments makes the remainder of the journey simpler.
2. Python: From Exploratory to Manufacturing-Prepared
I’ve the fundamentals. Pandas, NumPy, some Polars. However the Python I’ve been writing lives principally in notebooks. Exploratory, messy, not constructed to final. The aim now’s to put in writing cleaner, extra structured, reusable code. Capabilities, modules, error dealing with, scripting. The type of Python you’d truly put in a pipeline.
Why it issues: Python is the glue that holds most trendy information engineering stacks collectively. Airflow makes use of it. PySpark is constructed on it. Getting comfy right here is non-negotiable.
3. Git and GitHub: Model Management Finished Correctly
I’ll be trustworthy. My Git information is at present “copy the command, hope it really works.” That has to vary. Model management is prime to working like an engineer relatively than simply an analyst. I’ll be studying branching, pull requests, and the best way to handle code correctly throughout initiatives.
Why it issues: Each venture I construct from right here on goes on GitHub. It’s portfolio, it’s self-discipline, and it’s how actual groups work.
4. Apache Spark and PySpark: Massive Knowledge Processing
That is the place issues get genuinely thrilling. Apache Spark is among the most generally used engines for processing large-scale information. PySpark is the Python API for it, which suggests I can use a language I’m already considerably conversant in to work with distributed information at scale.
The bounce from Pandas to Spark is a mindset shift. Pandas works on a single machine. Spark is constructed to run throughout clusters. Studying to assume in that distributed manner is among the expertise that separates information engineers from analysts.
Why it issues: If you wish to work with large information in a manufacturing surroundings, Spark is sort of unavoidable. It exhibits up in job descriptions consistently and is core to the Databricks ecosystem I’ll be constructing towards.
5. Apache Airflow: Orchestrating Knowledge Pipelines
Knowledge pipelines don’t run themselves. You want one thing to schedule them, monitor them, and deal with failures gracefully. That’s the place workflow orchestration instruments are available in, and Airflow is my decide.
I thought-about a number of choices right here. Databricks Workflows is nice if you happen to’re already deep within the Databricks ecosystem. Azure Knowledge Manufacturing facility is sensible for Azure-heavy environments. However Airflow is free, open-source, cloud-agnostic, and broadly used throughout the business. It additionally teaches you the core ideas of orchestration in a manner that transfers to different instruments. Beginning with Airflow felt like the precise name, particularly since I’m making an attempt to maintain prices low.
Why it issues: Orchestration is what turns a set of scripts into an precise pipeline. Understanding Airflow is knowing how manufacturing information workflows are managed.
6. Databricks: The Knowledge Platform
In some unspecified time in the future it’s good to decide an information platform and go deep on it. I’m going with Databricks. It’s constructed on prime of Spark, it’s in excessive demand, and it has a free Group Version that permits you to follow with out paying for cloud credit.
The alternate options are stable too. Snowflake is a clear, quick SQL warehouse that numerous firms love. BigQuery is Google’s absolutely managed, serverless choice and genuinely wonderful if you happen to’re leaning towards Google Cloud. However Databricks sits on the intersection of huge information, machine studying, and information engineering in a manner that matches the place I need to go. It made essentially the most sense for my targets.
Why it issues: Employers need you to have platform expertise. Going deep on one is extra invaluable than realizing slightly about all of them.
How I’m Structuring the 12 Months
The trustworthy reply is that this would possibly take longer than 12 months. And I’m okay with that. I’d relatively take 15 months and really perceive what I’m doing than rush by means of in 12 and are available out shaky on the basics.
The overall strategy is to maneuver by means of every talent so as and never advance till I’ve constructed one thing with what I simply realized. Tutorials are advantageous for orientation however initiatives are the place actual studying occurs. My plan is to doc every part right here on In the direction of Knowledge Science: the ideas, the initiatives, the frustrations, and the wins.
For monitoring progress, I’m utilizing the Notion roadmap from Knowledge With Baraa as my spine. It breaks down every talent into core subjects and lets me observe the place I’m with out getting overwhelmed by the total image unexpectedly.
As for time dedication, three to 4 hours a day is the goal. A few of that will probably be structured studying. Some will probably be constructing. Some will probably be writing about what I simply realized, which is its personal type of learning.
What Success Appears Like
Touchdown a high-paying information engineering function is the aim. That’s actual and I’m not going to decorate it up.
However alongside that, I need to turn out to be a reputable voice on this area. Somebody who builds issues price speaking about, paperwork the journey with out filtering out the arduous components, and perhaps makes the trail slightly clearer for somebody developing behind me.
The writing and the educational feed one another. The portfolio turns into the proof. The proof builds the model. That’s the imaginative and prescient.
Beginning Immediately
This text is my official begin date. I’m not ready till I really feel prepared or till every part is completely deliberate. I’m beginning now, writing as I’m going, and letting the method be public and slightly messy.
For those who’re someplace on the same path. Whether or not you’re in analytics interested by engineering, in IT questioning what’s subsequent, or simply somebody making an attempt to construct expertise that maintain their worth in an AI-accelerated world. Comply with alongside.
I feel we’ll have quite a bit to speak about. I’ll even be sharing my learnings on my YouTube channel. So be happy to subscribe under and comply with alongside.
That is the primary article in an ongoing sequence documenting my information engineering journey. I’ll be publishing recurrently on my progress, the initiatives I’m constructing, and every part I be taught alongside the way in which.
And if you wish to get entry to the Notion template, in case you’re on the identical journey as I’m, you’ll be able to entry it here.
Comply with alongside on my journey under.
