What is a Data Engineer? The Crucial Role in Data Science

With data vastly growing daily over the past few years, the need for data engineers has grown significantly. Data engineers always appear to be overlooked and are outshined by roles such as Data Scientists and ML engineers, but they all play a crucial role together. Without data engineers, there wouldn’t be any data available for Data Analysts/Scientists or ML Engineers, but yet it isn’t talked about enough as the role itself isn’t seen as a ‘Sexy’ job like the title of ‘Data Scientist’ which was labeled as one of the ‘Sexiest Jobs’ of the 21st century.

I’m here to tell you that data engineering is just as important as the roles of Data Analysts/Scientists or ML Engineers. I see explanations from different people on this, so I decided to use an interesting Game of Thrones or medieval-like analogy to showcase the value of data engineers.

What is a Data Engineer and their role?

Before going into why data engineers are an important asset, you should probably get an idea of what their role is. Data engineers in simple terms are responsible for designing and building systems to collect, store, and prepare data. The data is then used by colleagues who are either analysts, data scientists, or ML engineers to visualize or model the data for the point of coming across findings to help drive business.

You’ve probably gone through hundreds of definitions of these, but this is what data engineers do at a high-level.

Now, on to the analogy.

The Data Kingdom and Its Defenders:

Data Engineers: The Stonemasons and Builders

Imagine them as the strong, skilled stonemasons who build the very foundation of the Data Kingdom. They construct robust databases, pipelines, and storage systems, like building sturdy castles and secure vaults to hold the kingdom’s treasures (data).

Data Scientists: The Scholars and Alchemists

These are the wise scholars who pore over the data collected by the engineers. They are like alchemists, transforming raw materials (data) into valuable insights and knowledge through analysis and modeling. They use their expertise to identify patterns, trends, and hidden truths within the data.

(And yes, this is pretty much what they see themselves as.)

Machine Learning Engineers: The Spellcasters and Inventors

Think of them as ingenious spellcasters and inventors who use the knowledge extracted by the data scientists. They take the insights and models and translate them into real-world applications, like crafting magical tools and enchanted devices powered by data. These tools can automate tasks, predict future outcomes, or even make decisions based on the data.

The Flow of Information and putting it all together in a real-world scenario:

The data engineers build the infrastructure, the data scientists gather the knowledge for modeling, and the machine learning engineers put that knowledge to work. They all rely on each other’s expertise to protect and utilize the kingdom’s most valuable resource – data.

Without the Stonemasons:

So overall, if the data engineers (stonemasons) aren’t there, the data wouldn’t have a safe and secure home. The data scientists (alchemists) wouldn’t have any materials to analyze/model, and the machine learning engineers (inventors) wouldn’t have the insights needed to create their tools. The kingdom would be vulnerable and unable to harness the power of data.

Together, they are stronger.

I hope this analogy helped you understand the importance of each role, especially for data engineering, and how they collaborate to unlock the true potential of data within the Data Kingdom.