The sexiest job title of the decade, data scientist, has spawned about a dozen equally sexy but somewhat confusing and overlapping job titles. The primary offshoot of a data scientist is a data engineer. Let’s talk about the difference between a data engineer and a back-end engineer, because frankly, the lines can get quite blurred.
- A data engineer (sometimes called a big data engineer) is responsible for designing and building systems that let organizations collect, store, and analyze large amounts of data.
- Meanwhile, a backend engineer is a more broad job description - they work on the application side of a product, building APIs and connecting databases to user-facing interfaces.
When comparing data engineers and backend engineers, it depends a lot on the company, too. Some companies might have a different understanding of data engineering than others. It’s a sexy job title, yes, but to one company it sounds like hard-code coding from scratch, and to another, it sounds like Excel.
One particularly convoluted answer on Reddit reads:
“I guess you could say that data engineers are the back-end developers of ‘data-intensive application development’.”
I preferred a different answer, which reads:
“TL:DR: Depending on the role it overlaps a lot and sometimes it’s the same.”
The simplest answer, in my opinion? Data engineers are a type of backend engineer. Data is a huge part of the backend. It makes sense that backend engineers have to build systems to collect and manage the data that makes the frontend run. But as the role of data has expanded, organizations have expanded the role of data engineer to be its own thing.
Let’s talk about each role on its own, and then come back to the comparison. By the way, if you’re interested in becoming a data engineer or a back-end engineer, check out Boot.dev - you can learn online in your spare time for waaaaay less than you would spend on college or a bootcamp.
What is a Data Engineer? 🔗
To be a data engineer means data is your baby at your job. Data engineering is the science of designing and building systems to collect, store and analyze all the day that your organization has or could ever possibly want in the future. (Big data engineering is the same, but with larger amounts of data.)
What does a data engineer do? 🔗
As a data engineer, your job is to build systems that collect, manage, and convert data into usable information. This information is used by data scientists and business analysts to interpret and turn into actionable insights. This is an interesting point - as a data engineer, you’re often building tools and systems to allow data scientists to do their jobs.
This itself causes a little bit of confusion – many people believe that data scientists are responsible for collecting the data themselves – but suffice it to say it’s a maturing field and more fine-grained specializations are evolving.
In your day-to-day, you might find yourself acquiring data sets, developing algorithms to transform data, building database pipeline architectures, working with management to understand business objectives, validating data, and ensuring that you comply with data governance and security policies. It’s a big job!
In simple terms, think of yourself as the food prep person. The data scientist in this instance is going to be the chef that combines all the ingredients to make something delicious. But your job is to take only the tastiest tomatoes, only the juiciest lettuce leaves, and make sure there are no bugs in those lettuce leaves. To continue the analogy, you’re also responsible for collecting the tomatoes, and building systems to make sure the tomatoes keep coming to you.
What skills do you need to be a data engineer? 🔗
To be a data engineer, you need a bunch of hard skills and soft skills. It’s a given that you should know how to program – languages like C++, Java, and Python are high up there. Since so much of your job is to do with Ada, you should also know a lot about databases and ask you out. You should be familiar with ETL processes and data warehousing.
You should also be familiar with statistical methods used to analyze data. Even if you’re not doing it yourself, be responsible for the collection of the data that will be analyzed so it’s helpful to know what that’s going to look like. You should also have critical thinking skills. Pretty much a given in any job nowadays, but especially important for data engineers. Everybody knows that data is messy. Collecting and maintaining data for an organization requires a lot of agile problem-solving. That’s where you come in.
You also need to be able to communicate. Not many people put the words data and communication together, but they’re a match made in heaven. Anyone can look at numbers. A data engineer needs to be able to communicate really critical concepts to other people.
What are data engineering job requirements? 🔗
This differs a lot from company to company. I picked GSK’s entry for Data Engineer as an arbitrary employer. To be hired as a data engineer, you may need:
- A BS/MS in Computer Science, Analytics, or similar experience/education
- 2+ years of professional software engineering experience
- Experience with Linux and Python
Your tasks include:
- Developing solutions for collecting, storing, processing, and analyzing huge sets of small molecule-protein interaction data.
- Choosing methods to collect, store etc data – and then implementing and maintaining those methods.
- Integrating your chosen methods with the rest of the information architecture across GSK
- Supporting legacy systems
- Collaborating with analysts, engineers, R&D, computational chemistry/biology teams
How much does a data engineer make? 🔗
What do you get for all this incredible work? As of September 2022, it looks like the average salary for a data engineer is $110,000 per year, with a high end of $174,000 per year. You also get job stability. LinkedIn called it the most in-demand job in 2021, above even the famously sexy data scientist job!
What is a Backend Engineer? 🔗
As the background engineer, data is still your baby, but now you have a whole family of additional responsibilities. You’re responsible for making sure your organization’s website or website products run as smoothly as possible. That includes databases, back-end logic, APIs, architecture, and servers. In many cases, especially at smaller companies, you may find yourself also doing some data engineering as a more generalized back-end engineer.
You can also check out this detailed breakdown of data engineering salary.
What does a Back-end engineer do? 🔗
Let’s break that down. As a back-end developer, you need the technical expertise, analytical thinking, and collaboration skills to design web infrastructure.
That includes: building and maintaining websites, writing code (of course), doing quality assurance testing, increasing the efficiency and speed of the website, doing the inevitable troubleshooting and debugging, and training and supporting client teams and junior developers.
In your day-to-day, you might find yourself developing new features in a secure way, maybe collaborating with other groups like product management or front-end engineering to make sure that your website is perfect, writing code for high-performance environments, and conducting code reviews.
If we continue our kitchen analogy from earlier, you can think of a back-end engineer as the person who is the supplier and systems person. You are ensuring that everybody’s knives are stainless steel–and they’re sharp. You’re also in charge of collecting tomatoes. If the soup team has a quality issue with the tomatoes, you need to be able to talk to that team about the quality and get a new system in place for them.
What skills do you need to be a Backend engineer? 🔗
Backend engineers and data engineers are very similar in terms of necessary skill sets. To become a back-end deverloper you also need to be able to communicate and think analytically.
Technical skills include: developing REST APIs, being familiar with SQL or other related relational databases, building robust APIs, having complete mastery of object-oriented programming strategy, being able to build scalable and distributed systems, and knowing one or more of the popular the backend programming languages like Go, Python, and Node.js (JavaScript). If you decide to self-teach and become a back-end engineer without getting a degree in computer science, it will probably take between 6-12 months to learn back-end.
What are back-end engineer job requirements? 🔗
Like data engineers, the responsibilities of backend engineers differ from company to company. Here’s just one example from Hired.
You’ll need:
- A degree in computer science / web development / computer engineering or equivalent experience
- Experience in any of: SQL, Ruby on Rails, Ruby, Java for front-end and back-end programming, Go, Swift, Docker, Azure, AngularJS and Angular 2
- Experience crafting rest APIs in version control light kit
- Experience with modern MVC Frameworks
- Knowledgeable about application architecture stocks, including API layer, business logic, data modeling, hosting, network and UI/UX
- Experience crafting mobile applications using Objective C, Swift, or Android
- Experience with cloud development, like AWS, Microsoft Azure, or Google Cloud
- The list goes on – there are a lot of potential requirements, but you don’t need to know it all. You just need to know about the tech stack specific to the kinds of companies you want to work at.
Backend engineers seem to need a wide variety of skills.
In that role, you’ll need to know how to:
- Construct all the back-end functionality for a website, and make it efficient
- Use frameworks to build service software and integrate APIs and cloud computing
- Analyze and compile data, codes, and processes used to identify areas of improvement
- Record data and report findings to the appropriate partners
- Build security settings and backup technologies for databases and websites
How much does a Backend engineer make? 🔗
In exchange for labor, you’ll earn about $131,000 a year. This has the potential to go up to $217,000 per year according to Glassdoor.
Data engineer vs Backend engineer: Summary 🔗
A lot of these jobs are hard to compare, partially because the people hiring don’t really know what the jobs actually entail. In many cases, you’ll get the job, and you’ll find that you’ve just been sort of plugged into the hole where the company thinks you should be. Nobody really knows what “big data” is. And if you got someone who says they do, well, I liked Taro Sato’s approach:
“I personally avoid getting to know people who uses the term Big Data frequently because it is likely they probably don’t know what they are talking about. :)”
That being said, there are some clear differences and comparisons between data engineers and back-end engineers. As a data engineer, all you do is data. It’s in the name. As a back-end engineer, you can expect to work with data inside of databases, but you’ll have to do a lot more on the application side - supporting the website or mobile app from the back end.
As these roles get more clarified over the years, I expect that data engineers will see their jobs ballooning to incorporate more of the new exciting functionalities around data collection, storage, and processing. Back-end engineers may have less to do with data as that role is taken on by the data engineers. For now, there’s a little bit of overlap.
Not sure which job you want, data engineer or backend engineer? I advise you to really study job roles at each company. That will tell you more than any explanatory article like this ever could. Companies need bodies that know how to program, know how to set up ETL processes, and know how to work with data. Once you’re hired, your role will evolve to fit that organization’s needs at that moment.
