Tech evolves quickly. When the buzzword Big Data started showing up more and more, the market was in need of people able to analyse and give meaning to what was collected. For instance, an article published in 2012 by Harvard Business Review was entitled: Data Scientist: The Sexiest Job of the 21st Century.
Today we have DBA, Data Scientist, Data Engineer, Data Analyst, a wealth of options with the "Data" as prefix. More often than not, people put everyone in the same basket and assume everyone knows and has the same set of skills.
From my point of view and perspective as a Data Engineer, these are the differences:
- DBA - Once the most hated person I ever had in the team. Seriously, why doesn't that human give me the necessary permissions on the database? If I had access, I would have done my job sooner... Well, that was my thought as a Software Engineer at the time. Turns out, DBAs are, what my friends and I used to call, the database baby sitter. You need to tune and figure out why performance is not as it should be? Need help with a complicated query? That's the go-to person for it. But notice, this is RDBMS specific and heavily focused on the operational part.
- Data Scientist - The market usually wants a professional with a PhD in statistics or an otherwise heavily math-oriented person. This person will be responsible for creating prediction models based on current data. Do you know how Amazon knows what you should buy next based on your browsing history? Yeah, this individual probably did the programming around that, has machine learning down to a T, and needs to possess Product, Engineering as well as Statistical knowledge.
- Data Analyst - This person also deals with a bit of statistics, but more in the business sense, dealing with and creating reports for Business Intelligence. It tries to answer business questions and see where data acquisition/quality is failing, for example.
- Data Engineer - This role I can explain with more passion: it is what I do, so it probably will be biased. We are the bridge. We help Software Engineers to build the application for storage and retrieval in a manner to provide the Data Scientists and Data Analysts with the information they need to do their job.
We do ETL (Extract-Transform-Load). We put data in the Datawarehouse, divise the best strategy for caching information, design database architecture, NoSQL clusters.
Should this JSON return from the Facebook API really be stored in the relational database? (Short answer: no.) Should this query with a LIKE '%string%' really be running in the application and not getting data from Elasticsearch?
We work with RDBMS, NoSQL, Search Engines, Cache engines. I particularly make much use of RDBMS since most of my work was on Legacy applications. As an example, one of our responsibilities is to lower the load on RDBMS for unnecessary stored data presented there.
It is still necessary to know about topics like: indexing, transactions, query profiling and performance tuning.
To sum up: We are the wild card of storage technology.
Developers don't usually care about the precise details of data. They just want what is fast and easy to use. They think about delivery, not so much about long term data retrieval.
"I am going to store this access log for my website on that table."
They probably didn't stop to think that that table will potentially have million of records within a span of months. Why not ELK? Cassandra?
## How should you work with a Data Engineer?
You know that feature you want to implement? Talk to us first. You can design the application the way you want, but we give you the insight in the data layer.
Do not isolate us on your team. Again, we are the bridge.