When I started as a data engineer, I worked on a team focused on DevOps. While it wasn’t exactly what I wanted to do in my first role, it taught me a lot. Now, looking back, if I hadn’t worked in that type of role then, I probably wouldn’t have the experience I have today as an analytics engineer.
Now working as an analytics engineer, I focus on something called DataOps. While this may seem similar to DevOps, they are very different. DevOps focuses on software as a product, while DataOps focuses on producing high-quality data. For those focused on DataOps, data is the product!
While working as a DevOps Data Engineer, I supported software engineers in making code changes to our web application. I focused on testing UI changes after each deployment instead of analyzing the details of the data. Not once did I check the number of rows in a table or if the values in a field were complete. Instead, I made sure no errors were thrown on the backend.
As an analytics engineer, every time I make a code change or push something to production, I need to focus on metadata, or data about data. This involves writing validation queries to ensure that things like row count, column count, and value distribution look as they did before you made a change. Or, if I want them to look different than they did before, reflect those changes!
Although DevOps and DataOps seem similar, they serve two different purposes. In this article we will delve into the differences, addressing the product they intend to offer and the different success metrics.
DevOps involves the deployment and testing of software code changes. When I worked as a DevOps engineer, it often involved long nights of deployment, testing code changes in many different environments, and validating the changes with the software engineers who made them.
