I have spent the last 18 months building a QA team from greenfield in order to support a large, existing tech stack that has been running for six years. My greatest challenges are in the complexity of our software and our lack of stable, accurate, and separate Test environments.
We have done our best to deal with these issues. We have over 85% of our system functionality documented in test cases and over 70% of those automated and part of our daily regression runs. But we can’t test everything, and we wouldn’t want to.
When we have honest conversations with our stakeholders, affiliates, and customers, the one thing they want the most is …..greater confidence in our software releases and they want to trust the data. In order to provide this as a technology team, we can test more, and we can shift-left to harden our processes with CI/CD and automation as early as possible. And we do that. But as mentioned, we can’t and won’t test everything.
My aha moment was realizing we need to do exactly what our customers want and that was…assess our data. This meant focusing “some” time analyzing our production data, thus bypassing just test case execution in the Test environment. Thus was born what we term Data Quality Reporting (DQR) and a conscience decision to shift-right….just a little.
To reiterate, shifting-left should always be a priority. But we can still take a wholistic approach in our strategy as it is not an “either/or” proposition.
So what does DQR look like?
It is a focus on monitoring our processes, providing immediate feedback, and making specific observations. We take a look at business functions that we understand and then build queries and reports based on that knowledge to validate our end result data.
For example, supporting subscription based businesses, let’s say we generate an invoice for a customer and expect payment. This process traverses through five different batch jobs, then feeds into a fulfillment process that either provides access to content or revokes it. We can test the five separate processes, but identifying an issue in the full integration test is harder. We have an almost endless set of variations and paths based on our complex business rules. This is where DQR can help.
What we do is define the state of the end result and develop a report to run against our production Data Warehouse to identify any data that does not match our expectations. In this example it may be as simple as this:
Report query to identify a payment problem
that still allows access to content.
We run this report daily and send results only if there are records returned. This becomes our DQR alert to a problem. We can then review the record in the report, review logs and jobs, look for potential edge cases or issues we had not tested for or encountered before, and so on.
Writing regression tests takes months. Writing data queries takes minutes. That was a great ROI and allowed us to effectively provide a measure on the confidence of our data.
In summary, when we can’t fully manage areas in the early stages of software development, we can shift-right (downstream). And that is not only okay, but our responsibility. We can provide our customers more confidence in the data and it is a quick win! Focus some time on data quality and not just test case execution because thinking beyond just test cases is part of our job!
Best,
Jon