IT Infrastructure & Cloud Strategies Live - SPEAKER'S ARTICLE : Integrating Data Quality Reporting into your Testing Strategy

Courtesy of Beacon Street's, Jon Szymanski wrote a synopsis of her speaking session discussing 'Integrating Data Quality Reporting into your Testing Strategy'.

pillar%20page%20line%201

I have spent the last 18 months building a QA team from greenfield in order to support a large, existing tech stack that has been running for six years. My greatest challenges are in the complexity of our software and our lack of stable, accurate, and separate Test environments.

We have done our best to deal with these issues. We have over 85% of our system functionality documented in test cases and over 70% of those automated and part of our daily regression runs. But we can’t test everything, and we wouldn’t want to.

When we have honest conversations with our stakeholders, affiliates, and customers, the one thing they want the most is …..greater confidence in our software releases and they want to trust the data. In order to provide this as a technology team, we can test more, and we can shift-left to harden our processes with CI/CD and automation as early as possible. And we do that. But as mentioned, we can’t and won’t test everything.

My aha moment was realizing we need to do exactly what our customers want and that was…assess our data. This meant focusing “some” time analyzing our production data, thus bypassing just test case execution in the Test environment. Thus was born what we term Data Quality Reporting (DQR) and a conscience decision to shift-right….just a little.

To reiterate, shifting-left should always be a priority. But we can still take a wholistic approach in our strategy as it is not an “either/or” proposition.

So what does DQR look like?

It is a focus on monitoring our processes, providing immediate feedback, and making specific observations. We take a look at business functions that we understand and then build queries and reports based on that knowledge to validate our end result data.

For example, supporting subscription based businesses, let’s say we generate an invoice for a customer and expect payment. This process traverses through five different batch jobs, then feeds into a fulfillment process that either provides access to content or revokes it. We can test the five separate processes, but identifying an issue in the full integration test is harder. We have an almost endless set of variations and paths based on our complex business rules. This is where DQR can help.

What we do is define the state of the end result and develop a report to run against our production Data Warehouse to identify any data that does not match our expectations. In this example it may be as simple as this:

Report query to identify a payment problem

that still allows access to content.

We run this report daily and send results only if there are records returned. This becomes our DQR alert to a problem. We can then review the record in the report, review logs and jobs, look for potential edge cases or issues we had not tested for or encountered before, and so on.

Writing regression tests takes months. Writing data queries takes minutes. That was a great ROI and allowed us to effectively provide a measure on the confidence of our data.

In summary, when we can’t fully manage areas in the early stages of software development, we can shift-right (downstream). And that is not only okay, but our responsibility. We can provide our customers more confidence in the data and it is a quick win! Focus some time on data quality and not just test case execution because thinking beyond just test cases is part of our job!

Best,

Jon

pillar%20page%20line%201

About the Author

more - 2021-07-12T151703.588 Jon Szymanski,
VP of Software Engineering & QA,
Beacon Street.

Jon has over 25 years of experience in software engineering, managing and leading both development and testing teams. He has worked for large, billion-dollar companies, as well as consulted with start-ups as they build out their technology stacks from scratch.

Regardless of his title or role, his passion has always been focused on quality. He recently built a new QA team, with a heavy focus on automated testing and CI/CD, and is now back leading several large-scale engineering projects. When not at work, Jon is spending time with his wife, daughter, dog, and watching sports.