System Verification

Why testability is the key

Faster releases at lower costs and with more satisfied end users. That is what every company wants. And some companies succeed. What is their secret? A new report from Forrester shows that they have implemented Agile+DevOps and prioritizing testing, having a high risk thinking and focusing on end-to-end test automation. So, what does this mean in practical terms?

We let Patrik Schalin, DevOps Specialist at System Verification, review Forrester’s report “The Definitive Software Quality Metrics for Agile+DevOps” (July 2018) and clarify what is required to increase the pace of the development process – without losing quality.

“It is primarily about testability. The more effort you put into creating a context where the software can be tested, the faster you will be able to develop with high quality. Many companies want to speed up the test automation process, which might seem logical, but to automate, the software must be tested and that must involve several parties in the organization. Furthermore, the report implies that it is critical to involve testers, but I believe it is important to involve testers as well as other functions such as IT infrastructure, to get a well-functioning DevOps. Together, the parties create a context that controls the software development and makes it testable. And that is an ongoing work.”

Can you give us an example?

“A large global company where I used to work wanted to implement DevOps, and me and the rest of the Development and Testing Department did everything we could when it came to programming and testing, but in the end, it turned out that the we could not reap the benefits of automation because of the poor IT infrastructure.”

What does high risk thinking mean?

“It means that you must understand what you are measuring and the consequences of the results. The report addresses Code Coverage as an important metric, but that could be precarious. High coverage does not necessarily mean that you are testing the right parts of the code. The part that was not tested can be the part that contains the critical bugs. Instead, you should analyze the code, for example with CodeScene, to find out where the critical hot spots are. The challenge is that these areas are often the hardest ones to test.

Another metric that is mentioned in the report, which can be a bit risky, is Unit test pass/fail. 100 % pass may be a result of testing the wrong parts of the code base, while 100% fail may be a result of neglecting the test results. Instead of using this metric as quality mark, developers should see their test suites as safety net when they create prototypes or fix bugs.

Nowadays we have access to a whole variety of metrics that are easy to report on, and it is easy to create a false sense of security just because we are testing a lot. Instead you should dare to choose the critical metrics. Companies that have good metrics on their product should not stop straining but move on to the next level of maturity and begin to measure the effectiveness of their processes.”



What is required for end-to-end test automation?

“With high testability it is possible to run end-to-end test automation and save a lot of time and deliver quality. Companies that run end-to-end manually because of low testability should see it as an enticement to address the testability. Manual end-to-end testing is also extremely expensive and there is a risk that other testing comes secondly, which is not a good thing.”

What is your advice for companies?

“Focus on testability. It gives you a high return of investment. Young companies are fortunate because they have no challenging legacy to consider. They should set up a development plan before they start programming, and determine which features are important, for example, testability, deployment and monitoring of the system, and ensure that the interaction between different parties in the organization is working properly. Older legacy companies have a greater challenge but should begin the change process. It can be done gradually with professional help.”

Do you want to discuss testability with us? Contact Patrik.

Author: Patrik Schalin,  DevOps Specialist at System Verification