Why test based on release notes?
Every functional/application administrator knows them, release notes. They are the logical consequence of new releases of applications. They describe bug fixes and new and changed features of your application.
Sooner or later you will encounter them and have to go through them to determine the impact of a new release. After all, every change in software carries visible and invisible risks. Reason enough to determine where the risks are and what needs to be tested.
Impact of release notes
The intention of a new release of an application is, of course, always to improve the software. Think improved functionality or bug fixes, for example. However, as quality expert Rik Marselis points out in Let's Talk About Test, no developer is infallible. Even new releases contain visible and invisible risks and bugs. On top of that, the application vendor doesn't take your setup into account. Let alone how you use the application and what integrations with other applications you have.
You want to find the impact of risks and errors on critical applications, integrations and business processes in a timely manner. That's why reviewing and testing based on release notes is so important.
Pitfalls in testing based on release notes
Application testing based on release notes takes many forms. How you apply this process depends, of course, on your particular situation. Despite the plurality of test forms, we often see three common pitfalls when testing release notes. We have listed them for you:
- A peer organization has already approved the release notes.
Due tolack of testing knowledge, manpower or testing experience, test results from other organizations are used to determine whether a change should be tested. There is a great danger in this. Do you know how well this other organization has tested? Are they using the application in the same way and using the same integrations as your organization?
Adopting test results from other organizations carries enormous risks. It provides false assurance that almost unconsciously slackens the focus on testing the changes. In addition, an auditor or accountant will never agree to such a testing approach. As an organization, you are responsible for testing the new release of the application you are putting into use.
- Weonly test the changed functionality as described in the release notes.
It's good that new functionality is tested. But how can you be sure that new functionality doesn't affect integrations, for example? And how do you test the interrelationships between functionalities? Take your cell phone as an example. You have just updated to the latest iOS or Android version and less than a week later another hotfix follows. This is due to the occurrence of regression bugs in related functionality.
This is why performing a regression test is so important when testing a new release. A good way to do this is to always run through a number of standard regression test scenarios before approving a new release.
- Thenumber of changes is too many to test. We'll hear if something doesn't work.
Happily, we still encounter this in practice. Why this is a big risk is already made clear in the previous two points. Of course, sometimes it is not possible to test all changes. Especially when you have to review and test an average of 1,500 changes each month.
However, applying a beep system (we hear it) is not the right solution. Therefore, it makes sense to adopt a risk-based testing approach. By assessing changes for error probability and impact, you ensure that the attention and testing effort goes to the riskiest changes.
Test tool
Application release testing is not an afterthought. It takes time and knowledge to perform it properly. Of course, it is not possible for every organization to hire test consultants to set up a thorough testing process. A testing tool like Testersuite contains the best practices for testing based on release notes and running regression test scenarios. This helps you set up your application release testing process quickly and properly. Regardless of how mature your organization's testing process is.
Want to know more about this topic? Schedule a no-obligation consultation with us.
