The Head of PMO of a $6bn Global Telecommunications company talked about their experience upgrading from Helsinki to Instanbul using AutotestPro.
How many manual tests did you previously have to run to complete a full regression test?
Around 20 test scenarios were ran manually on key areas of functionality only. The limited coverage was due to the amount of time and resource it took to do the tests - we would like to have done more of course to get better coverage. We had to take a risk that we didn’t miss an important defect and this often kept me awake at night.
How long did your previous manual testing process take and how many people?
Two weeks elapsed with two people full time.
What was your experience of previous upgrades using manual testing?
Our last upgrade using manual testing did not go well. Users reported lots of problems in production and not only did it take a lot of time and effort to get them resolved, but users started to lose faith in the system. We simply couldn’t do another upgrade like that.
Why did you choose AutotestPro?
AutotestPro was the only solution available that produced requirements and acceptance documents that the business could sign-off against before automated tests were ran. This ensured that when the tests ran and passed, we had complete confidence that the system did what the users wanted.
Vitally important for us is security testing to make sure users only see things they have permission to. AutotestPro could log in as various users with different roles and guarantee that the security profiles were working. We found that using impersonation that other automated testing solutions use did not always do this.
The AutotestPro team guided us through the whole process and were brilliant; really impressed with the AutotestPro team; seriously impressed.
How many automated tests now make up your regression test portfolio?
Test coverage increased significantly with the use of AutotestPro. The previous manual tests only covered the key ‘happy path’ scenarios and did not cover any negative tests or different data security profile scenarios. As a result of using AutotestPro and the ability to run many more test very quickly and reliably, test coverage increased by 1000%, and with the length of time taken reduced from 2 weeks to 3.5 hours, roughly a 98% reduction in testing time.
How long does the regression test take to run in AutotestPro?
How many bugs were found that would have previously escaped into production?
30 defects were found that would have previously escaped into production with manual testing. One defect occurred intermittently due to poor configuration in ServiceNow. This would likely have never been found without the repeatability of testing in AutotestPro.
How many defects were found in the new version of ServiceNow having run the regression tests in AutotestPro?
Two critical defects were found when the regression tests were run against the new version of ServiceNow.
One was a ServiceNow defect in editable list views. ServiceNow initially disputed the defect existed, however it was easy to provide evidence to ServiceNow with the screenshots from AutotestPro. As a result the issue was quickly resolved by ServiceNow.
The other defect was resolved in minutes by the development team.using the test results and automatically created defect.
Once these defects were fixed, the full regression test was re-run and passed fully allowing very good confidence and evidence to the change board that the ServiceNow upgrade on Production could be scheduled for the next day.
How many defects have been identified in Production since the upgrade?
No defects have been identified in Production since the upgrade. The user community reported hugely increased satisfaction compared with the last upgrade. Several users commented that the upgrade was so seamless, that they had not even noticed the upgrade had even taken place.
What do you see as the main benefits of AutotestPro?
We no longer need to use valuable resource testing and re-testing our system. The speed and ease at which we can now upgrade and still have complete confidence that the users won’t be affected. And of course, we now have a fully documented system with user guides and system documentation.
Would you use AutotestPro for your next upgrade and recommend it to other companies?
Absolutely. We are going to use AutotestPro for our upgrade to Jakarta and would highly recommend AutotestPro to other companies.