The Six Degrees team celebrates their seamless Jakarta upgrade using AutotestPro to do a full automated regression test.
James Bird, Six Degrees ServiceNow Product Owner said "We have successfully implemented the ServiceNow Jakarta update across all four of our instances. AutotestPro has allowed us to build an agile environment to maximise developer resource and reduce internal Customer demand for User Acceptance Testing."
"Reduced testing and sign-off time exponentially"
James went on to say "AutotestPro allowed 6DG to minimise customer impact when upgrading the live environment by reducing the testing and sign-off time exponentially. Now that we have the applicable AutotestPro structure and tests defined, our future upgrades will be completed in a fraction of time and certainly much faster than our previous manual testing approach.’
Test Driven Development Kick-off
Following the celebration, AutotestPro helped the 6DG team kick off the definition of new requirements using the unique AutotestPro Test Driven Development (TDD) features. This will transform the speed at which 6DG can deliver new features to the business in ServiceNow maximising the ROI from the platform.
A full Case Study will be published shortly with the metrics gathered from the Six Degrees experience.
The Head of PMO of a $6bn Global Telecommunications company talked about their experience upgrading from Helsinki to Instanbul using AutotestPro.
How many manual tests did you previously have to run to complete a full regression test?
Around 20 test scenarios were ran manually on key areas of functionality only. The limited coverage was due to the amount of time and resource it took to do the tests - we would like to have done more of course to get better coverage. We had to take a risk that we didn’t miss an important defect and this often kept me awake at night.
How long did your previous manual testing process take and how many people?
Two weeks elapsed with two people full time.
What was your experience of previous upgrades using manual testing?
Our last upgrade using manual testing did not go well. Users reported lots of problems in production and not only did it take a lot of time and effort to get them resolved, but users started to lose faith in the system. We simply couldn’t do another upgrade like that.
Why did you choose AutotestPro?
AutotestPro was the only solution available that produced requirements and acceptance documents that the business could sign-off against before automated tests were ran. This ensured that when the tests ran and passed, we had complete confidence that the system did what the users wanted.
Vitally important for us is security testing to make sure users only see things they have permission to. AutotestPro could log in as various users with different roles and guarantee that the security profiles were working. We found that using impersonation that other automated testing solutions use did not always do this.
The AutotestPro team guided us through the whole process and were brilliant; really impressed with the AutotestPro team; seriously impressed.
How many automated tests now make up your regression test portfolio?
Test coverage increased significantly with the use of AutotestPro. The previous manual tests only covered the key ‘happy path’ scenarios and did not cover any negative tests or different data security profile scenarios. As a result of using AutotestPro and the ability to run many more test very quickly and reliably, test coverage increased by 1000%, and with the length of time taken reduced from 2 weeks to 3.5 hours, roughly a 98% reduction in testing time.
How long does the regression test take to run in AutotestPro?
How many bugs were found that would have previously escaped into production?
30 defects were found that would have previously escaped into production with manual testing. One defect occurred intermittently due to poor configuration in ServiceNow. This would likely have never been found without the repeatability of testing in AutotestPro.
How many defects were found in the new version of ServiceNow having run the regression tests in AutotestPro?
Two critical defects were found when the regression tests were run against the new version of ServiceNow.
One was a ServiceNow defect in editable list views. ServiceNow initially disputed the defect existed, however it was easy to provide evidence to ServiceNow with the screenshots from AutotestPro. As a result the issue was quickly resolved by ServiceNow.
The other defect was resolved in minutes by the development team.using the test results and automatically created defect.
Once these defects were fixed, the full regression test was re-run and passed fully allowing very good confidence and evidence to the change board that the ServiceNow upgrade on Production could be scheduled for the next day.
How many defects have been identified in Production since the upgrade?
No defects have been identified in Production since the upgrade. The user community reported hugely increased satisfaction compared with the last upgrade. Several users commented that the upgrade was so seamless, that they had not even noticed the upgrade had even taken place.
What do you see as the main benefits of AutotestPro?
We no longer need to use valuable resource testing and re-testing our system. The speed and ease at which we can now upgrade and still have complete confidence that the users won’t be affected. And of course, we now have a fully documented system with user guides and system documentation.
Would you use AutotestPro for your next upgrade and recommend it to other companies?
Absolutely. We are going to use AutotestPro for our upgrade to Jakarta and would highly recommend AutotestPro to other companies.
AutotestPro today announced they have relocated their head office from the South West of England to central London.
Paul Chorley, Managing Director of AutotestPro said 'We are delighted to open up our London office at prestigious address in the heart of London near Trafalgar Square. This really demonstrates how far we have come in terms of growing the business and will allow us to better serve our growing international customer base.'