The Head of PMO of a $6bn Global Telecommunications company talked about their experience upgrading from Helsinki to Instanbul using AutotestPro.
How many manual tests did you previously have to run to complete a full regression test?
Around 20 test scenarios were ran manually on key areas of functionality only. The limited coverage was due to the amount of time and resource it took to do the tests - we would like to have done more of course to get better coverage. We had to take a risk that we didn’t miss an important defect and this often kept me awake at night.
How long did your previous manual testing process take and how many people?
Two weeks elapsed with two people full time.
What was your experience of previous upgrades using manual testing?
Our last upgrade using manual testing did not go well. Users reported lots of problems in production and not only did it take a lot of time and effort to get them resolved, but users started to lose faith in the system. We simply couldn’t do another upgrade like that.
Why did you choose AutotestPro?
AutotestPro was the only solution available that produced requirements and acceptance documents that the business could sign-off against before automated tests were ran. This ensured that when the tests ran and passed, we had complete confidence that the system did what the users wanted.
Vitally important for us is security testing to make sure users only see things they have permission to. AutotestPro could log in as various users with different roles and guarantee that the security profiles were working. We found that using impersonation that other automated testing solutions use did not always do this.
The AutotestPro team guided us through the whole process and were brilliant; really impressed with the AutotestPro team; seriously impressed.
How many automated tests now make up your regression test portfolio?
Test coverage increased significantly with the use of AutotestPro. The previous manual tests only covered the key ‘happy path’ scenarios and did not cover any negative tests or different data security profile scenarios. As a result of using AutotestPro and the ability to run many more test very quickly and reliably, test coverage increased by 1000%, and with the length of time taken reduced from 2 weeks to 3.5 hours, roughly a 98% reduction in testing time.
How long does the regression test take to run in AutotestPro?
How many bugs were found that would have previously escaped into production?
30 defects were found that would have previously escaped into production with manual testing. One defect occurred intermittently due to poor configuration in ServiceNow. This would likely have never been found without the repeatability of testing in AutotestPro.
How many defects were found in the new version of ServiceNow having run the regression tests in AutotestPro?
Two critical defects were found when the regression tests were run against the new version of ServiceNow.
One was a ServiceNow defect in editable list views. ServiceNow initially disputed the defect existed, however it was easy to provide evidence to ServiceNow with the screenshots from AutotestPro. As a result the issue was quickly resolved by ServiceNow.
The other defect was resolved in minutes by the development team.using the test results and automatically created defect.
Once these defects were fixed, the full regression test was re-run and passed fully allowing very good confidence and evidence to the change board that the ServiceNow upgrade on Production could be scheduled for the next day.
How many defects have been identified in Production since the upgrade?
No defects have been identified in Production since the upgrade. The user community reported hugely increased satisfaction compared with the last upgrade. Several users commented that the upgrade was so seamless, that they had not even noticed the upgrade had even taken place.
What do you see as the main benefits of AutotestPro?
We no longer need to use valuable resource testing and re-testing our system. The speed and ease at which we can now upgrade and still have complete confidence that the users won’t be affected. And of course, we now have a fully documented system with user guides and system documentation.
Would you use AutotestPro for your next upgrade and recommend it to other companies?
Absolutely. We are going to use AutotestPro for our upgrade to Jakarta and would highly recommend AutotestPro to other companies.
AutotestPro today announced they have relocated their head office from the South West of England to central London.
Paul Chorley, Managing Director of AutotestPro said 'We are delighted to open up our London office at prestigious address in the heart of London near Trafalgar Square. This really demonstrates how far we have come in terms of growing the business and will allow us to better serve our growing international customer base.'
Whilst giving a demonstration of AutotestPro to a prospect recently, we explained how we put the business processes at the very centre of the software delivery life-cycle and how the magic starts when you do this. Suddenly everything becomes very easy and very automatic.
When you step back and think about it, every stage of the software delivery life-cycle from requirements, through design, development, testing and release, acts in some way on the business processes. The processes the user will perform are after all, the reason why the system is being built!
The common thing throughout every stage of the life-cycle are the business processes. Additional information is added to those processes as they move through the lifecycle:
Software delivery without business processes at the centre, is like having a bicycle wheel with no central hub for the spokes to attach to. A wheel with no centre is weak, even useless and would make cycling very difficult or impossible.
A wheel with business processes at the centre is strong, fast and efficient. Software delivery is strong, fast and efficient when working with a solution that puts business processes at the centre.
Interestingly, the first wheel was invented in 3500 BC, but it wasn't until 1,500 years later, that the first spoked wheel was invented.
According to Wikipedia, the SDLC came into existence in the 1960s. Only 56 years later did AutotestPro apply for a patent for the idea of putting business processes at the centre of the software development lifecycle.
Success Secret 1#
Embrace Agile and use it correctly; don't use it as an excuse for not doing things e.g. documentation. Documentation should be produced of requirements, design and acceptance criteria, process documentation etc.
Success Secret #2
Ensure teams are co-located either physically or via IT systems and the Product Owner is part of the team. A collaboration environment which allows all team members to see requirements, development tasks, test plans and results, documentation etc all in one place is essential to success.
Success Secret #3
The team and Product Owner must be empowered to make decisions. It is all about getting decisions made quickly. Try and fail fast is the key to success.
Question 4 - Budget holders are nervous and using Agile because seemingly the budget is unconstrained. How do you manage this concern?
The prioritisation of requirements at the beginning of the project is key here. Once a budget holder is educated as to how the requirements are identified and prioritised then those concerns are often ameliorated.
The MoSCow approach allows the business to see that the most business critical requirements are delivered first and at a second level of prioritisation, the highest risk features are delivered early. If they highest priority and highest risk features are delivered first then at that stage generally the contingency and resource is available to deliver without impacting the overall project.
That last thing that any business wants to see is high priority features delivered at the end of the project when the budget is getting thin.
Another key approach here to reassure the business is focusing the testing of the platform from a requirements perspective, directly linking the testing of the system to the defined business requirements. It is all well and good providing and running test scripts that focus on the underlying technical configuration, that will prove that the code works, but does it prove that the business requirement has been met?
Question 3 - How do you avoid an Agile project being open ended? Surely if the requirements are defined as you go along there is a risk that you will never finish?
All projects tend to have a defined budget and a limited amount of resource. Generally the business case for a project will have already defined the high level features of an application and these will be defined and scoped at a very early stage using the MoSCoW (Must Should Could Would) methodology.
In a project with constrained resource and budget the emphasis will be to deliver all of the Musts, and then focus in order on the Should haves, the Could haves and the Would like to haves. The benefit of using Agile is that you are much more likely to get more of the features at the lower priority end of the spectrum that if you were using a waterfall methodology.
The product owners do need to take responsibility though for communicating at each incremental release what is and isn’t included. It is human nature that the requirements that you have identified are more important than other people’s and the absence of someone’s pet requirement can be emotional.
However a project cannot be open ended and in general, unless budgets are unlimited, not all of the Would like to haves will be delivered.
,A great article on https://www.edureka.co/blog/devops-tutorial showing the plethora of tools required to make DevOps a reality, got us thinking about how much simpler it is for ServiceNow users using AutotestPro.
19 tools required VS just 2
ServiceNow's Automated Test Framework module (ATF) was introduced in the Istanbul version of ServiceNow as part of the platform.
Many people came to our stand at NowForum London confused as to the pros and cons of each tool and ask us to explain what the differences are. There are many many differences but the fundamental one and the easiest to explain quickly, is that ATF does automated testing, whereas AutotestPro enables full life-cycle DevOps automation, including automated testing (and from a business scenario/user story perspective).
There are many other differences between the two tools, for example the underlying technology used, but too many to go into in detail, in a short blog article such as this.
Still confused and need more guidance? Contact us and ask for our test automation experts to take you through all the differences in detail.
Absolutely not and this is the biggest misconception that people have with Agile. It is true that Agile promotes conversations and collaboration over documentation but documentation and, more specifically, traceability is key. The difference here is that the documentation is done using a completely different approach to the traditional cycle of requirements capture, high level design, low level design and user documentation.
One of the key principles in Agile is using user stories which are direct pointers to a conversation.
So when you hear people say "no documentation" it is an interpretation of a different process. With an Agile approach you don't spend a lot of time writing design documentation and getting sign off from the business. Your requirements are all defined as acceptance criteria (AC) in user stories agreed at the time with the product owner.
A key principle here therefore is to make sure that user stories up to date with any decisions made and there is an audit trail to those decisions and decision makers. It is a challenge to try and track down a change or what a feature should be doing if the Acceptance Criteria is not kept up-to-date.
This is another area where ServiceNow excels. Use the system that you are developing to track the user stories being implemented.
To release features you still need user guides and manuals and any configuration that needs to be recorded centrally. So as a definition of done for a delivered feature it should always include a guide or manual. Herein lies a challenge in keeping those guides up to date.
The approach we have taken is to create the user guides as the testing is being done. Automation is key here, if the testing is automated then generating automated user guides is a logical step forward so that the user guide is automatically fully in line with the currently released version. This is one of the first features that we developed because we saw the link between Agile development and the need for constantly updated user guides, this is something very few people do successfully.
AutotestPro kicks off NowForum London by announcing that Paul Cash, co-founder of Partners in IT and former Managing Director of Fruition Partners UK, is joining AutotestPro on the next stage of their journey.
Paul Cash said "I'm delighted and excited to be involved with Autotestpro as a strategic advisor. Over the past two years the company has developed an innovative and powerful framework to enable Automated Testing and DevOps within the ServiceNow platform."
Paul Chorley, Managing Director of AutotestPro said, "We are are absolutely delighted to have someone of Paul's caliber join the team. His experience and knowledge of the ServiceNow market will be invaluable in helping us help more ServiceNow customers transform their use of ServiceNow with our solution."