Functional And Mobile Compatibility Testing, Delivered Remotely And At Pace

The client was implementing a new foreign exchange website to allow customers to transfer funds overseas. A minimum viable product (MVP) was being delivered for the phase 1 launch, initially targeted at a UK-based audience. As the client had no in-house testing capability, it engaged Ten10 to assist with functional, mobile compatibility and [functional] security testing of the site.

Collaboration with the client project manager and lead developer to understand site architecture and visitor journeys:

• Adoption of an exploratory testing approach to accommodate limited requirements and design documentation and in order to identify issues more quickly.
• First-class collaboration and communication critical to the successful delivery of the project.

The client was looking to develop a new foreign exchange website, initially to serve a UK-based audience but potentially with a view to serve a global audience in the future.

As the website would be supporting a new service, the client was looking to launch a minimum viable product (MVP) initially to monitor demand, gain user feedback and inform future development.

As there was no in-house test capability at the client, it was recognised that they would need to source some external testing expertise to help. There were three particular areas of interest for the client: functional, mobile compatibility and security of the site (from a functional point of view), with the main priority being the functional testing of the site.

Ten10 was initially engaged to carry out a short fixed-term piece of consultancy to advise on the best testing approach for the new site.

During the initial discovery phase, the test manager spent one day with the client’s project manager and lead developer to learn about the website; building an understanding of the site’s customer base and their likely objectives when visiting the site, as well as learning how the site had been developed and understanding the site’s architecture. This was essential preparatory work to determine an appropriate test approach.

Ten10 provided the client with a recommended Test Approach document, which included a comprehensive set of best practice recommendations specific to the client’s website testing project as well as offering advice on performance testing and operational acceptance testing (OAT).

Due to the client resources and timescales available to deliver the project objectives, Ten10 recommended using an exploratory approach to testing, rather than a more traditional, scripted approach. Using exploratory session-based testing would enable more issues to be flushed out and identified more quickly, and so would help to meet the timescales that the client needed to achieve.

Also included in the Test Approach document was advice from Ten10 on the combination of mobile devices, operating systems and browsers that the website should be tested against for compatibility.

This was based on UK market usage of Android and Apple devices to identify the most popular combinations and therefore the minimum to test against, but also included a secondary level of recommended, optional combinations.

Ten10 worked together with the client’s project manager to review the Test Approach and identify which elements were of the highest priority for the business. From this, Ten10 created and defined a detailed statement of work (SOW) that balanced the testing needs for the project with the budget and resource available.

From the approved SOW, Ten10 was then able to build out a comprehensive test plan, capturing the exploratory sessions that would be conducted for the agreed functional and mobile compatibility testing.

During the exploratory testing, Ten10 identified 350 defects. Ten10 assigned an initial priority to each defect in terms of risk to go-live, ranging from urgent to high, medium or low. The client was provided with full access to all defect details so they were able to review the findings and, where relevant, change the priority assigned based on their own in-depth understanding of the planned website usage and the business’ attitude to risk.

In a typical test project, the gap between the total number of defects found and the number of defects closed would be expected to shrink over time. However, in the early reporting provided to the client it was apparent that as defects were continuing to be identified, the defect closure rate was not gaining on the total number of defects found (see figure 1).

Ten10 discussed this finding with the client and agreed to provide an additional report separating the urgent and high priority defects from the medium and low priority defects to enable more granular monitoring of the progress being made to resolve significant defects (see figure 2).

The successful delivery of the project was heavily reliant on regular contact and communication between Ten10’s test team and the client’s in-house development team. Collaborative repositories for exploratory test sessions and defects, setup by Ten10, were central to the efficiency of the project management, as these enabled the client to have full and unrestricted access to real-time information at all times.

To effectively manage remote working by the test team, a stand-up between the test team and the client’s development team was held each morning via Skype. The stand-up gave both teams the opportunity to discuss any project risks or blockers, review a daily burndown of the sessions completed, as well as monitoring the defect closure rate against the plan. This ensured there was complete transparency on the project’s progress but also, importantly, it enabled the project priorities to be reviewed and adjusted in-light of the exploratory test findings.

Feedback from the client project manager was that the daily team stand-ups (via Skype) really minimised the effect of remote working across different locations and time zones. The collaborative repositories for test sessions and defects also facilitated open and up to date communications throughout the project.

The structure of the test sessions repository worked well and gave the client further confidence that all functional areas were being covered and allowed for prioritisation to ensure that the high priority areas were tackled first.

Working with a specialist testing consultancy provided confidence that testing had been successfully performed on the site and gave the client the insight to identify and address the urgent and high priority defects ahead of the site launch. An added benefit for the client was that Ten10 were able to provide some independent usability feedback as a result of its experience testing and using the site.

The client was able to deliver a first-class experience for their users, helping to develop trust and customer loyalty.