Early Optimisation Of Systems Through Early Performance Testing

The client is a leading global professional services firm. Its aim is to help organisations and individuals create the value they are looking for by delivering quality in assurance, tax and advisory services. The client was in the process of developing a new Software-as-a-Service (SaaS) solution that would be used by local councils to offer services to the public, with the additional aim of building best processes and practices.

The client was actively developing the solution during Ten10’s involvement. An external supplier managed the infrastructure on which the solution was to be deployed and made available a number of virtual environments for development and testing.

The development team also saw performance as a key success criterion. The plan was that one specific council would be the initial pilot user, although the system was designed to support concurrent use by multiple councils in the future.

The solution relied on the Drupal CMS for the frontend and on MuleSoft ESB for the integration with third-party and client services.

ForgeRock was used for authentication purposes. The solution also employed proxy and caching technologies, such as Varnish and Memcached, to improve server performance.

The client was looking to performance test the solution in order to ensure that the production system would be able to handle expected and peak loads while identifying the capacity of the application under gradually increasing load. Initially, traffic would be coming from the piloting council, but the system was designed to handle up to five councils concurrently.

We needed to understand the performance, stability and capacity of the CMS component of the solution, with expected peak user volumes of 175 concurrent users and a peak load of 300 concurrent users.

Specific performance metrics for the system were that 80% of all server transaction times should be under three seconds under peak load, that average CPU utilisation was under 70%, and that the time taken for the enterprise service bus to process a message was under 40ms end-to-end under peak load.

Further, we wanted to investigate the capacity of the application by gradually increasing the load until the system became unresponsive or the load generation servers become saturated. Following the system stress testing, we wanted to analyse the results, diagnose potential performance issues, recommend actions, and investigate the effect of the identified solutions.

Among the challenges were the complexity of the system and the early stage of the project when environments and system features were not always stable enough to undergo performance testing.

We integrated with the team early on, gathering requirements and aligning the performance testing approach to the development effort.

To overcome issues with features not being delivered on time, environments being unstable and limit delays and disruptions in testing, we followed an agile approach by adjusting load tests to the existing stable functionality. This allowed the early provision of feedback to the development team, the quick discovery and efficient diagnosis of performance issues, and the prompt resolution of bugs.

Ten10 integrated well with the development team and the load test rig was well provisioned.

Delivery was massively improved due to our extensive analysis and our recommended changes to improve performance. For example, based on our analysis, we recommended a configuration change that improved the system’s response times by about 40%, as highlighted in the before and after transaction diagrams shown earlier. We also provided our scripts to the client to allow them a repeatable execution, so that they could validate additional fixes using the Ten10 developed performance test assets, both load test scripts and reporting capability.