We understood the client’s requirement and proposed load testing with an objective of crossing 5,00,000 requests per minute to simulate actual production traffic of active users. The load was scaled incrementally to reach peak by iterating in each round with 2 times number of concurrent users from previous round along with required throughput controller configuration, based on the findings of previous tests.

 We performed load testing using JMeter in three stages

1. Test Preparation
2. Test Execution
3. Reporting and Tuning

Technical Challenges & Solution

 During Test Script Preparation:

• The APIs had multiple dynamic parameters which were derived from previous API calls. We developed custom scripts in JSR223 to cache the values in memory on the fly while running the API
• For secure website load testing, as the website was tightly coupled asp.net based application, it was not based on API calls. We had to manually monitor to find cookie mechanism and custom headers which were derived from parent pages while calling new page to get correct response.
• As it is secure, it required unique user to login, so we had to create & import dummy user credentials configuration file to generate the load.
• Even though each page returned HTTP 200, there were multiple AJAX calls in background, which were called on page load to fetch relevant content to dynamically update the webpage. We had to analyse end to end flow to automate the calls using JMeter for generating load.
• For Order Placement, the system expected to call twice, once for placement request and another for confirmation. Confirmation request cannot be sent before placement and both the request should be sent within an interval of 30sec. 

During Test Script Execution:

• Analysing the dry tests, our team observed that using single remote machine for load generators started throttling after crossing 100k requests per min even after rebalancing JVM configuration of Jmeter to get the max utilisation of machine memory
• Spawned multiple instance copies to horizontally scale the load generators using AWS infrastructure.

• Client could successfully assess the key bottleneck areas during peak load which included hardware related issues of on-premises servers, load balancer routing algorithm fault, unnecessary DB service calls etc. This load testing helped the client identify and reduce the risk of failures
• Based on the benchmarking results, client could devise a better BCP/DR strategy for fallback and RTO/RPO objectives.
• The client has identified potential performance improvements levers to an estimated extent of 20% as compared with pre-benchmarking live environment scenarios
• This activity fulfilled all the benchmarking objectives as expected by client while addressing and mitigating the challenges faced during the execution and improving overall application performance