Load testing of a stock trading platform under a huge concurrent user load with rate-limiting capability
Load testing stock trading 2

Client

Client is leading stock broking company providing services like stock broking, financial products distribution, wealth management, and investment banking. They are serving customers ranging from the retail & institutional investors, corporates and High Net-worth Individuals (HNIs)


Financial Markets players needs to protect their trading platforms against the stock market's euphoric highs and depressing lows using risk-based load testing. The Client was under tremendous pressure to get this done, regulator had set a stringent deadline to meet. 

Load testing with 5,00,000 requests per minute using open source JMeter tool leveraging cloud infrastructure


Problem Statement / Opportunity

Client observes 3,50,000 daily active users during business hours and has around a million registered users. Objective was to benchmark the operating capacity of client's platform which includes following:

  • Top 20 APIs of mobile application
  • Top 20 ASP.net based web pages post login
  • Top 20 ASP.net based web pages pre login
  • Top 20 ASP.net based web pages for low bandwidth mobile users
  • Order Placement via web page
  • Order Placement via API

Scalability : Application must be able to handle the load of at least current production scenario of concurrent users and support autoscaling to handle beyond.

Efficiency : Client expectation includes includes the highly efficient application with pre-fixed response time and number of transactions per second.

Fail Over : Application should be able to behave as expected under specified load of concurrent users and should have the Fail Over control.


Proposed Solution & Architecture

We understood the clients requirement and proposed load testing with an objective of crossing 5,00,000 requests per minute to simulate actual production traffic of active users. The load was scaled incrementally to reach peak by iterating in each round with 2 times number of concurrent users from previous round along with required throughput controller configuration, based on the findings of previous tests.

 We performed load testing using JMeter in three stages

  1. Test Preparation
  2. Test Execution
  3. Reporting and Tuning

Technical Challenges & Solution

 During Test Script Preparation:

  • The APIs had multiple dynamic parameters which were derived from previous API calls. We developed custom scripts in JSR223 to cache the values in memory on the fly while running the API
  • For secure website load testing, as the website was tightly coupled asp.net based application, it was not based on API calls. We had to manually monitor to find cookie mechanism and custom headers which were derived from parent pages while calling new page to get correct response.
  • As it is secure, it required unique user to login, so we had to create & import dummy user credentials configuration file to generate the load.
  • Even though each page returned HTTP 200, there were multiple AJAX calls in background, which were called on page load to fetch relevant content to dynamically update the webpage. We had to analyse end to end flow to automate the calls using JMeter for generating load.
  • For Order Placement, the system expected to call twice, once for placement request and another for confirmation. Confirmation request cannot be sent before placement and both the request should be sent within an interval of 30sec. 

During Test Script Execution:

  • Analysing the dry tests, our team observed that using single remote machine for load generators started throttling after crossing 100k requests per min even after rebalancing JVM configuration of Jmeter to get the max utilisation of machine memory
  • Spawned multiple instance copies to horizontally scale the load generators using AWS infrastructure.

 


Value Delivered
  • Client could successfully assess the key bottleneck areas during peak load which included hardware related issues of on-premises servers, load balancer routing algorithm fault, unnecessary DB service calls etc. This load testing helped the client identify and reduce the risk of failures
  • Based on the benchmarking results, client could devise a better BCP/DR strategy for fallback and RTO/RPO objectives.
  • The client has identified potential performance improvements levers to an estimated extent of 20% as compared with pre-benchmarking live environment scenarios
  • This activity fulfilled all the benchmarking objectives as expected by client while addressing and mitigating the challenges faced during the execution and improving overall application performance