Stock trading and Algo trading requires proper historical market data for technical analysis. The volume of live market data being received in Indian capital market is around 10GB per day across instruments including equity, derivatives, commodity and currency.

Our client was looking for a technology partner who could conceptualize and build a highly scalable and robust architecture, in a cost-effective manner. The solution should allow their platform i.e their web and mobile app as well as API and SDK users i.e retail algo traders or external partners to consume the OHLC candle data exposed via both API and WebSocket for their requirements.

Oneture put in place a team of engineers including a Solution Architect along with backend developers to evaluate limitations of existing system, transform and re-architect the existing system to cloud based scalable platform to cater client needs.

Existing system of client had below limitations:

1. Incapable to store and process tick by tick data of past 10 years
2. Unable to scale to provide 1 second interval OHLC candle data which is rarely provided by Indian market capital brokers.
3. Delayed response time due to on-demand computation instead of realtime computation of OHLC candles data
4. Data consumed only by pull based mechanism i.e API call

The new architecture addressed each and every points listed above, to make the system capable of providing OHLC candle data aggregated in realtime across all channels for intervals including 1 sec, 1 min, 5 mins, 30 mins and 1 day via API as well as WebSocket.

The new architecture proposed by us (inspired by our data reference solution) consists of a Kafka based system which listens to all the market feeds of various exchanges being received via the UDP Ports and aggregates the feed as per the intervals and OHLCV logic and pushes this aggregated OHLCV candle data to AWS ElasticCache (Redis) for memory caching. The cached data is migrated to AWS RDS at the End of Day. The API queries the data from Redis or RDS, based on the time interval for which data has been queried. For intraday data, the API fetches data from Redis and for past dated historical data, it queries data from RDS.

Key Components of the application :

• MSK Producer - This component listens to feeds of multiple exchanges via UDP ports and then pre process the feeds and pushes these feeds to different Kafka Topics. The number of topics is equal to the number of exchanges present
• MSK Consumer - This component consists of processes that read the feed messages from these Kafka topics and aggregates them based on the desired interval and stores this data to Redis on the completion of the interval.
• Database - The intraday stored in Redis is migrated to AWS RDS at the end of day.  The database consists of many tables which are segregated as per interval of data and the exchanges. This segregation of data in various tables eliminates any extra aggregation at the API layer and hence reduces the response time of the API responses.
• API Layer - A lightweight flask based API to query Redis or RDS based on the time interval for which data has been queried.

• Store and manage TBs of data natively in cloud
• Realtime computation of each interval for all contracts across all 5 segment exchanges (NSE, BSE, NFO, MCX, CDNSE) as compared to EOD computation of existing system opening doors for Algo Trading model integration to provide faster outcomes and maximizing trader profits
• Providing second level data for intraday and historical
• Better accuracy result of OHLC as compared to competitors
• Average response time of API less than 200ms
• Autoscale during high traffic