Client plans to setup skills, infrastructure, technologies and operations in place to leverage data that it possesses or can harness from public domains with the objective of becoming a data driven organization.

To build a Data Lake having as much relevant information as possible. Hence, the chosen use-case is “CRM Dashboard” in Ninja which essentially ensures that all important datasets are being sent to Data Lake ensuring sufficient completeness of the exercise.

Oneture has relevant expertise and experience in making organizations unlock true value of their data and is interested in partnering with them through this journey.

Oneture has been development partner in this journey; following is the list of use cases/features that we have partnered in implementing for a client

• Building Big data lake for CRM Dashboard
• Customer 360 data for searches and analysis

These diagrams represent a high level proposed architecture and data pipeline:

Following considerations are undertaken to design the solution.

Cloud Infrastructure

• Create S3 directory structure for dev/prod
• Setup encryption/decryption on S3
• Setup S3 lockdown rules
• Setup EMR
• Setup data purge policies using S3 lifecycle
• Cloud Formation Templates/Service Portal
• Data Governance

Data Ingestion

• Operational MongoDB exports to be stored to raw-bucket in S3
• Glue to perform preliminary ETL

Data Storage and Discovery

• S3 part covered as part of cloud infrastructure
• Data Discovery from S3 using Glue

Data Processing

• Create applications for various required use cases
• Fine-tune performance
• Schedule jobs

Data Serving

• Application dashboards
• Application reports
• DB or shared S3 bucket for application results

Following considerations are undertaken to design the solution.

Cloud Infrastructure

• Create S3 directory structure for dev/prod
• Setup encryption/decryption on S3
• Setup S3 lockdown rules
• Setup EMR
• Setup data purge policies using S3 lifecycle
• Cloud Formation Templates/Service Portal
• Data Governance

Data Ingestion

• Operational MongoDB exports to be stored to raw-bucket in S3
• Glue to perform preliminary ETL

Data Storage and Discovery

• S3 part covered as part of cloud infrastructure
• Data Discovery from S3 using Glue

Data Processing

• Create applications for various required use cases
• Fine-tune performance
• Schedule jobs

Data Serving

• Application dashboards
• Application reports
• DB or shared S3 bucket for application results

User Management

• Access roles for various AWS services to access data and perform appropriate tasks
• Data governance by logging AWS services used
• Access roles for various AWS services to access data and perform appropriate tasks
• Data governance by logging AWS services used

• Objective of this solution is to give the customer a faster big-data enabled CRM for carrying out their advanced analytics on their data in addition to regular operational tasks
• Cheaper implementation of new analytical models to improve customer experience by offering the right product to the right customer at the right time and
• Increase operational efficiency by using templates in respective data processes