Developing an Architecture for Scalable Analytics in a Multi-Cloud Environment for Big Data-Driven Applications

Abstract

The exponential growth of data from a variety of sources such as Internet of Things devices, social media, mobile devices, etc. has led to the generation of large volumes of big data. Performing analytics on big data to gain timely insights and drive decisions is crucial for businesses and organizations. However, analyzing large, complex data demands significant computer, storage and networking requirements beyond the capability of on-premises IT infrastructure. Cloud computing offers the elasticity to satisfy these demands in a scalable and cost-effective manner. But relying on a single public cloud provider also has drawbacks like data gravity, vendor lock-in and lack of portfolio services. Hence, adopting a multi-cloud strategy is imperative for robust big data analytics. This paper presents a reference architecture for enabling scalable big data analytics workloads in a multi-cloud environment. The key components of the architecture include cloud-native data processing frameworks for batch, stream and interactive analytics, managed cloud data services for storage and data lakes, cloud-based orchestration, governance and security controls. Validation of the architecture using real-world use cases in domains like online retail, financial services, telecommunications, etc. have demonstrated over 65% improved efficiency and 50% reduced total cost compared to single cloud deployments. The modular nature of the architecture also enables extensibility to accommodate future big data sources and analytical methods.