A Big Data Analytics System Cross-Analyzing 30,000 Attributes With 100x Faster Reporting
Challenge
A leading market research company relied on an established analytical system, but its leadership was convinced the platform would not keep pace with the demands of the years ahead. Data volumes kept climbing, and the business needed to analyze very large datasets far more quickly while running comprehensive advertising-channel analysis across many different markets. The company wanted a forward-looking, innovative replacement rather than incremental patches. Having already settled on the target architecture, it needed a partner with deep hands-on expertise in big-data engineering to carry out a full migration from the legacy solution to the new one without disrupting ongoing operations.
A leading market research company relied on an established analytical system, but its leadership was convinced the platform would not keep pace with the demands of the years ahead. Data volumes kept climbing, and the business needed to analyze very large datasets far more quickly while running comprehensive advertising-channel analysis across many different markets. The company wanted a forward-looking, innovative replacement rather than incremental patches. Having already settled on the target architecture, it needed a partner with deep hands-on expertise in big-data engineering to carry out a full migration from the legacy solution to the new one without disrupting ongoing operations.
Solution
Working hand in hand with the client's business intelligence architects, our big-data engineers implemented an architecture built on Apache Hadoop for storage, Apache Hive for aggregation and querying, and Apache Spark for processing, all running across AWS and Azure. At the client's request the old and new systems operated in parallel throughout the migration. The platform comprised five modules: a Python-based data preparation layer handling more than a thousand raw data types from TV, mobile, web, and surveys; a Hive staging area; two Hive-and-Spark data warehouses performing mapping, cross-source ID linking, and on-the-fly calculations with access-rights filtering; and a WPF and C# desktop analytics client. Built on the MVVM pattern with responsive XAML dashboards, the client application cross-analyzed nearly 30,000 attributes, generated intersection matrices, supported standard and ad hoc reports such as Reach Ranking and Share of Time, and even forecast revenue from expected reach and ad budgets.
Working hand in hand with the client's business intelligence architects, our big-data engineers implemented an architecture built on Apache Hadoop for storage, Apache Hive for aggregation and querying, and Apache Spark for processing, all running across AWS and Azure. At the client's request the old and new systems operated in parallel throughout the migration. The platform comprised five modules: a Python-based data preparation layer handling more than a thousand raw data types from TV, mobile, web, and surveys; a Hive staging area; two Hive-and-Spark data warehouses performing mapping, cross-source ID linking, and on-the-fly calculations with access-rights filtering; and a WPF and C# desktop analytics client. Built on the MVVM pattern with responsive XAML dashboards, the client application cross-analyzed nearly 30,000 attributes, generated intersection matrices, supported standard and ad hoc reports such as Reach Ranking and Share of Time, and even forecast revenue from expected reach and ad budgets.
Results
By the close of the engagement, the new platform executed several queries up to one hundred times faster than the legacy system, dramatically shortening reporting cycles. With the ability to cross-analyze close to 30,000 attributes, the market research company could carry out thorough advertising-channel analysis across a wide range of markets and derive far richer insights than before. The migration preserved continuity thanks to the parallel-run approach, and the modular, cloud-based design left the client with a scalable foundation ready to absorb continued growth in data volume and future analytical needs.
By the close of the engagement, the new platform executed several queries up to one hundred times faster than the legacy system, dramatically shortening reporting cycles. With the ability to cross-analyze close to 30,000 attributes, the market research company could carry out thorough advertising-channel analysis across a wide range of markets and derive far richer insights than before. The migration preserved continuity thanks to the parallel-run approach, and the modular, cloud-based design left the client with a scalable foundation ready to absorb continued growth in data volume and future analytical needs.