Customer Request
The client initially sought a solution to transfer data from Google Analytics to BigQuery, aiming to bypass the $5,000 monthly cost of Google Analytics 360. An other agency provided a solution that was economical but highly inefficient, running idle 90% of the time and costing $10/day.
Project Goal
Our objective was to overhaul the provided solution to be more efficient, reduce the cloud infrastructure cost significantly, and later assist the client in developing a new version of their platform.
The main challenges we faced:
-
Deciphering and understanding the initial code provided by an other agency.
-
Performing a full refactoring without affecting data integrity.
-
Implementing a flexible system for the addition of new clients.
-
Prioritizing the effective utilization of cloud tools to ensure the minimal cost of cloud infrastructure.
Solution and Technologies
We implemented the solution using Python, Airflow, and a variety of Google Cloud services like BigQuery, Storage, Compute Engine, Cloud Functions, and Pub/Sub.
For visualization and analytics, we utilized Tableau in conjunction with data from Google Analytics. In approximately 20 hours, the solution was refactored, leading to a significant reduction in daily infrastructure costs to roughly $0.25/day.
Conclusions on the Project
Throughout our engagement since May 2020, we successfully overhauled an inefficient system to be more cost-effective without compromising data integrity.
After rectifying the initial issue, we continued to work with Magnezium on the new version of their platform, ensuring a flexible system for client addition and maximizing cloud tool efficiency to minimize costs.
Analytics redesign cuts costs from $10/day to $0.25