[ Case Study ]
Magnezium: Data Transfer & Cost Optimization
The client initially sought a solution to transfer data from Google Analytics to BigQuery, aiming to bypass the $5,000 monthly cost of Google Analytics 360. An other agency provided a solution that was economical but highly inefficient, running idle 90% of the time and costing $10/day.
Our objective was to overhaul the provided solution to be more efficient, reduce the cloud infrastructure cost significantly, and later assist the client in developing a new version of their platform.
The main challenges we faced:
Deciphering and understanding the initial code provided by an other agency.
Performing a full refactoring without affecting data integrity.
Implementing a flexible system for the addition of new clients.
Prioritizing the effective utilization of cloud tools to ensure the minimal cost of cloud infrastructure.
Solution and Technologies
We implemented the solution using Python, Airflow, and a variety of Google Cloud services like BigQuery, Storage, Compute Engine, Cloud Functions, and Pub/Sub.
For visualization and analytics, we utilized Tableau in conjunction with data from Google Analytics. In approximately 20 hours, the solution was refactored, leading to a significant reduction in daily infrastructure costs to roughly $0.25/day.
Conclusions on the Project
Throughout our engagement since May 2020, we successfully overhauled an inefficient system to be more cost-effective without compromising data integrity.
After rectifying the initial issue, we continued to work with Magnezium on the new version of their platform, ensuring a flexible system for client addition and maximizing cloud tool efficiency to minimize costs.
Analytics redesign cuts costs from $10/day to $0.25