The client needed to automate the process of extracting, converting and loading data into Cloud Object Storage, and then into the final data storage based on DB2.
Converting business logic into Python and Apache Spark code
Created a script and tested it on local data
Created Airflow DAGs to automate data manipulation, dependency compliance, and sequencing
The required logic has been developed and tested, and the process of data delivery to the final storage has been automated