The customer required a data processing system with loading into the intermediate storage (AWS S3), and then to permanent data storage (AWS Redshift).
Determined the optimal amount of resources for AWS Glue
Created different kinds of AWS Glue Jobs: based on Apache Spark and Python Shell Jobs using Awswrangler
Developed data orchestration using AWS StepFunctions
Local testing using Docker
Created AWS StepFunctions to automate data manipulation
Automated data extraction, conversion and loading system created, tested and handed over to the customer