As an ESG data engineer, you will support our users by bringing the required data into Snowflake.
1. DATA RETRIEVAL
Data will be retrieved from http://data.footprintnetwork.org/#/api
2. DESIGN PIPELINE ARCHITECTURE
Propose a high-level AWS architecture schema for the ingestion pipeline.
Requirements:
• Triggering system (schedule, API, …)
• Raw data must be stored in a S3 bucket
• Data quality control
• Monitoring system
• Target DB: PostgreSQL
Decouple the pipeline operations as much