Senior Data Engineer
Job Description
1. AWS Cloud Engineering :
- Work with AWS services such as S3, Glue, Lambda, Redshift, and CloudWatch to build and operate data solutions.
- Apply appropriate security, access-control, and cloud data management practices.
- Optimize cloud workloads for performance, reliability, and cost.
2. Data Ingestion & Pipeline Engineering :
- Design and implement robust data ingestion pipelines for structured and unstructured data sources.
- Build production-grade ingestion frameworks to reliably process and move data into GCP or similar storage.
- Develop scalable and fault-tolerant data pipelines with appropriate validation, monitoring, and error handling.
- Optimize data ingestion workflows for performance, reliability, and cost.
- Ensure data pipelines maintain data quality, integrity, and traceability throughout the ingestion process.
3. Python Development & Automation :
- Develop production-grade data engineering applications and automation using Python.
- Build reusable libraries, services, and utilities to support data ingestion and processing workflows.
- Automate infrastructure and data engineering processes to improve operational efficiency.
- Implement appropriate logging, monitoring, error handling, and testing for Python-based services and pipelines.
4. Collaboration & Ownership :
- Work closely with Data Engineers, Software Engineers, Cloud/Infrastructure Engineers, Security teams, and business stakeholders.
- Take ownership of technical initiatives from architecture and design through production deployment and ongoing support.