Senior Data Engineer

se | Experience: 8+ Years | Bangalore | Posted 5 hours ago

Job Description

Design and implement enterprise-scale data architectures aligned with business, analytics, and technology objectives.
Architect scalable Data Lake, Data Lakehouse, and Enterprise Data Warehouse solutions on AWS.
Define architecture principles, engineering standards, reusable patterns, and development best practices.
Design multi-layer data architectures such as Raw, Standardised, Curated, and Consumption layers.
Evaluate technical requirements and select appropriate AWS services, data formats, processing engines, and orchestration patterns.
Translate complex business requirements into scalable, maintainable, and production-ready technical solutions.
Participate in architecture reviews and provide technical direction for data platform initiatives.
Evaluate emerging data technologies and recommend platform improvements based on business value, scalability, security, and cost.

Architect and develop end-to-end data solutions using AWS-native services.
Build scalable frameworks for batch, near-real-time, event-driven, and streaming data processing.
Design secure, highly available, fault-tolerant, and cost-optimised cloud data platforms.
Build metadata-driven and configuration-driven ingestion and transformation frameworks.
Design solutions for loading data from databases, APIs, files, message queues, SaaS applications, and enterprise platforms.
Implement orchestration, dependency management, retries, restartability, error handling, and operational recovery capabilities.
Establish monitoring, alerting, logging, auditability, and observability standards for production pipelines.
Optimise storage, processing, and query performance across AWS data services.

Design and implement modern Data Lakehouse architectures using technologies such as:
- Apache Iceberg
- Delta Lake
- Apache Parquet
Define appropriate partitioning, compaction, data retention, schema evolution, and file optimisation strategies.
Design enterprise and dimensional data models using:
- Fact and dimension modelling
- Star and snowflake schemas
- Slowly Changing Dimensions
- Data Vault modelling
Create scalable ETL and ELT frameworks supporting enterprise reporting, analytics, self-service BI, data science, and AI workloads.
Design data models at the correct business grain and establish reliable relationships across multiple source systems.
Implement history management, incremental processing, change data capture, late-arriving data handling, reconciliation, and audit controls.
Enable both batch and near-real-time data consumption while maintaining data consistency and traceability.

Develop complex production-grade data pipelines using Python, PySpark, SQL, AWS Glue, and related AWS services.
Process high-volume structured, semi-structured, and unstructured datasets.
Design efficient transformation logic capable of handling large-scale distributed workloads.
Develop reusable components for ingestion, transformation, validation, reconciliation, exception handling, and operational monitoring.
Review code and ensure compliance with engineering, security, performance, and maintainability standards.
Implement unit testing, integration testing, regression testing, and automated data validation.
Diagnose and resolve complex data processing, performance, stability, and production issues.

Develop and optimise complex PySpark workloads for large-scale data processing.
Analyse Spark execution plans and identify performance bottlenecks.
Apply appropriate optimisation techniques, including:
- Partitioning and repartitioning
- Broadcast joins
- Join strategy optimisation
- Data-skew handling
- Predicate pushdown
- Partition pruning
- Caching and persistence
- Memory and executor optimisation
- Small-file management
Troubleshoot failed or slow-running Spark jobs and implement sustainable fixes.
Select suitable processing patterns based on data size, workload behaviour, latency, and cost requirements.