Senior Databricks Engineer
prodapt
|
Experience: 12+ Years
|
Bangalore
|
Posted 6 hours ago
Job Description
- Design and architect scalable, secure, and highly available enterprise data platforms using Databricks, Snowflake, and AWS Cloud.
- Lead the development and optimization of large-scale ETL/ELT pipelines, supporting both batch and real-time data processing workloads.Design and implement modern data lakehouse architectures and cloud-native data ecosystems.
- Architect and deploy AWS-based data platform components while ensuring compliance with enterprise security, IAM, governance, and networking standards.Build and optimize data ingestion frameworks integrating Databricks APIs, REST APIs, GraphQL APIs, and third-party data sources.
- Develop real-time data processing solutions for business-critical, low-latency use cases using modern streaming technologies.
- Design and implement scalable data models, incorporating both traditional data warehousing methodologies (Kimball, Inmon, Star Schema, Snowflake Schema) and modern semi-structured/unstructured data patterns.
- Collaborate with business stakeholders, architects, data scientists, and engineering teams to define and execute data strategies.Drive performance optimization, cost management, scalability, and reliability initiatives across the data platform landscape.
- Provide technical leadership, architectural governance, code reviews, and mentorship to engineering teams.
- Support AI and Machine Learning initiatives by designing data foundations that enable LLMs, Generative AI, Machine Learning pipelines, and advanced analytics solutions.
- Establish best practices for data engineering, architecture, governance, observability, and platform operations.Prepare and maintain architecture documentation, design standards, and technical roadmaps.