Senior Data Engineer
spglobal
|
Experience: 6+ Years
|
Hyderabad
|
Posted 6 hours ago
Job Description
Data Pipeline Transition and Platform Delivery
- Partner with the core Databricks team to support the transition of existing data pipelines to the target enterprise data platform.
- Provide hands-on technical guidance for enhancing, stabilizing, and optimizing pipelines after transition, with a focus on performance, scalability, maintainability, operational reliability, and secure AWS deployment patterns.
- Implement and promote repeatable engineering patterns for ingestion, transformation, testing, deployment, monitoring, and support across onboarded datasets.
- Ensure pipelines are designed and operated in a way that supports long-term platform consistency, reliability, observability, and ease of maintenance.
- Support the design and operation of cloud-native data pipelines using AWS services such as:
- Amazon S3 for durable data lake storage
- AWS Glue for data integration, cataloging, and processing
- AWS Lambda for event-driven processing
- Amazon Kinesis for streaming use cases
- AWS Lake Formation for governed data lake controls
- AWS IAM for secure access control
- Promote the use of encryption, environment-level controls, access policies, and platform guardrails to support secure access to platform resources and data products.
- Apply lakehouse technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, and governed data access patterns.
AI-Assisted Engineering and Context Engineering
- Use AI-assisted engineering tools such as Claude, GitHub CoPilot, LLMs, and similar platforms to accelerate data engineering delivery while maintaining enterprise quality and security standards.
- Apply AI tooling to support common engineering activities, including:
- Code generation
- Code refactoring
- SQL and PySpark optimization
- Unit test generation
- Documentation creation
- Pipeline analysis
- Troubleshooting and debugging
- Data quality rule suggestions
- Runbook and operational support documentation
- Apply prompt engineering and context engineering practices to improve the accuracy, relevance, and reliability of AI-generated outputs.
- Structure technical context for AI-assisted work, including requirements, schemas, data contracts, architecture constraints, coding standards, logs, error messages, acceptance criteria, and governance requirements.
- Create or contribute to reusable prompt templates, context packs, and engineering playbooks for common data platform activities such as Databricks migration, AWS data engineering, pipeline troubleshooting, and test generation.
- Review, validate, and refine AI-generated code, recommendations, documentation, and analysis before applying them to production-grade engineering work.
- Ensure responsible use of AI tooling by following enterprise standards for security, privacy, sensitive data protection, compliance, and human review.
Data Onboarding and Asset-Agnostic Enablement
- Support asset-agnostic data onboarding patterns that can accommodate a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case.
- Work with platform, data engineering, architecture, governance, and business-aligned teams to simplify and standardize how data is ingested, transformed, governed, and published to the enterprise platform.
- Help offshore and global teams adopt common onboarding frameworks, technical standards, and delivery practices that improve speed, consistency, and reduce friction.
- Establish reusable ingestion and processing patterns across batch, streaming, and event-driven use cases.
- Create or contribute to reusable technical assets such as:
- Design patterns
- Reference implementations
- Onboarding templates
- Pipeline frameworks
- Technical documentation
- Operational runbooks
- AI-assisted engineering templates
- Support metadata-driven and configuration-driven pipeline patterns to improve scalability, consistency, and maintainability.
Data Mastering Platform Integration
- Support integration of platform pipelines and datasets with the enterprise data mastering platform.
- Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.