Senior Data Engineer

spglobal | Experience: 6+ Years | Hyderabad | Posted 6 hours ago

Job Description

Data Pipeline Transition and Platform Delivery

  • Partner with the core Databricks team to support the transition of existing data pipelines to the target enterprise data platform.
  • Provide hands-on technical guidance for enhancing, stabilizing, and optimizing pipelines after transition, with a focus on performance, scalability, maintainability, operational reliability, and secure AWS deployment patterns.
  • Implement and promote repeatable engineering patterns for ingestion, transformation, testing, deployment, monitoring, and support across onboarded datasets.
  • Ensure pipelines are designed and operated in a way that supports long-term platform consistency, reliability, observability, and ease of maintenance.
  • Support the design and operation of cloud-native data pipelines using AWS services such as:
    • Amazon S3 for durable data lake storage
    • AWS Glue for data integration, cataloging, and processing
    • AWS Lambda for event-driven processing
    • Amazon Kinesis for streaming use cases
    • AWS Lake Formation for governed data lake controls
    • AWS IAM for secure access control
  • Promote the use of encryption, environment-level controls, access policies, and platform guardrails to support secure access to platform resources and data products.
  • Apply lakehouse technologies and concepts such as Delta Lake, Apache Iceberg, Databricks Unity Catalog, metadata-driven pipelines, and governed data access patterns.

 

AI-Assisted Engineering and Context Engineering

  • Use AI-assisted engineering tools such as Claude, GitHub CoPilot, LLMs, and similar platforms to accelerate data engineering delivery while maintaining enterprise quality and security standards.
  • Apply AI tooling to support common engineering activities, including:
    • Code generation
    • Code refactoring
    • SQL and PySpark optimization
    • Unit test generation
    • Documentation creation
    • Pipeline analysis
    • Troubleshooting and debugging
    • Data quality rule suggestions
    • Runbook and operational support documentation
  • Apply prompt engineering and context engineering practices to improve the accuracy, relevance, and reliability of AI-generated outputs.
  • Structure technical context for AI-assisted work, including requirements, schemas, data contracts, architecture constraints, coding standards, logs, error messages, acceptance criteria, and governance requirements.
  • Create or contribute to reusable prompt templates, context packs, and engineering playbooks for common data platform activities such as Databricks migration, AWS data engineering, pipeline troubleshooting, and test generation.
  • Review, validate, and refine AI-generated code, recommendations, documentation, and analysis before applying them to production-grade engineering work.
  • Ensure responsible use of AI tooling by following enterprise standards for security, privacy, sensitive data protection, compliance, and human review.

 

Data Onboarding and Asset-Agnostic Enablement

  • Support asset-agnostic data onboarding patterns that can accommodate a broad range of data assets, domains, and source systems without requiring bespoke platform redesign for each use case.
  • Work with platform, data engineering, architecture, governance, and business-aligned teams to simplify and standardize how data is ingested, transformed, governed, and published to the enterprise platform.
  • Help offshore and global teams adopt common onboarding frameworks, technical standards, and delivery practices that improve speed, consistency, and reduce friction.
  • Establish reusable ingestion and processing patterns across batch, streaming, and event-driven use cases.
  • Create or contribute to reusable technical assets such as:
    • Design patterns
    • Reference implementations
    • Onboarding templates
    • Pipeline frameworks
    • Technical documentation
    • Operational runbooks
    • AI-assisted engineering templates
  • Support metadata-driven and configuration-driven pipeline patterns to improve scalability, consistency, and maintainability.

 

Data Mastering Platform Integration

  • Support integration of platform pipelines and datasets with the enterprise data mastering platform.
  • Collaborate with upstream and downstream stakeholders to ensure mastered data can be consumed reliably through standardized interfaces and governed data flows.

⚡ Hot DevOps & Cloud Jobs Trending Now

SRE
Sr. SRE Engineer
Stripe | Bangalore
DEV
Backend Developer
Coinbase | Remote
Infra
Cloud Infra Lead
Datadog | Pune
ML
MLOps Architect
Anthropic | Hyderabad
SRE
Sr. SRE Engineer
Stripe | Bangalore
DEV
Backend Developer
Coinbase | Remote
Infra
Cloud Infra Lead
Datadog | Pune
ML
MLOps Architect
Anthropic | Hyderabad