MODULE LEAD

• Design, develop, and maintain scalable data ingestion and transformation pipelines using Python and Databricks.

• Implement ETL/ELT processes to ingest data from multiple sources into the Databricks platform.

• Develop PySpark and Python scripts for data extraction, transformation, and loading.

• Work with Delta Lake tables for reliable data storage and processing.

• Implement data processing using Bronze, Silver, and Gold Medallion architecture.

• Develop and optimize complex SQL queries, stored procedures, joins, and window functions.

• Perform data cleansing, validation, deduplication, and reconciliation.

• Implement incremental data loads and handle historical data processing.

• Troubleshoot data pipeline failures and improve job performance.

• Collaborate with cross-functional teams to deliver high-quality data solutions.

Technical Skills & Requirements

1. Python

• Strong knowledge of Python programming.

• Experience with functions, modules, classes, exception handling, and file processing.

• Knowledge of Python data structures and collections.

• Ability to write reusable, modular, and optimized Python code.

• Experience working with JSON, CSV, and other file formats.

2. Data Engineering

• Strong understanding of ETL/ELT concepts and data pipeline development.

• Experience with batch data processing.

• Understanding of incremental and full data loads.

• Knowledge of data validation, data cleansing, and data quality checks.

• Understanding of structured and semi-structured data.

• Basic understanding of data warehouse concepts.

3. Databricks & Data Ingestion

• Hands-on experience with Databricks platform.

• Experience developing data ingestion pipelines using PySpark and Databricks.

• Knowledge of reading and writing data in CSV, JSON, Parquet, and other formats.

• Understanding of Databricks notebooks, jobs, and workflows.

• Experience ingesting data from databases, files, and external sources.

• Understanding of incremental ingestion and schema evolution.

• Knowledge of Bronze, Silver, and Gold layers.

4. SQL

• Strong knowledge of SQL queries and relational databases.

• Hands-on experience with different types of joins:

  • INNER JOIN

  • LEFT JOIN

  • RIGHT JOIN

  • FULL OUTER JOIN

• Experience with subqueries and Common Table Expressions (CTEs).

• Strong understanding of window functions such as ROW_NUMBER, RANK, DENSE_RANK, and LEAD/LAG.

• Experience developing and debugging stored procedures.

• Knowledge of GROUP BY, HAVING, aggregate functions, and CASE statements.

• Basic understanding of data modeling concepts, including fact and dimension tables.

• Ability to analyze and optimize SQL queries.

5. Delta Lake

• Understanding of Delta Lake architecture and capabilities.

• Hands-on experience creating and managing Delta tables.

• Knowledge of Delta Lake operations such as INSERT, UPDATE, DELETE, and MERGE.

• Understanding of ACID transactions.

• Knowledge of schema enforcement and schema evolution.

• Understanding of Delta Lake time travel and history.

• Understanding of OPTIMIZE and VACUUM operations.

  • Knowledge of implementing incremental loads and handling duplicate records.