Key Responsibilities

* Design, develop, and implement Databricks solutions to support data integration, analytics, and business intelligence requirements.
* Collaborate with data architects, data scientists, and data analysts to understand data requirements and to ensure Databricks solutions meet business needs.
* Manage and optimize Databricks environments, including performance tuning, scaling, and cost management.
* Develop and maintain CI/CD pipelines for Databricks projects using tools like Jenkins, GitHub Actions, or Azure DevOps.
* Implement infrastructure as code (IaC) solutions using Terraform to automate the deployment and management of Databricks resources on Azure Cloud.
* Perform data cleansing, transformation, and integration tasks to ensure data quality and integrity within Databricks.
* Integrate third-party application data into Databricks, ensuring seamless data flow and compatibility.
* Monitor and troubleshoot Databricks jobs, clusters, and workflows, and provide technical support and guidance to other team members.
* Evaluate and recommend new tools, technologies, and best practices to enhance Databricks implementations and data management processes.
* Develop and maintain documentation for Databricks solutions, including architecture diagrams, data flow diagrams, and technical specifications.
* Stay current with industry trends, emerging technologies, and best practices in Databricks, data engineering, and data management.

Qualifications

* Bachelor's or master’s degree in Computer Science, Information Systems, or a related field.
* Minimum of 5 years of overall experience in software engineering, with at least 3 years of hands-on experience with Databricks application.
* Extensive experience in CI/CD processes and tools, such as Jenkins, GitHub Actions, or Azure DevOps.
* Proficiency in managing Databricks on Azure Cloud, including performance tuning, scaling, and cost management.
* Strong knowledge of data integration, transformation, and analytics processes within Databricks.
* Proficiency in programming languages, such as SQL, Python, or Scala.
* Familiarity with data modeling, data quality, and data governance concepts and best practices.
* Familiarity with Terraform IaC is beneficial.
* Excellent problem-solving and analytical skills.
* Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.