Your Role Accountabilities

• Own the end-to-end architecture of the data platform, ensuring its efficiency, cost-effectiveness, security, and governance.

• Collaborate closely with other engineers to design and build an optimal and cost-efficient platform solution.

• Work in partnership with other engineers and managers to design and develop foundational elements of the platform. Create tools and frameworks that enhance data processing, information retrieval, governance, and data quality in a cost-effective and user-friendly manner.

• Develop platform tools and frameworks to the same standard as seen in open-source projects. Engage closely with the engineering community and open-source them upon reaching maturity.

• Implement cutting-edge AI/ML techniques, including Generative AI, to boost the efficiency of the platform and data engineering processes.

• Take on the responsibility for the platform's 24x7 support availability and oversee budgetary concerns, including cloud expenditure.

• Promote a culture of experimentation and data-driven innovation. Inspire and motivate through internal and external presentations and other speaking opportunities.

• Assist in hiring, mentoring, and coaching engineers. Help build an engineering team that prioritizes empathy, diversity, and inclusion.

Qualifications and Experience

• 15+ years of experience with general-purpose programming (e.g. Python, Java, Go), dealing with a variety of data structures, algorithms, and serialization formats

• 12+ years of commendable track record of delivering complex software engineering systems and platforms.

• 12+ years of building and maintaining Tier-1 (require 24*7 uptime and strict SLA) data driven applications.

• 10+ years of relevant experience in designing and implementing large-scale data processing pipelines

• 10+ years of proficiency in big data technologies such as Apache Hadoop, Apache Spark and other distributed computing frameworks such as Apache Spark, Kafka, Airflow, Kubernetes, Cassandra, Druid, Snowflake and more

• 10+ years of experience designing and implementing real-time data processing pipelines and streaming data processing technologies such as Apache Kafka and Apache Flink

• 10+ years of experience to design and implement data solutions using cloud-based services, including storage, compute, and analytics tools. In-depth knowledge of cloud platforms like AWS, Google Cloud or Azure

• 5+ years of Cloud (AWS preferred) experience.

• Experience in building AI/ML driven platform is a huge plus

• Experience in Analytics Tools and Frameworks (such as Looker, Tableau or similar) or building native web and mobile applications (ReactiveJS or similar ) is a huge plus.

• Priority will be given to applicants with prior open-source contributions.

Characteristics and Traits

• Naturally inquisitive, critical thinker, proactive problem-solver, and detail-oriented.

• Positive attitude and an open mind

• Strong organizational skills with the ability to act independently and responsibly

• Self-starter, comfortable initiating projects from design to execution with minimal supervision

• Ability to manage and balance multiple (and sometimes competing) priorities in a fast-paced, complex business environment and can manage time effectively to consistently meet deadlines

• Team player and relationship builder