Big Data Administrator & Development Engineer
Job Description
Manage and Optimize Big Data Clusters
Ensure the reliable, secure, and high-performance operation of on-premise Hadoop, Kafka, and Druid clusters, proactively monitoring performance and system health.
• Implement and Maintain Monitoring Solutions
Deploy and maintain monitoring and alerting tools such as Cloudera Manager and Splunk to track system availability, detect anomalies, and ensure proactive issue resolution.
• Administer Access and Security Controls
Manage role-based access control (RBAC), encryption, and compliance policies to safeguard data integrity and meet security standards.
• Develop and Automate Data Workflows
Design and implement ETL processes, data transfers, and analysis pipelines using SQL, NiFi, Python, or Spark to support evolving business requirements.
• Support Architectural and Technical Evolution
Collaborate with cross-functional teams on architecture design, system upgrades, and technology evaluations to improve scalability, reliability, and performance.
Required Qualifications
• Minimum 8 years of hands-on experience in administering on-premise Hadoop, Kafka, and Druid clusters.
• Proven expertise with HDFS, Hive, Spark, Hue, and NiFi in enterprise data environments.
• Strong SQL proficiency, including query tuning and performance optimization.
• Solid understanding of Linux system administration and cluster maintenance best practices.
• Experience with automation and scripting tools such as Ansible, Python, or Bash.