Spark EKS Engineer - Remote

Overview

We are seeking a Spark lead focused on operations to administer/scale our multi-petabyte Spark EKS clusters and related services that go with it.

In Short

• Provisioning and ongoing capacity planning for Spark EKS clusters.
• Monitoring and management of Spark EKS platform on AWS.
• Enhance performance of applications running on the platform.
• Maintain large scale production Spark clusters (1000+ nodes).
• Troubleshoot Spark related issues for application teams.
• Implement and administer Spark, Flink & Trino infrastructure.
• Improve scalability, reliability, capacity, and performance of clusters.
• Write automation code for managing Big Data clusters.
• Participate in on-call rotation for incident response.
• Conduct ongoing maintenance across large scale deployments.

Requirements

• AWS - EMR/S3 knowledge, preferably with AWS certification.
• Experience with troubleshooting and debugging Spark jobs.
• Hands-on experience with Spark ecosystem.
• Ability to work with logs to identify issues.
• Experience in managing large scale Big Data clusters.
• Strong understanding of Spark and its components.
• Ability to work in a team and collaborate with engineering teams.
• Experience with automation and scripting.
• Ability to conduct maintenance and improve system performance.
• Experience in incident response and triaging production issues.

Benefits

• Opportunity to work with cutting-edge technology.
• Collaborative work environment.
• Opportunities for professional development.
• Work on large scale deployments across the world.
• Participate in innovative projects.