Remote Otter LogoRemoteOtter

Spark EKS Engineer (AWS Big Data Platform) - Remote

Posted 2 days ago

Overview

The Spark EKS Engineer will be responsible for administering and scaling multi-petabyte Spark EKS clusters and related services on AWS, focusing on performance enhancement, troubleshooting, and managing large-scale global deployments.

In Short

  • Administer and scale multi-petabyte Spark EKS clusters on AWS.
  • Perform provisioning, capacity planning, monitoring, and management of Spark EKS platforms.
  • Enhance the performance of applications and middleware.
  • Troubleshoot and resolve Spark-related issues.
  • Implement and maintain Spark, Flink, and Trino infrastructure.
  • Improve scalability, service reliability, capacity, and performance.
  • Triage production issues in collaboration with teams.
  • Conduct ongoing maintenance for global deployments.
  • Write automation code for managing Big Data clusters.
  • Participate in on-call rotations.

Requirements

  • 5+ years of experience managing large-scale Spark production clusters.
  • Strong expertise in AWS services (EMR, S3, EKS).
  • Hands-on experience with Hadoop and Spark ecosystems.
  • Proficiency in Python for development and automation.
  • Experience with configuration management and automation tools.
  • Deep understanding of Spark ecosystem and troubleshooting techniques.
  • Ability to analyze logs and debug issues.
  • Strong organizational skills focused on delivering results.
  • Excellent communication and collaboration skills.

Benefits

  • Work for a top IT service provider recognized for sustainability.
  • Opportunity to work with next-generation technologies.
  • Be part of a large, global team.
  • Competitive salary and benefits.
  • Remote work flexibility.

Similar Jobs:

T.W

Data Platform Engineer - Remote

Trust Wallet

7 days ago

Join Trust Wallet as a Data Platform Engineer to design and maintain robust data infrastructure for seamless data processing and analytics.

Data Lakes
Data Pipelines
Infrastructure AS Code
Terraform
Worldwide
Full-time
Software Development

Riskified

Data Platform Engineer - Remote

Riskified

1 week ago

Riskified is seeking a Data Platform Engineer to enhance their core streaming platform.

Data Engineering
BIG Data
Data Streaming
Apache Kafka
Portugal
Full-time
Software Development
ClassDojo logo

Data Platform Engineer - Remote

ClassDojo

2 weeks ago

ClassDojo is looking for a Data Platform Engineer to enhance data accessibility and support decision-making in education.

Data Engineering
Analytics Engineering
SQL
Python
Worldwide
Full-time
Software Development
$146,000 - $244,000 (USD)/year
Formula.Monks logo

Data Platform Engineer - Remote

Formula.Monks

2 weeks ago

Join Formula.Monks as a Data Platform Engineer to build innovative data platforms for Fortune 100 clients.

Data Platforms
Machine Learning
APIs
Data Management
CO, USA
Full-time
Data Analysis
BigID logo

Data Platform Engineer - Remote

BigID

3 weeks ago

Join BigID as a Data Platform Engineer to design and maintain ETL pipelines in a collaborative environment.

Data Engineering
ETL
NodeJS
Snowflake
India
Full-time
Software Development