Lead Cluster Operations Support Engineer - Remote

Overview

We are seeking a highly skilled Lead Cluster Operations Support Engineer with extensive experience in cloud infrastructure, Kubernetes, and GPU clusters. The ideal candidate will possess a strong background in operations, cloud architecture, and managing large-scale environments, particularly in the context of machine learning model training and high-performance computing.

In Short

• You will help shape and iterate this new white glove model training support service on large GPU clusters.
• You will work in a collaborative team with Machine Learning Engineers and Infrastructure Engineers.
• You will contribute to accelerator development: find gaps in the tooling, or needed automation, or patterns we would develop accelerators to make the next round of this more efficient and faster.
• You will help assess model training readiness and data preparation.
• You will provide model training support during rotating daytime weekend shifts.
• You will facilitate collaborative problem-solving within the team.
• You will proactively identify and address challenges related to the white glove service for continued pre-training.

Requirements

• Deep expertise in Kubernetes administration and debugging at scale.
• Extensive experience managing large clusters with thousands of nodes using Kubernetes.
• Knowledge of running training workloads on thousands of GPUs.
• Familiarity with the Lustre filesystem is a plus.
• Experience working with the NVIDIA NeMo Framework.
• Proficiency with cloud platforms such as GCP, AWS, and Azure.
• Experience with Terraform/Pulumi, Helm Charts, and Infrastructure-as-Code tools.

Benefits

• Support for career development and learning opportunities.
• Hybrid working model with remote work options.
• Equal opportunity employer policies.