Machine Learning Infrastructure Engineer - Remote

Overview

Black Forest Labs is a cutting-edge startup pioneering generative image and video models. Our team, which invented Stable Diffusion, Stable Video Diffusion, and FLUX.1, is currently looking for a strong candidate to join us in developing and maintaining our ML infra including large GPU training and inference clusters.

In Short

• Design, deploy, and maintain cloud-based ML training (Slurm) and inference (Kubernetes) clusters
• Implement and manage network-based cloud file systems and blob/S3 storage solutions
• Develop and maintain Infrastructure as Code (IaC) for resource provisioning
• Implement and optimize CI/CD pipelines for ML workflows
• Design and implement custom autoscaling solutions for ML workloads
• Ensure security best practices across the ML infrastructure
• Provide developer-friendly tools and practices for efficient ML operations

Requirements

• Strong proficiency in cloud platforms (AWS, Azure, or GCP) with focus on ML/AI services
• Extensive experience with Kubernetes and Slurm cluster management
• Expertise in Infrastructure as Code tools (e.g., Terraform, Ansible)
• Proven track record in managing and optimizing network-based cloud file systems and object storage
• Experience with CI/CD tools and practices (e.g., CircleCI, GitHub Actions, ArgoCD)
• Strong understanding of security principles and best practices in cloud environments
• Experience with monitoring and observability tools (e.g., Prometheus, Grafana, Loki)
• Familiarity with ML workflows and GPU infrastructure management
• Demonstrated ability to handle complex migrations and breaking changes in production environments

Benefits

• Experience with custom autoscaling solutions for ML workloads
• Knowledge of cost optimization strategies for cloud-based ML infrastructure
• Familiarity with MLOps practices and tools
• Experience with high-performance computing (HPC) environments
• Understanding of data versioning and experiment tracking for ML
• Knowledge of network optimization for distributed ML training
• Experience with multi-cloud or hybrid cloud architectures
• Familiarity with container security and vulnerability scanning tools