Member of Engineering - Pre-training and Inference Fault Tolerance - Remote

Overview

As a member of the engineering team at Poolside, you will be focused on building out distributed training and inference for Large Language Models (LLMs), ensuring software reliability and fault tolerance.

In Short

• Work in a remote-first team across Europe and North America.
• Focus on distributed training and inference of LLMs.
• Hands-on role with an emphasis on software reliability.
• Debugging Linux kernel modules is part of the job.
• Strong engineering skills and knowledge of Torch and NVIDIA GPU architecture required.
• Design and develop tools to enhance training recovery.
• Minimize GPU idle time during faults.
• Write high-quality code in Python, Cython, C/C++, and CUDA.
• Collaborate with a diverse team focused on quality systems.
• Access to thousands of GPUs for testing changes.

Requirements

• Understanding of Large Language Models (LLM) and Transformers.
• Strong engineering background.
• Programming experience in Linux API, Linux kernel.
• Familiarity with Python, PyTorch, C/C++, NCCL.
• Experience with distributed systems and reliability concepts.
• Strong algorithmic skills.
• Critical thinking and questioning of code quality policies.
• Ability to work well in a fast-paced environment.
• Prepared for a steep learning curve.
• Strong communication skills.

Benefits

• Fully remote work and flexible hours.
• 37 days/year of vacation and holidays.
• Health insurance allowance for you and dependents.
• Company-provided equipment.
• Wellbeing and home office allowances.
• Frequent team get-togethers.
• Diverse and inclusive culture.