Member of Technical Staff - LLM Inference - Remote

Overview

At .txt, our mission is to make AI reliable. We are the authors of outlines and outlines-core, both leading open source libraries (+10k ⭐️) for structured generation. We raised $11.9 million, which is fueling the efforts of our global, fully remote team to create software that goes beyond simple conversation.

In Short

• Looking for strong engineers to drive breakthroughs in LLM inference optimization.
• Focus on backend engineering to cut latency, boost throughput, and reduce resource costs.
• Experience deploying inference engines like vLLM, SGLang, or TensorRT is required.
• Experience with distributed inference and low-latency communication (NCCL).
• Hands-on knowledge of NVIDIA GPU architecture (CUDA, CUDA cores, memory hierarchy).
• Track record of improving inference performance.
• Background in LLM MLOps is beneficial.
• Proficiency in Python and willingness to learn Rust.
• Understanding of containerization (Docker, Kubernetes) and Linux systems.
• Remote first work culture.

Requirements

• Proven experience deploying inference engines like vLLM, SGLang, or TensorRT.
• Experience with distributed inference (multi-GPU, single node) and low-latency communication (NCCL).
• Hands-on knowledge of NVIDIA GPU architecture (CUDA, CUDA cores, memory hierarchy).
• Track record of improving inference performance (e.g., improving throughput by 20% through kernel optimizations).
• Background in LLM MLOps (monitoring, scaling, fault tolerance for inference services).
• Proficiency in Python and familiarity with (or willingness to learn) Rust.
• Understanding of containerization (Docker, Kubernetes) and Linux systems.

Benefits

• Cutting-edge technology in structured generation.
• Remote first work culture.
• Competitive compensation and benefits.
• Health and dental insurance offered.
• 401k available for US employees.
• Provision of GPU if not already owned.