LLM Data Engineer | United States | Fully Remote

Overview

We are seeking an experienced AI/LLM Data Engineer to build and maintain the data pipeline for our Generative AI platform.

In Short

• Design, implement, and maintain an end-to-end multi-stage data pipeline for LLMs.
• Identify, evaluate, and integrate diverse data sources and domains.
• Develop and optimize data processing workflows for chunking, indexing, ingestion, and vectorization.
• Benchmark and implement various vector stores and embedding techniques.
• Create a flexible pipeline supporting multiple embedding algorithms and search types.
• Implement and maintain auto-tagging systems and data preparation processes.
• Develop tools for text and image data crawling, cleaning, and refinement.
• Collaborate with cross-functional teams to ensure data quality.
• Work with data lake house architectures to optimize data storage.
• Integrate and optimize workflows using Snowflake and vector store technologies.

Requirements

• Strong background in data engineering.
• Experience with Generative AI and LLM technologies.
• Proven track record of delivering innovative AI applications.
• Familiarity with Retrieval-Augmented Generation techniques.
• Experience in data processing and workflow optimization.
• Ability to collaborate with cross-functional teams.
• Knowledge of vector stores and embedding techniques.
• Experience with data lake house architectures.
• Strong problem-solving skills.
• Passion for AI and machine learning.

Benefits

• Opportunity to work on highly visible strategic projects.
• Collaborate with cross-functional teams.
• Be part of a passionate team focused on Generative AI.
• Work in a dynamic and innovative environment.
• Access to the latest technologies in AI.