Senior AI Data Engineer - Remote

Overview

As a Senior AI Data Engineer, you will play a vital role in building and managing data ingestion pipelines and Gen AI infrastructure that powers innovative, intelligent systems.

In Short

• Own and maintain ingestion pipelines from various document formats including PDF, PPTX, DOCX, XLSX, TXT, and MD.
• Leverage PySpark to efficiently process large-scale datasets in distributed environments.
• Utilize Gen AI tools such as LangChain, LlamaParser, docling, OpenAI, and Hugging Face.
• Work with vector databases like Databricks Vector Search, Pinecone, and Azure AI Search.
• Design and manage Databricks workflows and scheduled jobs.
• Serve as a client-facing engineer to ensure ingestion pipelines meet business needs.

Requirements

• Proven experience in processing unstructured data from various formats.
• Strong proficiency in PySpark for distributed data processing.
• Experience with Gen AI/LLM tools including LangChain, LlamaParser, OpenAI, and Hugging Face.
• Solid understanding of chunking, embedding, and indexing techniques.
• Experience with vector databases such as Databricks Vector Search and Pinecone.
• Hands-on experience managing Databricks workflows.
• Ability to interface with clients and understand requirements.

Benefits

• Join a powerful tech workforce and help change the world through technology.
• Professional development opportunities with international customers.
• Collaborative work environment.
• Career path and mentorship programs.