Data Engineer - Remote

Overview

We are seeking an experienced Data Engineer with deep expertise in data transformation at scale, particularly in integrating and processing data from third-party public APIs. This role is critical to enhancing and maintaining data pipelines that feed into Natural Language Processing (NLP) models.

In Short

• Design, build, and optimize scalable ETL/ELT data pipelines using Apache Spark, Apache Kafka, and orchestration tools such as Prefect or Airflow.
• Integrate external data sources and public APIs with internal data systems.
• Work with large-scale datasets to support NLP model training and inference.
• Analyze existing pipelines and recommend enhancements for performance, reliability, and scalability.
• Collaborate with cross-functional teams, including data scientists and ML engineers.
• Own the end-to-end engineering process—from planning and technical design to implementation.
• Regularly report progress and outcomes to client stakeholders.

Requirements

• Proficiency in Python and experience with data transformation and data engineering best practices.
• Strong experience with Apache Spark, Apache Kafka, and Google Cloud Platform (GCP).
• Hands-on experience with workflow orchestration tools (e.g., Prefect, Airflow).
• Demonstrated experience working with large datasets and real-time data processing.
• Experience building and maintaining ETL/ELT pipelines for analytical or machine learning use cases.
• Self-motivated, with excellent communication and project ownership skills.

Benefits

• Work on cutting-edge projects with top IT engineers.
• Control which projects to work on and choose your own pay rate.
• Opportunity to work with entrepreneurs and Fortune 500 companies.
• Flexible working hours.
• Gain experience in cloud services and modern data engineering practices.