Big Data Engineer - Professional Services - Remote

Overview

As a Big Data Engineer at BrightEdge, you will design, build, and maintain high-performance data pipelines that process terabytes of data, ensuring the data infrastructure can handle increasing volumes while maintaining exceptional performance standards.

In Short

• Design and implement scalable batch processing systems using Python and big data technologies
• Optimize database performance, focusing on slow-running queries and latency improvements
• Use Python profilers and performance monitoring tools to identify bottlenecks
• Reduce P95 and P99 latency metrics across our data platform
• Build efficient ETL pipelines that can handle large-scale data processing
• Collaborate with data scientists and product teams to understand data requirements
• Monitor and troubleshoot data pipeline issues in production
• Implement data quality checks and validation mechanisms
• Document data architecture and engineering processes
• Stay current with emerging big data technologies and best practices

Requirements

• Bachelor's degree in Computer Science, Engineering, or related technical field
• 4+ years of experience in data engineering roles
• Strong Python programming skills with focus on data processing libraries
• Experience with big data technologies (Spark, Hadoop, etc.)
• Proven experience optimizing database performance (SQL or NoSQL)
• Knowledge of data pipeline orchestration tools (Airflow, Luigi, etc.)
• Understanding of performance optimization techniques and profiling tools

Benefits

• Opportunity to work with cutting-edge technologies
• Collaborative and innovative work environment
• Professional development and growth opportunities
• Competitive salary and benefits package