Senior Data Engineer - Remote

Overview

We are looking for a highly skilled Data Engineer with strong experience in Databricks to play a key role in overseeing and optimizing the work of System Integrators (SIs). In this role, you will ensure best practices are followed, hold SIs accountable for their deliverables, and prevent costly inefficiencies caused by poor data management.

In Short

• Oversee the work of System Integrators (SIs), ensuring deliverables are clean, efficient, and adhere to best practices.
• Proactively hold SIs accountable for maintaining high standards, addressing issues, and ensuring data management is efficient and cost-effective.
• Provide expert guidance on cluster management and data pipeline best practices, ensuring performance and scalability.
• Identify and address inefficiencies in data storage to avoid excessive costs and long-term technical debt.
• Act as an influencer within the team, promoting a culture of accountability, excellence, and collaboration.
• Design, implement, and optimize data pipelines and workflows, leveraging Databricks and the Microsoft Azure stack.
• Ensure robust data governance, including cleanup processes to avoid build-up of redundant or costly data.
• Stay hands-on with data engineering to bring practical insights, while stepping into a management role to guide the team and SIs.
• Collaborate with data architects, analysts, and business stakeholders to align data engineering solutions with business needs.

Requirements

• Proven expertise in Databricks, including hands-on experience with best practices for cluster management, performance optimization, and scalable data solutions.
• Recent hands-on experience in data engineering, coupled with readiness to step into a management and oversight role.
• Strong knowledge of CI/CD pipelines and DataOps best practices.
• Solid understanding of Azure Data Factory, Synapse, Databricks, and PySpark.

Benefits

• Opportunity to work with a leading beverage company.
• Engage in innovative projects and solutions.
• Collaborate with cross-functional teams.
• Foster a culture of excellence and collaboration.