Lead Data Engineer - Remote

Overview

At Lunavi, Lead Data Engineers collaborate to create modern data estates and robust business intelligence reporting dashboards. We are seeking an individual with excellent soft skills who can lead a team of developers as a tech lead. This individual must be versed in Agile methodologies and have experience with development in Azure.

In Short

• Lead data engagements including defining architecture, creating databases structures and diagrams, optimizing speed and resiliency, and intelligent data strategies
• Implement and manage data solutions using Microsoft Fabric, Databricks and other data platforms, ensuring optimal performance and scalability
• Implement machine learning models for automated data classification and tagging, automated data quality validation, predictive insights and intelligent anomaly detection in data streams
• Create and deploy resources using Infrastructure as Code and DevOps pipelines
• Design, develop, and maintain ETL pipelines and notebooks to ensure efficient data flow and integration across various systems and incorporate predictive analytics and automated insights
• Apply architecture frameworks and methodologies (Medallion, Lambda, data mesh, etc.) to organize data into layers for efficient processing and analysis
• Design and implement resilient disaster recovery and high availability strategies that optimize system performance, minimize downtime, and balance operational efficiency with cost-effectiveness
• Develop and maintain reports and dashboards and reports to provide actionable insights and key performance indicators to stakeholders using PowerBI, PowerApps and other data visualization platforms
• Communicate effectively and translate complex technical concepts to technical and non-technical stakeholders and within cross-functional internal and client teams
• Provide leadership and mentoring for developers, data engineers and data analysts on your project teams

Requirements

• 5+ years of experience working with enterprise data warehouses/lakehouses with a recent focus on modern cloud native data architecture and development
• 4+ years of experience working with a diverse range of data platforms including Fabric, Databricks, Microsoft SQL, Oracle, Postgres, or other data platforms
• 4+ years of experience with PowerBI Pro or Premium datasets and developing strong visualizations
• Provide leadership on implementation of engineering practices such as automated CI/CD processes, ETL pipelines, notebooks and other modern data transformation processes and modern data architecture patterns
• Proficiency in programming languages commonly used in data analytics and AI/ML projects (e.g., Python, R, SQL)
• Experience with AI and ML frameworks, tools, and libraries such as TensorFlow, PyTorch, and scikit-learn
• Proficiency in data governance and security strategies including row level security, access controls using Purview, Unity Catalog, Active Directory, etc
• 4+ years of experience in customer-facing leadership roles
• 3+ years experience providing technical leadership on Agile data projects
• Desire to learn, investigate, and provide guidance on new technology in a rapidly changing environment
• The desire to bring out the best of the people working on the project to make a positive, shared impact

Benefits

• Certifications in Data Engineering, Data Governance, and AI/ML technologies (Azure AI, Google Cloud AI, AWS Machine Learning)
• Experience with advanced ML ops and MLflow implementations
• Background in developing and tuning custom machine learning models for data engineering optimization