Senior Big Data Engineer - Remote

Overview

At Definitive Healthcare, our passion is to transform data, analytics and expertise into healthcare commercial intelligence. We help clients uncover the right markets, opportunities and people, so they can shape tomorrow’s healthcare industry. Our SaaS platform creates new paths to commercial success in the healthcare market, so companies can identify where to go next.

In Short

• Design and develop scalable data pipelines using Python, Spark, and Databricks.
• Implement data workflows and ETL processes using Apache Airflow.
• Integrate data from various sources into a unified data warehouse.
• Ensure data quality and integrity through effective data cleansing practices.
• Manage and optimize data storage solutions for high availability.
• Implement and manage Unity Catalog for metadata management.
• Optimize Spark jobs for performance and efficiency.
• Work closely with data scientists and stakeholders to deliver solutions.
• Implement CI/CD pipelines for data infrastructure.
• Stay up to date with industry trends to improve data engineering practices.

Requirements

• Hands-on Python or Scala programming experience.
• Strong experience with Apache Spark and Databricks.
• Experience with Apache Airflow or similar tools.
• Knowledge of data governance principles and best practices.
• Experience with cloud platforms (AWS and GCP).
• Proficiency in CI/CD tools and practices.
• Experience with Medallion architecture for data maturity lifecycle.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• Detail-oriented with a focus on delivering high-quality work.

Benefits

• Industry leading products.
• Flexible and dynamic culture.
• Competitive benefits package including healthcare and 401(k) match.
• Opportunities for community service through DefinitiveCares.
• Recognition as a best place to work in Massachusetts.