Senior Data Engineer (GCP/Databricks) - Remote

Overview

We are looking for a Senior Data Engineer to design, develop, and optimize our data infrastructure on Google Cloud Platform (GCP). You will architect scalable pipelines using Databricks, BigQuery, Google Cloud Storage, Apache Airflow, dbt, Dataflow, and Pub/Sub, ensuring high availability and performance across our ETL/ELT processes. You will leverage great expectations to enforce data quality standards. The role also involves building our Data Mart (Data Mach) environment, containerizing services with Docker and Kubernetes (K8s), and implementing CI/CD best practices.

In Short

• Define and implement the overall data architecture on GCP.
• Integrate Terraform for Infrastructure as Code.
• Establish batch and real-time data processing frameworks.
• Design, build, and optimize ETL/ELT pipelines using Apache Airflow.
• Implement dbt transformations for data models in BigQuery.
• Use Google Dataflow and Pub/Sub for large-scale data processing.
• Implement event-driven data workflows between microservices.
• Employ Docker and Kubernetes for containerization.
• Enforce data quality standards using Great Expectations.
• Integrate with Looker for data visualization.

Requirements

• Extensive knowledge of cloud-native data solutions.
• Strong proficiency with ETL/ELT frameworks.
• Experience with Docker and Kubernetes.
• Familiarity with CI/CD best practices.
• Understanding of data quality and governance.
• Ability to collaborate with cross-functional teams.

Benefits

• Opportunity to work with cutting-edge technologies.
• Flexible working environment.
• Professional growth and development.
• Competitive salary and benefits.