Azure Databricks Engineer - Remote

Overview

As an Azure Databricks Engineer, you will play a pivotal role in our technology solutions team, focusing on the deployment and management of Azure Databricks environments. This position requires a deep understanding of cloud-based data engineering and analytics, particularly within the Azure ecosystem. You will be responsible for implementing best practices in CI/CD deployment using Databricks Asset Bundles (DAB) and Azure DevOps, ensuring that our data solutions are robust, scalable, and efficient. Your expertise will help drive our clients' digital transformation initiatives by leveraging the power of data analytics and machine learning.

In Short

• Collaborate with cross-functional teams to design, implement, and manage Azure Databricks environments.
• Deploy Azure Databricks and configure workspaces to meet project requirements.
• Create and manage Databricks Asset Bundles (DAB) to facilitate CI/CD deployment processes.
• Implement CI/CD pipelines using Azure DevOps, ensuring seamless integration and delivery of data solutions.
• Manage Unity Catalog changes and workspace configurations to enhance data governance and accessibility.
• Utilize Terraform for infrastructure as code (IaC) to automate the provisioning of Databricks resources.
• Monitor and optimize Databricks performance, ensuring efficient resource utilization and cost management.
• Provide technical guidance and support to team members and stakeholders regarding best practices in data engineering and analytics.
• Stay updated with the latest trends and advancements in Azure Databricks and related technologies to continuously improve our solutions.

Requirements

• Deep understanding of Azure Databricks and cloud-based data engineering.
• Experience with CI/CD processes and Azure DevOps.
• Strong knowledge of data governance and management.
• Proficient in Terraform and infrastructure as code principles.
• Ability to monitor and optimize performance in Databricks.
• Strong collaboration and communication skills.
• Experience in machine learning and data analytics is a plus.
• Willingness to stay updated with emerging technologies.

Benefits

• Opportunity to work on cutting-edge technologies.
• Collaborative and innovative work environment.
• Support for continuous learning and professional development.
• Flexible working arrangements.