Talend to Databricks Lead - Remote

Overview

This role involves leading the manual migration of 45 Talend ETL jobs to Databricks PySpark. The lead will analyze existing Talend jobs, map them to PySpark equivalents, oversee development, and ensure the migration maintains data accuracy and performance. This is a hands-on technical leadership position working with offshore developers. Preliminary job inventory has been conducted with BladeBridge Analyzer.

In Short

• Analyze and map 45 Talend ETL jobs to Databricks PySpark, including transformations, error handling, and integrations.
• Develop migration strategies, code PySpark implementations, and optimize for Databricks features like Delta Lake and workflows.
• Lead a team of offshore developers, assign tasks, review code, and provide guidance on Talend and PySpark.
• Conduct testing (unit, integration, end-to-end) to validate migrated jobs against original outputs.
• Collaborate with stakeholders to identify risks, resolve issues, and document processes.
• Provide progress updates and ensure timelines are met.

Requirements

• 8-12 years of experience in ETL development.
• Deep expertise in Talend (e.g., job design, components, repositories).
• Strong experience with Databricks PySpark (e.g., DataFrames, SQL, UDFs, performance tuning).
• Proficiency in SQL, Python, and Java.
• Experience leading teams on data migration projects.
• Familiarity with version control (e.g., Git) and CI/CD.

Benefits

• Prior Talend to PySpark migration experience.
• Knowledge of big data tools (e.g., Hadoop, Kafka).
• Certifications in Talend or Databricks.