The Big Data Developer plays a key role in the modernization of data ecosystems, supporting the migration of legacy MAPR/Cloudera/Hortonworks applications to open-source frameworks compatible with NexusOne. This individual will focus on refactoring, optimizing, and validating data processing pipelines, ensuring performance, scalability, and alignment with enterprise data standards.

In Short

Analyze, refactor, and modernize Spark/MapReduce/Hive/Tez jobs for execution within NexusOne’s managed Spark and Trino environments.
Design, build, and optimize batch and streaming pipelines using Spark, NiFi, and Kafka.
Convert existing ETL jobs and DAGs from Cloudera/MAPR ecosystems to open-source equivalents.
Collaborate with Data Engineers and Architects to define new data ingestion and transformation patterns.
Tune performance across large-scale data processing workloads (partitioning, caching, resource allocation).
Implement data quality and validation frameworks to ensure consistency during migration.
Support code reviews, performance tests, and production readiness validation for migrated workloads.
Document conversion approaches, dependencies, and operational runbooks.
Partner with Wells Fargo application SMEs to ensure domain alignment and business continuity.

Requirements

Core Frameworks: Apache Spark, PySpark, Airflow, NiFi, Kafka, Hive, Iceberg, Oozie
Programming Languages: Python, Scala, Java
Data Formats & Storage: Parquet, ORC, Avro, S3, HDFS
Orchestration & Workflow: Airflow, DBT
Performance Optimization: Spark tuning, partitioning strategies, caching, YARN/K8s resource tuning
Testing & Validation: Great Expectations, Deequ, SQL-based QA frameworks
Observability & Monitoring: Datadog, Grafana, Prometheus

Benefits

4–8 years of experience in big data engineering or application modernization in enterprise settings.
Prior experience with Cloudera, MAPR, or Hadoop ecosystems, transitioning to open-source frameworks.
Strong understanding of distributed data architectures and data lake design principles.
Exposure to hybrid or cloud-native environments (AWS, GCP, or Azure).
Familiarity with regulated environments (financial services, telecom, healthcare) is a plus.

Nexus Cognitive Technologies

Nexus Cognitive Technologies is a forward-thinking company specializing in the modernization of data ecosystems. They focus on migrating legacy applications to open-source frameworks, particularly through their NexusOne platform. With a strong emphasis on performance, scalability, and adherence to enterprise data standards, Nexus Cognitive Technologies leverages advanced technologies such as Apache Spark, Kafka, and NiFi to optimize data processing pipelines. The company values collaboration and innovation, working closely with clients in various sectors, including financial services, to ensure seamless transitions and high-quality data management.

Share This Job!

Save This Job!