zAnswer LLC
ETL Developer - No C2C - Onsite Role - No Relo
zAnswer LLC, Jersey City, New Jersey, United States, 07390
We are seeking a highly skilled
Senior Data Engineer
to design, develop, and manage robust ETL and ELT pipelines entirely within the AWS ecosystem. This role is crucial for integrating diverse financial data sources into performant data warehouses and data lakes, ensuring scalability, security, and data quality.
Deep domain experience within the Banking/Financial Services industry is mandatory.
Core Responsibilities
Pipeline Architecture: Architect and implement robust, scalable ETL/ELT pipelines leveraging native AWS services for optimal data ingestion and processing.
Data Integration: Integrate data from a variety of sources, including external APIs, transactional databases, and flat files, into centralized AWS-based data platforms.
Transformation Development: Develop complex data transformation logic utilizing PySpark, Python, and SQL, primarily executed within AWS Glue and AWS Lambda environments.
Orchestration & Monitoring: Establish, monitor, and maintain workflow orchestration using tools such as AWS Step Functions, Glue Workflows, or Apache Airflow on Amazon MWAA.
Data Governance & Quality: Ensure stringent data quality, consistency, and lineage tracking, utilizing services like the AWS Glue Data Catalog and AWS Lake Formation.
Performance Optimization: Proactively identify and execute optimizations for ETL performance and cost‑efficiency through techniques such as partitioning, parallelism, and resource tuning.
Security & Compliance: Implement and enforce security best practices, including data encryption, management of IAM roles, and precise VPC configurations.
Collaboration & Documentation: Partner closely with data engineers, analysts, and DevOps teams to support critical analytics and reporting needs. Maintain comprehensive documentation of ETL processes, data flows, and architectural diagrams.
Required Technical Stack
Serverless ETL: AWS Glue for serverless data integration and AWS Lambda for lightweight, real‑time transformations.
Data Storage: Amazon S3 for building and managing the core Data Lake.
Data Warehousing: Experience with Amazon Redshift or Amazon RDS for relational data warehousing needs.
Orchestration: Experience with AWS Step Functions, AWS Glue Workflows, or Apache Airflow on Amazon MWAA.
Transformation Languages: Expert level skills in PySpark, Python, and SQL.
Seniority level Not Applicable
Employment type Contract
Job function Business Development and Sales
Industries Software Development
#J-18808-Ljbffr
Senior Data Engineer
to design, develop, and manage robust ETL and ELT pipelines entirely within the AWS ecosystem. This role is crucial for integrating diverse financial data sources into performant data warehouses and data lakes, ensuring scalability, security, and data quality.
Deep domain experience within the Banking/Financial Services industry is mandatory.
Core Responsibilities
Pipeline Architecture: Architect and implement robust, scalable ETL/ELT pipelines leveraging native AWS services for optimal data ingestion and processing.
Data Integration: Integrate data from a variety of sources, including external APIs, transactional databases, and flat files, into centralized AWS-based data platforms.
Transformation Development: Develop complex data transformation logic utilizing PySpark, Python, and SQL, primarily executed within AWS Glue and AWS Lambda environments.
Orchestration & Monitoring: Establish, monitor, and maintain workflow orchestration using tools such as AWS Step Functions, Glue Workflows, or Apache Airflow on Amazon MWAA.
Data Governance & Quality: Ensure stringent data quality, consistency, and lineage tracking, utilizing services like the AWS Glue Data Catalog and AWS Lake Formation.
Performance Optimization: Proactively identify and execute optimizations for ETL performance and cost‑efficiency through techniques such as partitioning, parallelism, and resource tuning.
Security & Compliance: Implement and enforce security best practices, including data encryption, management of IAM roles, and precise VPC configurations.
Collaboration & Documentation: Partner closely with data engineers, analysts, and DevOps teams to support critical analytics and reporting needs. Maintain comprehensive documentation of ETL processes, data flows, and architectural diagrams.
Required Technical Stack
Serverless ETL: AWS Glue for serverless data integration and AWS Lambda for lightweight, real‑time transformations.
Data Storage: Amazon S3 for building and managing the core Data Lake.
Data Warehousing: Experience with Amazon Redshift or Amazon RDS for relational data warehousing needs.
Orchestration: Experience with AWS Step Functions, AWS Glue Workflows, or Apache Airflow on Amazon MWAA.
Transformation Languages: Expert level skills in PySpark, Python, and SQL.
Seniority level Not Applicable
Employment type Contract
Job function Business Development and Sales
Industries Software Development
#J-18808-Ljbffr