ZipRecruiter
Job DescriptionJob DescriptionPrimary Job Title: Lead AWS Cloud Apache Spark Architect
Industry Sector:
Enterprise Cloud Data Engineering and Big Data Analytics. We design, deploy and operate high-scale AWS- data platforms and analytics pipelines for enterprise customers—supporting batch and real-time ML/BI workloads across finance, healthcare, and adtech. This is an onsite U.S. role focused on architecting secure, cost-efficient Spark-based processing at scale.
Role Responsibilities
Architect and deliver AWS- big data platforms and data lake solutions using S3, EMR, Glue, Redshift and EKS—designing for performance, scale and resiliency.
Lead migration efforts from on-prem Hadoop/Cloudera ecosystems to AWS (EMR/EKS/Glue), defining cutover strategies, data validation, and rollback plans.
Optimize Apache Spark (PySpark/Scala) jobs and clusters for throughput, latency and cost—tuning shuffle, partitioning, memory/executor settings and job scheduling.
Implement IaC and production-grade CI/CD for data pipelines using Terraform/CloudFormation and pipelines (Jenkins, GitLab CI), including automated testing and deployment safeguards.
Define and enforce security, governance and networking best practices (IAM, VPC design, encryption, data lineage, access controls) for enterprise workloads.
Mentor engineering teams, run architecture reviews, set operational runbooks, and drive capacity planning and observability standards.
Skills Qualifications
Must-Have : 7+ years hands-on AWS experience (EMR, S3, Glue, Redshift, EC2) and deep Apache Spark expertise (PySpark and/or Scala) including production performance tuning and debugging.
Must-Have : Proven track record migrating on-prem Hadoop or legacy ETL to AWS and operating Spark in EMR/EKS at enterprise scale.
Must-Have : Strong IaC CI/CD skills (Terraform/CloudFormation, Jenkins/GitLab/GitHub Actions), containerization (Docker) and Kubernetes/EKS experience.
: Experience with streaming (Kafka/Kinesis), Spark Structured Streaming, Delta Lake or Iceberg and event-driven architectures.
: Solid understanding of security compliance (IAM, encryption, SOC2/HIPAA awareness), VPC/networking and observability tooling (CloudWatch, Prometheus, Grafana).
: Bachelor’s/Master’s in CS or related field and prior leadership/architect role in enterprise data platform projects.
Benefits Culture Highlights
On-site U.S. role with ownership of high-impact modernization projects and visible cross-functional influence.
Engineering-first culture that values mentorship, technical excellence, and measurable business outcomes.
Learning development support—conferences, certifications, and hands-on opportunities to build large-scale, production data systems.
Location Work Type:
United States — Onsite (candidate must be based in or willing to relocate to the U.S. and work from the office).
Keywords: AWS, Apache Spark, EMR, PySpark, Scala, Terraform, EKS, Kafka, Glue, Redshift, S3, data-lake, streaming, performance tuning, migration, IaC, CI/CD, security, observability.
Industry Sector:
Enterprise Cloud Data Engineering and Big Data Analytics. We design, deploy and operate high-scale AWS- data platforms and analytics pipelines for enterprise customers—supporting batch and real-time ML/BI workloads across finance, healthcare, and adtech. This is an onsite U.S. role focused on architecting secure, cost-efficient Spark-based processing at scale.
Role Responsibilities
Architect and deliver AWS- big data platforms and data lake solutions using S3, EMR, Glue, Redshift and EKS—designing for performance, scale and resiliency.
Lead migration efforts from on-prem Hadoop/Cloudera ecosystems to AWS (EMR/EKS/Glue), defining cutover strategies, data validation, and rollback plans.
Optimize Apache Spark (PySpark/Scala) jobs and clusters for throughput, latency and cost—tuning shuffle, partitioning, memory/executor settings and job scheduling.
Implement IaC and production-grade CI/CD for data pipelines using Terraform/CloudFormation and pipelines (Jenkins, GitLab CI), including automated testing and deployment safeguards.
Define and enforce security, governance and networking best practices (IAM, VPC design, encryption, data lineage, access controls) for enterprise workloads.
Mentor engineering teams, run architecture reviews, set operational runbooks, and drive capacity planning and observability standards.
Skills Qualifications
Must-Have : 7+ years hands-on AWS experience (EMR, S3, Glue, Redshift, EC2) and deep Apache Spark expertise (PySpark and/or Scala) including production performance tuning and debugging.
Must-Have : Proven track record migrating on-prem Hadoop or legacy ETL to AWS and operating Spark in EMR/EKS at enterprise scale.
Must-Have : Strong IaC CI/CD skills (Terraform/CloudFormation, Jenkins/GitLab/GitHub Actions), containerization (Docker) and Kubernetes/EKS experience.
: Experience with streaming (Kafka/Kinesis), Spark Structured Streaming, Delta Lake or Iceberg and event-driven architectures.
: Solid understanding of security compliance (IAM, encryption, SOC2/HIPAA awareness), VPC/networking and observability tooling (CloudWatch, Prometheus, Grafana).
: Bachelor’s/Master’s in CS or related field and prior leadership/architect role in enterprise data platform projects.
Benefits Culture Highlights
On-site U.S. role with ownership of high-impact modernization projects and visible cross-functional influence.
Engineering-first culture that values mentorship, technical excellence, and measurable business outcomes.
Learning development support—conferences, certifications, and hands-on opportunities to build large-scale, production data systems.
Location Work Type:
United States — Onsite (candidate must be based in or willing to relocate to the U.S. and work from the office).
Keywords: AWS, Apache Spark, EMR, PySpark, Scala, Terraform, EKS, Kafka, Glue, Redshift, S3, data-lake, streaming, performance tuning, migration, IaC, CI/CD, security, observability.