Eight Eleven Group
Senior Spark Architect/Engineer
Remote
6-month Contract
Our client is seeking a highly skilled Senior Spark Architect/Engineer for a remote, 6-month contract to lead the design, development, and deployment of scalable data solutions within an AWS cloud environment. This hands-on role blends architecture, engineering, and leadership to drive innovation in big data processing. The successful candidate will architect and implement robust Spark-based data pipelines, automate ETL processes, and drive data migration and transformation initiatives with precision and efficiency. You will also mentor and train internal team memberswho are new to Sparkto ensure they can confidently support and maintain Spark-based systems after the engagement concludes. If you are passionate about building high-performance data systems, enjoy coaching others, and thrive in a fast-paced environment, wed love to hear from you!
MINIMUM QUALIFICATIONS
7+ years of experience in data engineering or architecture roles Deep expertise in Apache Spark (Spark SQL, PySpark, Structured Streaming) and Kafka Proficiency in SQL, Python, Java, REST APIs, and Linux/RHEL 8 Hands-on experience with AWS EMR, Delta Tables, and hybrid cloud architectures Experience with Kubernetes or similar frameworks for managing Spark clusters Strong interpersonal skills with the ability to communicate complex technical concepts to both technical and non-technical stakeholders Proven success in mentoring and coaching technical teams PREFERRED QUALIFICATIONS
Familiarity with HL7 FHIR specifications Experience with serverless architectures Exposure to machine learning APIs and performance tuning of Spark jobs RESPONSIBILITIES
Architect, develop, test, and deploy scalable data lake/warehouse solutions using modern development tools in AWS cloud infrastructure Design and implement robust data ingestion pipelines for high-volume datasets Lead data migration and transformation initiatives with precision and efficiency Automate end-to-end ETL processes for diverse datasets within the big data ecosystem Collaborate with Business Analysts, Architects, and Senior Developers to define and establish the AWS application framework Mentor and train internal team members to ensure long-term sustainability of the Spark-based solution
#J-18808-Ljbffr
7+ years of experience in data engineering or architecture roles Deep expertise in Apache Spark (Spark SQL, PySpark, Structured Streaming) and Kafka Proficiency in SQL, Python, Java, REST APIs, and Linux/RHEL 8 Hands-on experience with AWS EMR, Delta Tables, and hybrid cloud architectures Experience with Kubernetes or similar frameworks for managing Spark clusters Strong interpersonal skills with the ability to communicate complex technical concepts to both technical and non-technical stakeholders Proven success in mentoring and coaching technical teams PREFERRED QUALIFICATIONS
Familiarity with HL7 FHIR specifications Experience with serverless architectures Exposure to machine learning APIs and performance tuning of Spark jobs RESPONSIBILITIES
Architect, develop, test, and deploy scalable data lake/warehouse solutions using modern development tools in AWS cloud infrastructure Design and implement robust data ingestion pipelines for high-volume datasets Lead data migration and transformation initiatives with precision and efficiency Automate end-to-end ETL processes for diverse datasets within the big data ecosystem Collaborate with Business Analysts, Architects, and Senior Developers to define and establish the AWS application framework Mentor and train internal team members to ensure long-term sustainability of the Spark-based solution
#J-18808-Ljbffr