ZipRecruiter
We are looking for an experienced Staff Data Engineer to lead the development of our high-performance data pipelines and scalable APIs using Databricks and Rust. As a key technical leader, you will design, implement, and optimize data infrastructure, ensuring efficient data processing and real-time analytics capabilities. This role requires deep expertise in big data frameworks, distributed systems, and backend engineering, as well as a strong ability to mentor engineers and drive architectural decisions.
Essential Duties and Responsibilities
Design and develop scalable, high-performance data pipelines and ETL processes on
Databricks/Microsoft Azure
to acquire, ingest, process, and transform data from various sources into structured, usable formats. Leverage your knowledge of Microsoft Azure services such as Azure Databricks, Azure Storage, and Azure SQL Database to architect and optimize data solutions. Develop large-scale data pipelines in Databricks, utilizing Apache Spark (PySpark/Scala) for batch and streaming workloads. Implement and optimize high-performance Rust APIs for data access, ingestion, and real-time processing. Lead the development of scalable data infrastructure, ensuring reliability, security, and cost efficiency. Integrate Databricks with cloud data lakes (Azure Data Lake, AWS S3, GCP BigQuery) and databases such as MongoDB & Postgres. Implement data governance and security best practices to ensure data security, compliance, and privacy. Optimize Spark jobs for efficiency and cost-effectiveness, following distributed computing best practices. Establish and enforce data governance, quality, and security standards, including access control, encryption, and compliance. Create event-driven and real-time data pipelines using Kafka, Pulsar, or similar messaging systems. Promote the adoption of Rust-based backend services, ensuring robust API security, logging, and monitoring. Mentor and coach junior and mid-level engineers on system design, performance tuning, and scalable architecture. Collaborate with data scientists, software engineers, and DevOps teams to enable data-driven decision-making. Document technical processes, best practices, and guidelines, and provide training to team members. Stay updated with the latest trends, tools, and technologies in data engineering, and identify opportunities for innovation and improvement. Requirements 5-7 years of experience in data engineering or related fields, with expertise in Azure data services. Strong experience with Databricks and Apache Spark (PySpark or Scala). Proficiency in Rust for backend/API development. Experience designing and implementing data solutions on Microsoft Azure. Experience deploying microservices using Docker and Kubernetes. Strong programming skills in Python, SQL, Scala, and Rust. Excellent communication skills for technical and non-technical audiences. Experience with Delta Lake, Apache Iceberg, or similar data lakes. Knowledge of GraphQL and gRPC for data API development. Experience with machine learning data pipelines and feature stores. Ability to analyze complex data requirements, troubleshoot issues, and propose solutions. Experience with medical imaging data (X-ray, CT, MRI) and DICOM format is a plus. Certifications such as Azure Data Engineer Associate or Azure Solutions Architect Expert are highly desirable. Proven leadership and mentoring experience in data engineering teams. Strong analytical skills are a plus. Education and Experience Bachelor’s degree in computer science or a STEM field; Master’s degree is a plus. For U.S.-based roles requiring hospital access, candidates must maintain necessary credentials, including vaccination requirements. ATEC is committed to equal employment opportunities and provides reasonable accommodations as required by law. Salary Range Full-Time Annual Salary: $140,000 to $160,000, based on qualifications, experience, and internal equity.
#J-18808-Ljbffr
Databricks/Microsoft Azure
to acquire, ingest, process, and transform data from various sources into structured, usable formats. Leverage your knowledge of Microsoft Azure services such as Azure Databricks, Azure Storage, and Azure SQL Database to architect and optimize data solutions. Develop large-scale data pipelines in Databricks, utilizing Apache Spark (PySpark/Scala) for batch and streaming workloads. Implement and optimize high-performance Rust APIs for data access, ingestion, and real-time processing. Lead the development of scalable data infrastructure, ensuring reliability, security, and cost efficiency. Integrate Databricks with cloud data lakes (Azure Data Lake, AWS S3, GCP BigQuery) and databases such as MongoDB & Postgres. Implement data governance and security best practices to ensure data security, compliance, and privacy. Optimize Spark jobs for efficiency and cost-effectiveness, following distributed computing best practices. Establish and enforce data governance, quality, and security standards, including access control, encryption, and compliance. Create event-driven and real-time data pipelines using Kafka, Pulsar, or similar messaging systems. Promote the adoption of Rust-based backend services, ensuring robust API security, logging, and monitoring. Mentor and coach junior and mid-level engineers on system design, performance tuning, and scalable architecture. Collaborate with data scientists, software engineers, and DevOps teams to enable data-driven decision-making. Document technical processes, best practices, and guidelines, and provide training to team members. Stay updated with the latest trends, tools, and technologies in data engineering, and identify opportunities for innovation and improvement. Requirements 5-7 years of experience in data engineering or related fields, with expertise in Azure data services. Strong experience with Databricks and Apache Spark (PySpark or Scala). Proficiency in Rust for backend/API development. Experience designing and implementing data solutions on Microsoft Azure. Experience deploying microservices using Docker and Kubernetes. Strong programming skills in Python, SQL, Scala, and Rust. Excellent communication skills for technical and non-technical audiences. Experience with Delta Lake, Apache Iceberg, or similar data lakes. Knowledge of GraphQL and gRPC for data API development. Experience with machine learning data pipelines and feature stores. Ability to analyze complex data requirements, troubleshoot issues, and propose solutions. Experience with medical imaging data (X-ray, CT, MRI) and DICOM format is a plus. Certifications such as Azure Data Engineer Associate or Azure Solutions Architect Expert are highly desirable. Proven leadership and mentoring experience in data engineering teams. Strong analytical skills are a plus. Education and Experience Bachelor’s degree in computer science or a STEM field; Master’s degree is a plus. For U.S.-based roles requiring hospital access, candidates must maintain necessary credentials, including vaccination requirements. ATEC is committed to equal employment opportunities and provides reasonable accommodations as required by law. Salary Range Full-Time Annual Salary: $140,000 to $160,000, based on qualifications, experience, and internal equity.
#J-18808-Ljbffr