Atec Spine
Overview
We are looking for an experienced
Staff Data Engineer
to lead the development of our
high-performance data pipelines
and
scalable APIs
using
Databricks and Azure . As a key technical leader, you will design, implement, and optimize data infrastructure, ensuring efficient data processing and real-time analytics capabilities. This role requires deep expertise in
big data frameworks, distributed systems, and backend engineering , as well as a strong ability to mentor engineers and drive architectural decisions. Responsibilities
Design and Develop Data Pipelines: Design, build, and maintain scalable, high-performance data pipelines and ETL processes on Databricks and Microsoft Azure to acquire, ingest, process, and transform data from various sources into structured, usable form Azure Data Platform Expertise: Leverage knowledge of Microsoft Azure services such as Azure Databricks, Azure Storage, and Azure SQL Database to architect and optimize data solutions Architect and Develop Pipelines: Architect and develop large-scale data pipelines in Databricks, leveraging Apache Spark (PySpark/Scala) for batch and streaming workloads APIs for Data Access: Design, implement, and optimize high-performance APIs to support data access, ingestion, and real-time processing Data Infrastructure Leadership: Lead the development of scalable data infrastructure, ensuring reliability, security, and cost efficiency Data Lakes and Databases: Integrate Databricks with cloud data lakes (Azure Data Lake, AWS S3) and relational/NoSQL databases such as MongoDB and PostgreSQL Data Governance and Security: Implement data governance best practices and ensure data security, compliance and privacy measures are in place to protect sensitive data Performance and Efficiency: Optimize Spark jobs for efficiency and cost-effectiveness, implementing best practices for distributed computing Governance and Compliance: Define and enforce data governance, quality, and security standards, including access control, encryption, and compliance Real-time Processing: Develop event-driven and real-time data processing pipelines using Kafka, Pulsar, or similar messaging systems Backend Services and Monitoring: Drive the adoption of best practices for backend services, ensuring API security, logging, and monitoring Team Leadership: Mentor and coach junior and mid-level engineers, providing guidance on system design, performance tuning, and scalable architecture Cross-Functional Collaboration: Collaborate with data scientists, software engineers, and DevOps to enable seamless data-driven decision-making Architecture and Standards: Lead and participate in technical discussions, driving architecture decisions and improving engineering standards Documentation and Training: Create technical documentation, best practices, and guidelines for data engineering processes, and provide training to team members Research and Innovation: Stay updated with trends and technologies in data engineering and identify opportunities for innovation Qualifications
Minimum 5-7 years of experience in data engineering or related fields with experience on Azure data services Strong expertise with Databricks and Apache Spark (PySpark/Scala) Proficiency in designing and implementing data solutions on the Microsoft Azure platform Experience with creating and deploying microservices (Docker, Kubernetes) Strong programming skills in Python, SQL or Scala; Excellent verbal and written communication skills Preferred
Delta Lake, Apache Iceberg, or similar transactional data lakes Experience with GraphQL and gRPC for data APIs Machine learning data pipelines and feature stores Experience with medical image data and DICOM formats Professional Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert) Proven experience in leading and mentoring a team of data engineers Strong analytical skills Education
Bachelor's degree in computer science or a STEM field; Masters degree preferred ATEC is committed to providing equal employment opportunities to its employees and applicants without regard to race, color, religion, national origin, age, sex, sexual orientation, gender identity, gender expression, or any other protected status in accordance with applicable laws. Salary and Benefits
Salary range: $140,000 to $160,000. Full-Time, Mid-Senior level, Information Technology, Technology industries. Note: For roles based in the United States that require access to hospital facilities, candidates must be eligible for and maintain credentials at required hospitals, including any physical or vaccination requirements as applicable. Referrals increase your chances of interviewing at ATEC Spine. Get notified about new Data Engineer jobs in Carlsbad, CA. #J-18808-Ljbffr
We are looking for an experienced
Staff Data Engineer
to lead the development of our
high-performance data pipelines
and
scalable APIs
using
Databricks and Azure . As a key technical leader, you will design, implement, and optimize data infrastructure, ensuring efficient data processing and real-time analytics capabilities. This role requires deep expertise in
big data frameworks, distributed systems, and backend engineering , as well as a strong ability to mentor engineers and drive architectural decisions. Responsibilities
Design and Develop Data Pipelines: Design, build, and maintain scalable, high-performance data pipelines and ETL processes on Databricks and Microsoft Azure to acquire, ingest, process, and transform data from various sources into structured, usable form Azure Data Platform Expertise: Leverage knowledge of Microsoft Azure services such as Azure Databricks, Azure Storage, and Azure SQL Database to architect and optimize data solutions Architect and Develop Pipelines: Architect and develop large-scale data pipelines in Databricks, leveraging Apache Spark (PySpark/Scala) for batch and streaming workloads APIs for Data Access: Design, implement, and optimize high-performance APIs to support data access, ingestion, and real-time processing Data Infrastructure Leadership: Lead the development of scalable data infrastructure, ensuring reliability, security, and cost efficiency Data Lakes and Databases: Integrate Databricks with cloud data lakes (Azure Data Lake, AWS S3) and relational/NoSQL databases such as MongoDB and PostgreSQL Data Governance and Security: Implement data governance best practices and ensure data security, compliance and privacy measures are in place to protect sensitive data Performance and Efficiency: Optimize Spark jobs for efficiency and cost-effectiveness, implementing best practices for distributed computing Governance and Compliance: Define and enforce data governance, quality, and security standards, including access control, encryption, and compliance Real-time Processing: Develop event-driven and real-time data processing pipelines using Kafka, Pulsar, or similar messaging systems Backend Services and Monitoring: Drive the adoption of best practices for backend services, ensuring API security, logging, and monitoring Team Leadership: Mentor and coach junior and mid-level engineers, providing guidance on system design, performance tuning, and scalable architecture Cross-Functional Collaboration: Collaborate with data scientists, software engineers, and DevOps to enable seamless data-driven decision-making Architecture and Standards: Lead and participate in technical discussions, driving architecture decisions and improving engineering standards Documentation and Training: Create technical documentation, best practices, and guidelines for data engineering processes, and provide training to team members Research and Innovation: Stay updated with trends and technologies in data engineering and identify opportunities for innovation Qualifications
Minimum 5-7 years of experience in data engineering or related fields with experience on Azure data services Strong expertise with Databricks and Apache Spark (PySpark/Scala) Proficiency in designing and implementing data solutions on the Microsoft Azure platform Experience with creating and deploying microservices (Docker, Kubernetes) Strong programming skills in Python, SQL or Scala; Excellent verbal and written communication skills Preferred
Delta Lake, Apache Iceberg, or similar transactional data lakes Experience with GraphQL and gRPC for data APIs Machine learning data pipelines and feature stores Experience with medical image data and DICOM formats Professional Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect Expert) Proven experience in leading and mentoring a team of data engineers Strong analytical skills Education
Bachelor's degree in computer science or a STEM field; Masters degree preferred ATEC is committed to providing equal employment opportunities to its employees and applicants without regard to race, color, religion, national origin, age, sex, sexual orientation, gender identity, gender expression, or any other protected status in accordance with applicable laws. Salary and Benefits
Salary range: $140,000 to $160,000. Full-Time, Mid-Senior level, Information Technology, Technology industries. Note: For roles based in the United States that require access to hospital facilities, candidates must be eligible for and maintain credentials at required hospitals, including any physical or vaccination requirements as applicable. Referrals increase your chances of interviewing at ATEC Spine. Get notified about new Data Engineer jobs in Carlsbad, CA. #J-18808-Ljbffr