Overview
Sr. Data Pipeline Engineer (AWS/Databricks)
Location: San Jose, CA (onsite 2X a week). Local candidates only. Face to Face Interview required.
Description
We're looking for a Data Pipeline Engineer with deep experience building and orchestrating large-scale ingestion pipelines. This role is ideal for someone who enjoys working across high-volume telemetry sources, optimizing data workflows, and solving schema drift challenges in real-world distributed environments.
You'll be part of the Security Data Platform and ML Engineering team, helping to onboard and normalize security data that powers analytics, detection, and ML workflows across the BU.
Key Responsibilities
- Design and build scalable batch and streaming data pipelines for ingesting telemetry, log, and event data
- Develop and maintain orchestration workflows using tools like Apache Airflow or similar schedulers
- Onboard new data sources, build connectors (API/Kafka/file-based), and normalize security-related datasets
- Monitor and manage schema drift across changing source systems and formats
- Implement observability into pipelines - logging, metrics, and alerts for health and performance
- Optimize ingestion for performance, resilience, and cost-efficiency
- Collaborate across detection, threat intel, and platform teams to align ingestion with security use cases
Required Qualifications
- 5+ years of experience in data engineering or infrastructure roles focused on pipeline development
- Strong experience with Python and distributed data processing tools like Apache Spark or PySpark
- Hands-on experience with orchestration frameworks like Apache Airflow, Dagster, or similar
- Deep understanding of ingestion best practices, schema evolution, and drift handling
- Experience working with Kafka, S3, or cloud-native storage and messaging systems
- Experience in cloud environments (AWS, Azure, or GCP)
- Bonus: Familiarity with security tools (e.g., Crowdstrike, Wiz), OCSF, or compliance-related data
Compensation & Benefits
The base pay range above represents the low and high end of the base compensation range we reasonably expect to pay for this position. Actual base compensation will vary and may be above or below the range based on various factors including geographic location, actual experience, and job performance. This job posting is not a promise of any specific pay for any specific employee.
The range listed is just one component of the total compensation package. Benefits include medical, dental, and vision plans, pre-tax saves, parking and commuter plans, supplemental health and welfare plans, a retirement savings plan, an employee assistance program, pet insurance, paid holidays, and other rewards may include short-term incentives and paid time off.
About Advantage Technical
Advantage Technical is an engineering and information technology services company with over 30 years of experience. Services include Staff Augmentation, Direct Placement, Project Resourcing and Outsourcing across North America. This employer is an equal opportunity employer and an E-Verify participant. Travel, background checks, and other employment prerequisites may apply.
#J-18808-Ljbffr