Get new jobs for this search by email

Lead Pyspark / SQL Engineer - Threat Detection & DataBricks

Lead Pyspark / SQL Engineer - Threat Detection & DataBricks

Get AI-powered advice on this job and more exclusive features.

This range is provided by MDA Edge. Your actual pay will be based on your skills and experience talk with your recruiter to learn more.

Base pay range

$76.55/hr - $129.16/hr

Job Summary: We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for creating equivalent PySpark log pipelines, rule configuration files, unit tests, and data validation checks, and deploying these pipelines using Terraform. A strong background in data engineering, particularly with large-scale log analytics, is essential.
Requirements:

• 5-10+ years of experience as a PySpark/SQL Engineer , with a strong focus on data engineering and analytics.
• Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks.
• Hands-on experience with Terraform for deploying data infrastructure.
• Proficient in PySpark for large-scale data processing and transformation.
• Familiarity with cloud platforms such as AWS or Azure is preferred.
• Strong analytical skills and attention to detail when working with complex datasets.
• Proven ability to work effectively in collaborative, cross-functional teams.
• Excellent verbal and written communication skills in English.

• Design and build threat detection pipelines using PySpark, SQL, and Databricks.
• Support the migration of detection rules and content from a legacy platform to Databricks.
• Create and maintain PySpark log pipelines and associated rule configuration files .
• Write unit tests to ensure pipeline accuracy and stability.
• Perform data validation checks to ensure data integrity.
• Deploy pipelines and infrastructure using Terraform .
• Optimize existing data workflows and queries for performance and scalability.
• Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Other

• Industries

  IT Services and IT Consulting

Referrals increase your chances of interviewing at MDA Edge by 2x

Get notified about new Lead Software Engineer jobs in San Francisco, CA .

Founding Team Senior Software Engineer Manager for a Social Travel Platform targeting Digital Nomads

San Francisco, CA $150,000.00-$200,000.00 4 months ago

Software Engineering Leader - SLS, 3D Imaging, & Point Cloud

San Francisco, CA $150,000.00-$250,000.00 3 days ago

Senior Software Engineer, Backend - Fintech

San Francisco, CA $150,000.00-$207,000.00 5 months ago

San Francisco, CA $185,200.00-$274,400.00 4 days ago

San Francisco, CA $170,000.00-$205,000.00 2 months ago

Senior Software Engineer, Game Experience

United States $196,000.00-$220,500.00 1 week ago

San Francisco, CA $154,000.00-$210,000.00 4 weeks ago

Senior Software Engineer, Infrastructure

San Francisco, CA $130,000.00-$200,000.00 11 months ago

San Francisco, CA $180,000.00-$220,000.00 8 months ago

San Francisco, CA $150,000.00-$175,000.00 2 months ago

Redwood City, CA $200,000.00-$275,000.00 3 weeks ago

United States $196,000.00-$220,500.00 1 week ago

Senior Software Engineer, Static Code Analysis

San Francisco, CA $170,000.00-$200,000.00 3 hours ago

San Francisco, CA $185,200.00-$274,400.00 2 weeks ago

San Francisco, CA $105,000.00-$215,000.00 4 days ago

San Francisco, CA $150,000.00-$170,000.00 3 weeks ago

Senior/Staff+ Software Engineer - Ledger Team

San Mateo, CA $100,000.00-$180,000.00 11 months ago

San Francisco, CA $170,000.00-$190,000.00 4 days ago

Senior Software Engineer - STCE New Products

San Francisco, CA $150,000.00-$275,000.00 1 year ago

Senior Software Engineer - STCE New Products

San Francisco, CA $150,000.00-$180,000.00 2 days ago

Were unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.