Macpower Digital Assets Edge
Lead Pyspark / SQL Engineer - Threat Detection & DataBricks
Macpower Digital Assets Edge, San Francisco, California, United States, 94199
Job Summary:
We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for creating equivalent PySpark log pipelines, rule configuration files, unit tests, and data validation checks, and deploying these pipelines using Terraform. A strong background in data engineering, particularly with large-scale log analytics, is essential. Requirements: 5-10+ years of experience as a
PySpark/SQL Engineer , with a strong focus on data engineering and analytics. Prior experience in
building threat detection or log analytics pipelines
using
PySpark, SQL, and Databricks. Hands-on experience with
Terraform
for deploying data infrastructure. Proficient in
PySpark
for large-scale data processing and transformation. Familiarity with cloud platforms such as
AWS or Azure
is preferred. Strong analytical skills and attention to detail when working with complex datasets. Proven ability to work effectively in
collaborative, cross-functional teams. Excellent verbal and written communication skills in English. Responsibilities:
Design and build
threat detection pipelines
using
PySpark, SQL, and Databricks. Support the
migration of detection rules and content
from a legacy platform to Databricks. Create and maintain
PySpark log pipelines
and associated
rule configuration files . Write unit tests to ensure pipeline accuracy and stability. Perform data validation checks to ensure data integrity. Deploy pipelines and infrastructure using
Terraform . Optimize existing data workflows and queries for performance and scalability. Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives.
We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for creating equivalent PySpark log pipelines, rule configuration files, unit tests, and data validation checks, and deploying these pipelines using Terraform. A strong background in data engineering, particularly with large-scale log analytics, is essential. Requirements: 5-10+ years of experience as a
PySpark/SQL Engineer , with a strong focus on data engineering and analytics. Prior experience in
building threat detection or log analytics pipelines
using
PySpark, SQL, and Databricks. Hands-on experience with
Terraform
for deploying data infrastructure. Proficient in
PySpark
for large-scale data processing and transformation. Familiarity with cloud platforms such as
AWS or Azure
is preferred. Strong analytical skills and attention to detail when working with complex datasets. Proven ability to work effectively in
collaborative, cross-functional teams. Excellent verbal and written communication skills in English. Responsibilities:
Design and build
threat detection pipelines
using
PySpark, SQL, and Databricks. Support the
migration of detection rules and content
from a legacy platform to Databricks. Create and maintain
PySpark log pipelines
and associated
rule configuration files . Write unit tests to ensure pipeline accuracy and stability. Perform data validation checks to ensure data integrity. Deploy pipelines and infrastructure using
Terraform . Optimize existing data workflows and queries for performance and scalability. Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives.