TOSCA Automation Engineer

Azure Databricks Administration & Management: Deploy, configure, and manage

Azure Databricks workspaces

in a scalable, cost-efficient, and secure manner. Administer

clusters, jobs, notebooks, and workflows , ensuring high availability and performance. Monitor and optimize

compute resource utilization and autoscaling

strategies to improve cost efficiency. Manage

Databricks Runtime versions, libraries, and dependencies

across environments. Security & Compliance: Implement and manage

Unity Catalog

for

fine-grained access control

and

data governance . Enforce

Role-Based Access Control (RBAC)

and integrate Databricks with

Azure Active Directory (AAD) . Ensure compliance with

SOC 2, HIPAA, GDPR, and internal security standards . Set up

audit logging, monitoring, and alerting

for security and operational insights. Performance Optimization & Troubleshooting: Tune

Apache Spark

workloads to improve query performance and resource efficiency. Analyze and troubleshoot

performance bottlenecks

in ETL and ML workloads. Optimize

Delta Lake storage, caching, and indexing strategies

for better query execution. Automation & Infrastructure as Code (IaC): Automate Databricks workspace deployment using

Terraform, ARM Templates, or Databricks REST API . Develop and maintain

CI/CD pipelines

for Databricks job deployment and configuration management. Implement

monitoring solutions

using

Azure Monitor, Prometheus, or Grafana . Collaboration & Integration: Work closely with

data engineers, data scientists, and DevOps teams

to support data pipelines and analytics workloads. Integrate Databricks with

Azure Data Lake, Azure Synapse Analytics, and Snowflake . Provide technical guidance and best practices for efficient

Spark job execution and cost optimization . Required Skills & Experience: 5+ years

of experience in

Azure Databricks administration and performance optimization . Expertise in

Apache Spark, PySpark, SQL, and Scala . Hands-on experience with

Databricks Unity Catalog, Delta Lake, and MLflow . Strong knowledge of

Azure cloud services

(Azure Data Lake, Azure Synapse, Azure Key Vault, etc.). Experience in

Infrastructure as Code (Terraform, Bicep, or ARM Templates) . Proficiency in

CI/CD pipeline automation

using

Azure DevOps, GitHub Actions, or Jenkins . Strong understanding of

network security, identity management (AAD), and encryption best practices . Excellent problem-solving skills and ability to troubleshoot complex Databricks workloads. Strong communication and documentation skills. Preferred Qualifications: Databricks Certified Associate or Professional

certification. Experience with

Azure Kubernetes Service (AKS) and serverless computing . Familiarity with

Kafka, Apache Airflow, and event-driven architectures . Knowledge of

Python, PowerShell, or Bash scripting

for automation.

#J-18808-Ljbffr