EDI Staffing
Key Responsibilities: Databricks Handon SME Engineer
Architect, configure, and optimize Databricks Pipelines for large-scale data processing within an Azure Data Lakehouse environment.
Set up and manage Azure infrastructure components including Databricks Workspaces, Azure Containers (AKS/ACI), Storage Accounts, and Networking.
Design and implement a monitoring and observability framework using tools like Azure Monitor, Log Analytics, and Prometheus/Grafana.
Collaborate with platform and data engineering teams to enable microservices-based architecture for scalable and modular data solutions.
Drive automation and CI/CD practices using Terraform, ARM templates, and GitHub Actions/Azure DevOps.
Required Skills & Experience:
Strong hands-on experience with Azure Databricks, Delta Lake, and Apache Spark.
Deep understanding of Azure services: Resource Manager, AKS, ACR, Key Vault, and Networking.
Proven experience in microservices architecture and container orchestration.
Expertise in infrastructure-as-code, scripting (Python, Bash), and DevOps tooling.
Familiarity with data governance, security, and cost optimization in cloud environments.
Bonus:
Experience with event-driven architectures (Kafka/Event Grid).
Knowledge of data mesh principles and distributed data ownership.