Diverse Lynx
Job Tittle:
Azure Databricks Engineer Location: Louisville, Kentucky Duration: 12+ Months Job Type: Long Term Contract
Note: Hybrid Job (Local Preferred) Job Description:
Overall 10+ Years of experience We are seeking an experienced Azure Databricks Engineer to design, build, and optimize scalable data processing pipelines on the Microsoft Azure cloud platform. The ideal candidate will have strong expertise in Azure Data Services, Databricks, and data engineering best practices, enabling efficient data integration, transformation, and analytics to support business intelligence and AI/ML initiatives.
Key Responsibilities:
Design, develop, and maintain data pipelines using Azure Databricks, Azure Data Factory (ADF), and Azure Synapse Analytics. Implement ETL/ELT workflows to integrate structured and unstructured data from multiple sources. Optimize Spark and PySpark jobs for performance, scalability, and cost efficiency. Collaborate with data scientists, analysts, and business stakeholders to deliver reliable data solutions. Develop and enforce data quality, validation, and governance frameworks. Automate deployment and version control using DevOps pipelines (CI/CD) and Git. Monitor, troubleshoot, and enhance data systems to ensure high availability and performance. Work with Azure storage solutions (ADLS Gen2, Blob Storage) and Azure Key Vault for secure data management. Support advanced analytics and machine learning workloads using Databricks ML features.
Required Skills & Qualifications:
Bachelor's degree in Computer Science, Information Systems, or related field. 3-6 years of experience in data engineering or big data development.
Strong hands-on experience with:
Azure Databricks (Spark, PySpark, Delta Lake) Azure Data Factory, Synapse, Data Lake Storage (ADLS) SQL, Python, and Scala Understanding of data modeling, data warehousing, and ETL best practices. Experience with Azure DevOps, Git, and CI/CD pipelines. Familiarity with JSON, Parquet, Avro, and other data formats. Excellent problem-solving, analytical, and communication skills.
Preferred Qualifications:
Experience integrating Databricks with Power BI or other visualization tools. Exposure to ML Flow, Delta Live Tables, or Unity Catalog. Knowledge of cloud cost optimization strategies and infrastructure-as-code (IaC) using Terraform or ARM templates. Azure Data Engineer Associate or Databricks Certified Data Engineer certification preferred.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Azure Databricks Engineer Location: Louisville, Kentucky Duration: 12+ Months Job Type: Long Term Contract
Note: Hybrid Job (Local Preferred) Job Description:
Overall 10+ Years of experience We are seeking an experienced Azure Databricks Engineer to design, build, and optimize scalable data processing pipelines on the Microsoft Azure cloud platform. The ideal candidate will have strong expertise in Azure Data Services, Databricks, and data engineering best practices, enabling efficient data integration, transformation, and analytics to support business intelligence and AI/ML initiatives.
Key Responsibilities:
Design, develop, and maintain data pipelines using Azure Databricks, Azure Data Factory (ADF), and Azure Synapse Analytics. Implement ETL/ELT workflows to integrate structured and unstructured data from multiple sources. Optimize Spark and PySpark jobs for performance, scalability, and cost efficiency. Collaborate with data scientists, analysts, and business stakeholders to deliver reliable data solutions. Develop and enforce data quality, validation, and governance frameworks. Automate deployment and version control using DevOps pipelines (CI/CD) and Git. Monitor, troubleshoot, and enhance data systems to ensure high availability and performance. Work with Azure storage solutions (ADLS Gen2, Blob Storage) and Azure Key Vault for secure data management. Support advanced analytics and machine learning workloads using Databricks ML features.
Required Skills & Qualifications:
Bachelor's degree in Computer Science, Information Systems, or related field. 3-6 years of experience in data engineering or big data development.
Strong hands-on experience with:
Azure Databricks (Spark, PySpark, Delta Lake) Azure Data Factory, Synapse, Data Lake Storage (ADLS) SQL, Python, and Scala Understanding of data modeling, data warehousing, and ETL best practices. Experience with Azure DevOps, Git, and CI/CD pipelines. Familiarity with JSON, Parquet, Avro, and other data formats. Excellent problem-solving, analytical, and communication skills.
Preferred Qualifications:
Experience integrating Databricks with Power BI or other visualization tools. Exposure to ML Flow, Delta Live Tables, or Unity Catalog. Knowledge of cloud cost optimization strategies and infrastructure-as-code (IaC) using Terraform or ARM templates. Azure Data Engineer Associate or Databricks Certified Data Engineer certification preferred.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.