Gandiv Insights LLC
Databricks Engineer - California
Gandiv Insights LLC, Los Angeles, California, United States, 90079
Databricks Engineer
Work Location: Remote/CA. Engagement Type: Remote (candidate should be in PST). Project Duration: 1 year+.
Primary Skill Set
Databricks, Apache Spark, Python, SQL, Scala (optional), ETL/ELT development, Delta Lake, Cloud platforms (AWS, Azure, GCP)
Data modeling, cross‑functional collaboration, communication
Secondary Skill Set
Airflow, dbt, Kafka, Hadoop, MLflow, Unity Catalog, Delta Live Tables, cluster optimization
Data governance, security and compliance, Databricks certifications
Required Qualifications
Experience: 5+ years in data engineering with hands‑on experience using Databricks and Apache Spark.
Programming: Proficiency in Python and SQL; experience with Scala is a plus.
Cloud Platforms: AWS, Azure, or GCP (e.g., S3, Glue, Redshift; Data Factory, Synapse; or GCP equivalents).
Data Engineering Tools: Familiarity with Airflow, Kafka, dbt.
Data Modeling: Experience designing data models for analytics and machine learning.
Collaboration: Proven ability to work in cross‑functional teams and communicate effectively with non‑technical stakeholders.
Key Responsibilities
Design and implement robust ETL/ELT pipelines using Databricks, PySpark, and Delta Lake to process structured and unstructured data efficiently.
Tune and optimize Databricks clusters and notebooks for performance, scalability, and cost efficiency.
Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions that meet business needs.
Leverage cloud platforms (AWS, Azure, GCP) to build and deploy data solutions, ensuring seamless integration with existing infrastructure.
Develop and maintain data models that support analytics and machine learning workflows.
Implement automated testing, monitoring, and alerting mechanisms to ensure data pipeline reliability and data quality.
Maintain comprehensive documentation of data workflows and adhere to best practices in coding, version control, and data governance.
#J-18808-Ljbffr
Primary Skill Set
Databricks, Apache Spark, Python, SQL, Scala (optional), ETL/ELT development, Delta Lake, Cloud platforms (AWS, Azure, GCP)
Data modeling, cross‑functional collaboration, communication
Secondary Skill Set
Airflow, dbt, Kafka, Hadoop, MLflow, Unity Catalog, Delta Live Tables, cluster optimization
Data governance, security and compliance, Databricks certifications
Required Qualifications
Experience: 5+ years in data engineering with hands‑on experience using Databricks and Apache Spark.
Programming: Proficiency in Python and SQL; experience with Scala is a plus.
Cloud Platforms: AWS, Azure, or GCP (e.g., S3, Glue, Redshift; Data Factory, Synapse; or GCP equivalents).
Data Engineering Tools: Familiarity with Airflow, Kafka, dbt.
Data Modeling: Experience designing data models for analytics and machine learning.
Collaboration: Proven ability to work in cross‑functional teams and communicate effectively with non‑technical stakeholders.
Key Responsibilities
Design and implement robust ETL/ELT pipelines using Databricks, PySpark, and Delta Lake to process structured and unstructured data efficiently.
Tune and optimize Databricks clusters and notebooks for performance, scalability, and cost efficiency.
Work closely with data scientists, analysts, and business stakeholders to understand data requirements and deliver solutions that meet business needs.
Leverage cloud platforms (AWS, Azure, GCP) to build and deploy data solutions, ensuring seamless integration with existing infrastructure.
Develop and maintain data models that support analytics and machine learning workflows.
Implement automated testing, monitoring, and alerting mechanisms to ensure data pipeline reliability and data quality.
Maintain comprehensive documentation of data workflows and adhere to best practices in coding, version control, and data governance.
#J-18808-Ljbffr