TechDigital Group
Experience Level Required:
7-10 years
Mandatory Required Skills:
Azure Databricks Apache Spark Data Modelling Azure Data Lake Creation Python Programming Preferred /Desired Skills:
predictive analytics Experience in Client Libraries Responsibilities:
Develop and maintain ETL (Extract, Transform, Load) pipelines using Databricks to process and transform large datasets. Collaborate with data engineers and data scientists to design and implement scalable and efficient data processing workflows. Build and optimize Apache Spark jobs and clusters on the Databricks platform. Develop and maintain data ingestion processes to acquire data from various sources and systems. Implement data quality checks and validation procedures to ensure accuracy and integrity of data. Perform data analysis and exploratory data mining to derive insights from complex datasets. Design and implement machine learning workflows using Databricks for predictive analytics and model training. Troubleshoot and debug issues related to data processing, performance, and job failures. Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. Stay updated with the latest advancements in big data technologies and contribute to the improvement of existing systems and processes. Requirements:
Solid experience in developing data processing workflows using Apache Spark and Databricks. Proficiency in programming languages like Python, Scala, or SQL for data manipulation and analytics. Strong understanding of distributed computing principles and experience with large-scale data processing frameworks. Familiarity with cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). Experience with data modeling, database systems, and SQL. Knowledge of machine learning concepts and experience with Client libraries and frameworks. Excellent problem-solving skills and ability to work independently and in a team. Strong communication skills to collaborate with stakeholders from different technical backgrounds.
#J-18808-Ljbffr
7-10 years
Mandatory Required Skills:
Azure Databricks Apache Spark Data Modelling Azure Data Lake Creation Python Programming Preferred /Desired Skills:
predictive analytics Experience in Client Libraries Responsibilities:
Develop and maintain ETL (Extract, Transform, Load) pipelines using Databricks to process and transform large datasets. Collaborate with data engineers and data scientists to design and implement scalable and efficient data processing workflows. Build and optimize Apache Spark jobs and clusters on the Databricks platform. Develop and maintain data ingestion processes to acquire data from various sources and systems. Implement data quality checks and validation procedures to ensure accuracy and integrity of data. Perform data analysis and exploratory data mining to derive insights from complex datasets. Design and implement machine learning workflows using Databricks for predictive analytics and model training. Troubleshoot and debug issues related to data processing, performance, and job failures. Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions. Stay updated with the latest advancements in big data technologies and contribute to the improvement of existing systems and processes. Requirements:
Solid experience in developing data processing workflows using Apache Spark and Databricks. Proficiency in programming languages like Python, Scala, or SQL for data manipulation and analytics. Strong understanding of distributed computing principles and experience with large-scale data processing frameworks. Familiarity with cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP). Experience with data modeling, database systems, and SQL. Knowledge of machine learning concepts and experience with Client libraries and frameworks. Excellent problem-solving skills and ability to work independently and in a team. Strong communication skills to collaborate with stakeholders from different technical backgrounds.
#J-18808-Ljbffr