Diverse Lynx
Job Description:
Data Engineer with Dremio experience is responsible for developing and maintaining robust data infrastructure and pipelines using Dremio's query engine and Lakehouse platform to enable accessible, reliable, and secure data for enterprise use cases, which includes gathering data requirements, transforming raw data, supporting data modeling, and collaborating with data scientists and analysts to support business needs and provide data for analysis and reporting.
Key Responsibilities
Data Architecture & Pipelines: Design, build, and deploy scalable data pipelines and platforms to manage and transform data from various sources into a usable format.
Dremio Integration: Leverage Dremio's query engine and data virtualization capabilities to provide a unified access point for enterprise data, allowing for self-service analytics and simplifying data management.
Data Requirements Gathering: Identify and understand customer needs and the intended use of data to inform database requirements and data strategy.
Data Modeling: Apply data modeling techniques to organize and structure data, ensuring it aligns with business objectives and requirements.
Data Transformation & Integration: Utilize ETL (Extract, Transform, Load) processes and tools to convert raw data into formats suitable for data scientists, analysts, and AI platforms.
Data Quality & Governance: Implement measures to ensure data quality, security, and governance, including monitoring and maintaining metadata about the data.
Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand their data needs and deliver solutions that support their analytical projects.
Essential Skills
Dremio Proficiency: Experience with Dremio's query engine, Sonar, and other Lakehouse technologies.
SQL: Strong command of Structured Query Language (SQL) for data querying and manipulation.
Programming: Proficiency in programming languages like Python for ETL and data pipeline development.
Cloud Technologies: Experience with cloud platforms such as AWS, including services like S3, Glue, and Redshift.
Data Modeling: Knowledge of various data modeling techniques.
ETL Tools: Experience with ETL tools for data extraction, transformation, and loading.
Data Governance: Understanding of data governance principles, metadata management, and data security.
Agile Methodologies: Experience working in an agile environment using Scrum or Kanban.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Key Responsibilities
Data Architecture & Pipelines: Design, build, and deploy scalable data pipelines and platforms to manage and transform data from various sources into a usable format.
Dremio Integration: Leverage Dremio's query engine and data virtualization capabilities to provide a unified access point for enterprise data, allowing for self-service analytics and simplifying data management.
Data Requirements Gathering: Identify and understand customer needs and the intended use of data to inform database requirements and data strategy.
Data Modeling: Apply data modeling techniques to organize and structure data, ensuring it aligns with business objectives and requirements.
Data Transformation & Integration: Utilize ETL (Extract, Transform, Load) processes and tools to convert raw data into formats suitable for data scientists, analysts, and AI platforms.
Data Quality & Governance: Implement measures to ensure data quality, security, and governance, including monitoring and maintaining metadata about the data.
Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand their data needs and deliver solutions that support their analytical projects.
Essential Skills
Dremio Proficiency: Experience with Dremio's query engine, Sonar, and other Lakehouse technologies.
SQL: Strong command of Structured Query Language (SQL) for data querying and manipulation.
Programming: Proficiency in programming languages like Python for ETL and data pipeline development.
Cloud Technologies: Experience with cloud platforms such as AWS, including services like S3, Glue, and Redshift.
Data Modeling: Knowledge of various data modeling techniques.
ETL Tools: Experience with ETL tools for data extraction, transformation, and loading.
Data Governance: Understanding of data governance principles, metadata management, and data security.
Agile Methodologies: Experience working in an agile environment using Scrum or Kanban.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.