Iqclarity
Lead Data Engineer
Iqclarity - Irving, Texas, United States, 75084
Work at Iqclarity
Overview
- View job
Overview
Get AI-powered advice on this job and more exclusive features.
This range is provided by IQ Clarity, LLC. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.
Base pay range
$180,000.00/yr - $200,000.00/yr We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, data warehousing, and cloud platforms. This role will involve leading the design and development of the enterprise data platform, including robust data pipelines, development frameworks, and scalable data architectures to support our data-driven initiatives. You will collaborate closely with cross-functional teams to ensure data integrity, availability, and accessibility across the organization. Responsibilities: • Actively contribute to the overall vision and strategy for data, including topics like architecture, infrastructure and governance. • Lead the design, development, and maintenance of scalable data pipelines, ETL/ELT processes, and data models. • Use big data tools such as Spark, Flink, Trino and Kafka to ensure efficient data ingestion, storage, and processing. • Design and implement reusable data processing frameworks that support different data types (structured, semi-structured, unstructured) and patterns (batch, streaming, CDC) • Develop and optimize data storage solutions, including data warehousing, data lakes, and NoSQL databases on AWS. • Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver solutions that meet those needs. • Ensure data quality, governance, and security through the implementation of best practices and robust monitoring systems. • Mentor and provide technical leadership to junior data engineers, fostering a culture of continuous learning and improvement. Requirements: • Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred. • 7+ years of proven experience as a Data Engineer or in a similar role, with a strong focus on data warehousing, ETL/ELT processes, and data modeling. • Deep experience in AWS data services and data lakes (Redshift, Glue, Airflow, S3 data lakes) • Deep proficiency in Python and other programming languages such as Java or Scala. • Experience with real-time streaming (the team has primarily focused on batch-processing and is moving to real-time streaming) • Experience migrating workloads (Such as Kubernetes on prem, VMWare, and Openstack) • Hands-on experience with big data technologies such as Kafka, Spark, HDFS, Flink, Trino, Iceberg, etc. • Strong SQL skills and experience working with both relational and NoSQL databases. • Experience designing and implementing data processing frameworks that establish reusable development patterns. • Experience configuring workflows within orchestration tools such as Apache Airflow. • Experience creating data solutions in a hybrid cloud environment, particularly running workflows on Kubernetes. • Experience supporting intensive workflows that make use of HPC (high performance compute) and/or supporting terabyte to petabyte scale workflows. • Experience with high performance file systems such as Lustre or FSx. • Excellent problem-solving and analytical skills. • Strong communication and collaboration skills, with the ability to work effectively in a team environment. • Experience with building and managing data pipelines in a multi-cloud environment. • Familiarity with data governance and data quality frameworks. • Experience in mentoring or leading a team of data engineers. • Terraform experience is a big plus Seniority level
Seniority level Mid-Senior level Employment type
Employment type Full-time Job function
Industries IT Services and IT Consulting Referrals increase your chances of interviewing at IQ Clarity, LLC by 2x Get notified about new Data Engineer jobs in
Irving, TX . Dallas, TX $128,000.00-$181,000.00 3 months ago Dallas, TX $104,000.00-$130,000.00 2 weeks ago Data Engineer [Multiple Positions Available]
Irving, TX $80,000.00-$100,000.00 1 week ago Software Engineer, Machine Learning (Multiple Levels) - Slack
Dallas, TX $167,300.00-$334,600.00 2 days ago Irving, TX $165,000.00-$200,000.00 2 weeks ago Plano, TX $111,000.00-$136,500.00 3 weeks ago Dallas-Fort Worth Metroplex $65,000.00-$75,000.00 1 day ago Dallas, TX $140,000.00-$160,000.00 1 month ago Plano, TX $120,000.00-$150,000.00 2 weeks ago We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr
$180,000.00/yr - $200,000.00/yr We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, data warehousing, and cloud platforms. This role will involve leading the design and development of the enterprise data platform, including robust data pipelines, development frameworks, and scalable data architectures to support our data-driven initiatives. You will collaborate closely with cross-functional teams to ensure data integrity, availability, and accessibility across the organization. Responsibilities: • Actively contribute to the overall vision and strategy for data, including topics like architecture, infrastructure and governance. • Lead the design, development, and maintenance of scalable data pipelines, ETL/ELT processes, and data models. • Use big data tools such as Spark, Flink, Trino and Kafka to ensure efficient data ingestion, storage, and processing. • Design and implement reusable data processing frameworks that support different data types (structured, semi-structured, unstructured) and patterns (batch, streaming, CDC) • Develop and optimize data storage solutions, including data warehousing, data lakes, and NoSQL databases on AWS. • Collaborate with data scientists, analysts, and business stakeholders to understand data needs and deliver solutions that meet those needs. • Ensure data quality, governance, and security through the implementation of best practices and robust monitoring systems. • Mentor and provide technical leadership to junior data engineers, fostering a culture of continuous learning and improvement. Requirements: • Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred. • 7+ years of proven experience as a Data Engineer or in a similar role, with a strong focus on data warehousing, ETL/ELT processes, and data modeling. • Deep experience in AWS data services and data lakes (Redshift, Glue, Airflow, S3 data lakes) • Deep proficiency in Python and other programming languages such as Java or Scala. • Experience with real-time streaming (the team has primarily focused on batch-processing and is moving to real-time streaming) • Experience migrating workloads (Such as Kubernetes on prem, VMWare, and Openstack) • Hands-on experience with big data technologies such as Kafka, Spark, HDFS, Flink, Trino, Iceberg, etc. • Strong SQL skills and experience working with both relational and NoSQL databases. • Experience designing and implementing data processing frameworks that establish reusable development patterns. • Experience configuring workflows within orchestration tools such as Apache Airflow. • Experience creating data solutions in a hybrid cloud environment, particularly running workflows on Kubernetes. • Experience supporting intensive workflows that make use of HPC (high performance compute) and/or supporting terabyte to petabyte scale workflows. • Experience with high performance file systems such as Lustre or FSx. • Excellent problem-solving and analytical skills. • Strong communication and collaboration skills, with the ability to work effectively in a team environment. • Experience with building and managing data pipelines in a multi-cloud environment. • Familiarity with data governance and data quality frameworks. • Experience in mentoring or leading a team of data engineers. • Terraform experience is a big plus Seniority level
Seniority level Mid-Senior level Employment type
Employment type Full-time Job function
Industries IT Services and IT Consulting Referrals increase your chances of interviewing at IQ Clarity, LLC by 2x Get notified about new Data Engineer jobs in
Irving, TX . Dallas, TX $128,000.00-$181,000.00 3 months ago Dallas, TX $104,000.00-$130,000.00 2 weeks ago Data Engineer [Multiple Positions Available]
Irving, TX $80,000.00-$100,000.00 1 week ago Software Engineer, Machine Learning (Multiple Levels) - Slack
Dallas, TX $167,300.00-$334,600.00 2 days ago Irving, TX $165,000.00-$200,000.00 2 weeks ago Plano, TX $111,000.00-$136,500.00 3 weeks ago Dallas-Fort Worth Metroplex $65,000.00-$75,000.00 1 day ago Dallas, TX $140,000.00-$160,000.00 1 month ago Plano, TX $120,000.00-$150,000.00 2 weeks ago We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr