Datapelago
Principal Data Processing Engineer
DataPelago is revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing. The Opportunity
As a Principal Data Processing Engineer, you will be a key technical leader of core execution engine components of our data processing engine. You will lead architecture, design, and implementation that will enhance functional breadth, performance, scale, and reliability of the engine to deliver a product that will redefine how users extract intelligence from their data. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers. What You'll Do
Architectural Leadership: Drive the evolution of our parallel and distributed execution engine architecture, with a strong focus on leveraging accelerated computing technologies. End to End Ownership: Lead the execution engine team in the complete lifecycle of design, implementation, and rollout of an enterprise-grade product. Core Development: Individually design, implement, test, and maintain critical components of the data processing execution engine. Innovation and Differentiation: Analyze technology advances from industry and academia to identify opportunities for the engine to enhance technology and product leadership. Collaboration and Mentorship: Partner effectively with engineering, product management, and customer success teams. Guide and mentor engineers on the execution engine team. Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain highest product quality, security, efficiency, & productivity. What You'll Bring
Bachelors degree in Computer Science, or a related field with 15+ years of relevant experience OR a Masters degree in Computer Science or a related field with 10+ years of relevant experience. 10+ years of deep technical experience in developing core components of enterprise-grade database or analytics execution engines designed for large-scale data processing. Proven expertise in developing high-performance parallel implementations of data processing operators and functions on rich data types. Significant experience developing for and understanding the internals of platforms such as Apache Spark, Apache Flink, Apache Doris, Apache Gluten, Velox, Apache DataFusion, or Apache DataFusion Comet is highly preferred. Demonstrated experience leading teams of 10+ engineers in the design, development, and successful release of high-performance data processing engines for large production deployments. Exceptional programming skills in C, C++, and Rust. Extensive development experience in Linux environments. Strong analytical and problem-solving skills with a passion for performance optimization. Excellent communication and collaboration skills, with the ability to articulate complex technical concepts to both technical and non-technical audiences. Location Considerations
We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations. This specific position will be based at our headquarters in Mountain View, CA. Why Join DataPelago?
Technical Leadership: Take a leadership role in shaping the architecture and development of our core parallel analytics engine. Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated computing and data processing. Significant Impact: Your contributions will directly impact the performance and scalability of our mission-critical platform. Mentorship and Growth: Mentor and guide other talented engineers while expanding your own technical expertise. Competitive compensation, stock options, comprehensive benefits package, leadership development opportunities.
DataPelago is revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing. The Opportunity
As a Principal Data Processing Engineer, you will be a key technical leader of core execution engine components of our data processing engine. You will lead architecture, design, and implementation that will enhance functional breadth, performance, scale, and reliability of the engine to deliver a product that will redefine how users extract intelligence from their data. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers. What You'll Do
Architectural Leadership: Drive the evolution of our parallel and distributed execution engine architecture, with a strong focus on leveraging accelerated computing technologies. End to End Ownership: Lead the execution engine team in the complete lifecycle of design, implementation, and rollout of an enterprise-grade product. Core Development: Individually design, implement, test, and maintain critical components of the data processing execution engine. Innovation and Differentiation: Analyze technology advances from industry and academia to identify opportunities for the engine to enhance technology and product leadership. Collaboration and Mentorship: Partner effectively with engineering, product management, and customer success teams. Guide and mentor engineers on the execution engine team. Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD, and issue resolution to maintain highest product quality, security, efficiency, & productivity. What You'll Bring
Bachelors degree in Computer Science, or a related field with 15+ years of relevant experience OR a Masters degree in Computer Science or a related field with 10+ years of relevant experience. 10+ years of deep technical experience in developing core components of enterprise-grade database or analytics execution engines designed for large-scale data processing. Proven expertise in developing high-performance parallel implementations of data processing operators and functions on rich data types. Significant experience developing for and understanding the internals of platforms such as Apache Spark, Apache Flink, Apache Doris, Apache Gluten, Velox, Apache DataFusion, or Apache DataFusion Comet is highly preferred. Demonstrated experience leading teams of 10+ engineers in the design, development, and successful release of high-performance data processing engines for large production deployments. Exceptional programming skills in C, C++, and Rust. Extensive development experience in Linux environments. Strong analytical and problem-solving skills with a passion for performance optimization. Excellent communication and collaboration skills, with the ability to articulate complex technical concepts to both technical and non-technical audiences. Location Considerations
We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations. This specific position will be based at our headquarters in Mountain View, CA. Why Join DataPelago?
Technical Leadership: Take a leadership role in shaping the architecture and development of our core parallel analytics engine. Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated computing and data processing. Significant Impact: Your contributions will directly impact the performance and scalability of our mission-critical platform. Mentorship and Growth: Mentor and guide other talented engineers while expanding your own technical expertise. Competitive compensation, stock options, comprehensive benefits package, leadership development opportunities.