Diverse Lynx
Python Developer with Airflow
Diverse Lynx - Virginia Beach, Virginia, us, 23450
Work at Diverse Lynx
Overview
- View job
Overview
Python Developer with Airflow
Location Day One Onsite (Reston, VA)
Experience Level: Overall Professional Experience: 15+ years Apache Airflow Specific Experience: 3+ years Autosys Specific Experience: 3+ years AWS MWAA Specific Experience: Proven track record of successful migrations Job Summary:
We are seeking a highly accomplished and strategically minded hands-on Lead Engineer to drive the modernization of our customer's batch processing system. This pivotal role requires over 15 years of progressive experience in application development, integration of batch processes, with a specialized focus on Apache Airflow (and a deep understanding of legacy scheduling tools like Autosys). A critical component of this role will be leading the migration of existing Autosys workflows and jobs to AWS Managed Workflows for Apache Airflow (MWAA). Based in Virginia, this individual will play a key role in designing, implementing, and migrating batch processes from Autosys to Airflow. Responsibilities:
MWAA Migration & Strategy Leadership: Lead the end-to-end migration of existing Autosys workflows and jobs to AWS MWAA, including assessment, planning, re-platforming, testing, and validation support. Develop comprehensive migration strategies, roadmaps, and execution plans, minimizing disruption to ongoing operations. Design and implement robust, scalable, and secure batch pipelines within MWAA, translating Autosys concepts and logic into efficient Airflow DAGs. Serve as the primary technical expert for the Autosys to MWAA migration, providing guidance and troubleshooting support. Airflow & MWAA Expertise:
Architect, develop, migrate highly scalable, reliable, and efficient batch pipelines using Python and Airflow DAGs within MWAA. Manage and optimize MWAA environments, including infrastructure setup, configuration, monitoring, and scaling. Implement best practices for Airflow DAG development, testing, deployment, and version control (e.g., Git, CI/CD pipelines). Troubleshoot and resolve complex issues related to Airflow DAGs, infrastructure, and performance within MWAA. Team Leadership & Mentorship:
Provide technical leadership, guidance, and mentorship to a team of migration engineers, fostering a collaborative and high-performing environment focused on migration and cloud adoption. Conduct code reviews, provide constructive feedback, and ensure adherence to coding standards and best practices. Contribute to the professional development of team members through training, knowledge sharing, and coaching on Airflow, MWAA, and migration best practices. Architecture & Operational Excellence:
Collaborate with SMEs, product owners, and other stakeholders to define data pipeline requirements and translate them into robust Airflow solutions. Contribute to the overall batch process architecture strategy, identifying opportunities for optimization, automation, and innovation post-migration. Implement robust monitoring, alerting, and logging solutions for Airflow DAGs and MWAA environments. Ensure compliance with security best practices for process handling and access within Airflow and MWAA. Expertise in AWS:
Hands on knowledge and expertise in S3, SQS, SNS, EKS, ECS Fargate, Step Functions Evaluate, and Implement architecture options for implementing end to end batch processes on AWS Qualifications:
Bachelor's or master's degree in computer science, or a related quantitative field. 10+ years of overall professional experience in Java/ JEE and RDBMS databases, and software development. 5+ years of hands-on experience in Springboot and Spring batch 5+ years of hands-on experience specifically with Apache Airflow, including complex DAG development, custom operators/hooks, and plugin creation. 3+ years of hands-on experience with Autosys, demonstrating a solid understanding of its features, job scheduling, and administration. Proven track record of successfully leading and executing migrations from legacy scheduling tools (specifically Autosys) to cloud-native orchestration platforms like AWS MWAA. Demonstrable expert-level experience with AWS Managed Workflows for Apache Airflow (MWAA), including environment setup, configuration, scaling, security, and troubleshooting. 5+ years of hands-on experience in Python programming for Airflow DAG development. Extensive experience with AWS cloud services, including but not limited to CloudWatch, and IAM. Proven experience designing and implementing highly scalable and fault-tolerant batch pipelines. Experience with relational and NoSQL databases (e.g., PostgreSQL, Oracle, DynamoDB, MongoDB). Familiarity with CI/CD practices and tools (e.g., Jenkins, GitLab CI, AWS CodePipeline). Strong problem-solving skills and the ability to diagnose and resolve complex technical issues, especially during migration. Excellent communication, interpersonal, and leadership skills, with the ability to effectively collaborate with cross-functional teams and mentor junior engineers.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
Location Day One Onsite (Reston, VA)
Experience Level: Overall Professional Experience: 15+ years Apache Airflow Specific Experience: 3+ years Autosys Specific Experience: 3+ years AWS MWAA Specific Experience: Proven track record of successful migrations Job Summary:
We are seeking a highly accomplished and strategically minded hands-on Lead Engineer to drive the modernization of our customer's batch processing system. This pivotal role requires over 15 years of progressive experience in application development, integration of batch processes, with a specialized focus on Apache Airflow (and a deep understanding of legacy scheduling tools like Autosys). A critical component of this role will be leading the migration of existing Autosys workflows and jobs to AWS Managed Workflows for Apache Airflow (MWAA). Based in Virginia, this individual will play a key role in designing, implementing, and migrating batch processes from Autosys to Airflow. Responsibilities:
MWAA Migration & Strategy Leadership: Lead the end-to-end migration of existing Autosys workflows and jobs to AWS MWAA, including assessment, planning, re-platforming, testing, and validation support. Develop comprehensive migration strategies, roadmaps, and execution plans, minimizing disruption to ongoing operations. Design and implement robust, scalable, and secure batch pipelines within MWAA, translating Autosys concepts and logic into efficient Airflow DAGs. Serve as the primary technical expert for the Autosys to MWAA migration, providing guidance and troubleshooting support. Airflow & MWAA Expertise:
Architect, develop, migrate highly scalable, reliable, and efficient batch pipelines using Python and Airflow DAGs within MWAA. Manage and optimize MWAA environments, including infrastructure setup, configuration, monitoring, and scaling. Implement best practices for Airflow DAG development, testing, deployment, and version control (e.g., Git, CI/CD pipelines). Troubleshoot and resolve complex issues related to Airflow DAGs, infrastructure, and performance within MWAA. Team Leadership & Mentorship:
Provide technical leadership, guidance, and mentorship to a team of migration engineers, fostering a collaborative and high-performing environment focused on migration and cloud adoption. Conduct code reviews, provide constructive feedback, and ensure adherence to coding standards and best practices. Contribute to the professional development of team members through training, knowledge sharing, and coaching on Airflow, MWAA, and migration best practices. Architecture & Operational Excellence:
Collaborate with SMEs, product owners, and other stakeholders to define data pipeline requirements and translate them into robust Airflow solutions. Contribute to the overall batch process architecture strategy, identifying opportunities for optimization, automation, and innovation post-migration. Implement robust monitoring, alerting, and logging solutions for Airflow DAGs and MWAA environments. Ensure compliance with security best practices for process handling and access within Airflow and MWAA. Expertise in AWS:
Hands on knowledge and expertise in S3, SQS, SNS, EKS, ECS Fargate, Step Functions Evaluate, and Implement architecture options for implementing end to end batch processes on AWS Qualifications:
Bachelor's or master's degree in computer science, or a related quantitative field. 10+ years of overall professional experience in Java/ JEE and RDBMS databases, and software development. 5+ years of hands-on experience in Springboot and Spring batch 5+ years of hands-on experience specifically with Apache Airflow, including complex DAG development, custom operators/hooks, and plugin creation. 3+ years of hands-on experience with Autosys, demonstrating a solid understanding of its features, job scheduling, and administration. Proven track record of successfully leading and executing migrations from legacy scheduling tools (specifically Autosys) to cloud-native orchestration platforms like AWS MWAA. Demonstrable expert-level experience with AWS Managed Workflows for Apache Airflow (MWAA), including environment setup, configuration, scaling, security, and troubleshooting. 5+ years of hands-on experience in Python programming for Airflow DAG development. Extensive experience with AWS cloud services, including but not limited to CloudWatch, and IAM. Proven experience designing and implementing highly scalable and fault-tolerant batch pipelines. Experience with relational and NoSQL databases (e.g., PostgreSQL, Oracle, DynamoDB, MongoDB). Familiarity with CI/CD practices and tools (e.g., Jenkins, GitLab CI, AWS CodePipeline). Strong problem-solving skills and the ability to diagnose and resolve complex technical issues, especially during migration. Excellent communication, interpersonal, and leadership skills, with the ability to effectively collaborate with cross-functional teams and mentor junior engineers.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.