Diverse Lynx
Job Title:
PySpark/Glue Developer
Location:
Philadelphia, PA
Duration:
Fulltime
Skills
Minimum 7 Years of experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL.
Experience in designing, building and deploying Python based applications.
Experience in writing complex SQL queries/procedures using relational databases such as SQL Server or Oracle.
Experience with version control systems like Git and CI/CD pipelines.
Experience with Delta Lake APIs is a plus.
Experience with Docker and Kubernetes is a plus.
Knowledge of AWS services such as S3, Athena, Glue, Lambda, Redshift, or other cloud platforms is a plus.
Proficiency in PySpark, Delta Lake, shell scripting, AWS services (Glue, Lambda, Athena, Step Functions, EventBridge, EC2, Batch, MSK, etc.), PostgreSQL, RedShift, DynamoDB, Jenkins, Udeploy, and GitHub.
Roles & Responsibilities
Designing, implementing, and maintaining PySpark jobs for extracting raw data, applying transformations, and writing results into files or tables on an Apache Spark cluster.
Writing complex queries according to business rules and data extraction logic.
Tuning queries and optimizing performance of various components as needed.
Collaborating with development leads, system architects, and other teams to manage dependencies, risks, and issues.
Contributing across all phases of the development lifecycle.
Supporting testers and resolving defects.
Updating status, risks, issues, and impediments during daily scrum calls to the customer and tracking them to closure.
Writing testable, scalable, and efficient code.
Preparing necessary documentation as needed.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence, and proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
#J-18808-Ljbffr
PySpark/Glue Developer
Location:
Philadelphia, PA
Duration:
Fulltime
Skills
Minimum 7 Years of experience in design, development and deployment of big data applications and ETL jobs using PySpark APIs/sparkSQL.
Experience in designing, building and deploying Python based applications.
Experience in writing complex SQL queries/procedures using relational databases such as SQL Server or Oracle.
Experience with version control systems like Git and CI/CD pipelines.
Experience with Delta Lake APIs is a plus.
Experience with Docker and Kubernetes is a plus.
Knowledge of AWS services such as S3, Athena, Glue, Lambda, Redshift, or other cloud platforms is a plus.
Proficiency in PySpark, Delta Lake, shell scripting, AWS services (Glue, Lambda, Athena, Step Functions, EventBridge, EC2, Batch, MSK, etc.), PostgreSQL, RedShift, DynamoDB, Jenkins, Udeploy, and GitHub.
Roles & Responsibilities
Designing, implementing, and maintaining PySpark jobs for extracting raw data, applying transformations, and writing results into files or tables on an Apache Spark cluster.
Writing complex queries according to business rules and data extraction logic.
Tuning queries and optimizing performance of various components as needed.
Collaborating with development leads, system architects, and other teams to manage dependencies, risks, and issues.
Contributing across all phases of the development lifecycle.
Supporting testers and resolving defects.
Updating status, risks, issues, and impediments during daily scrum calls to the customer and tracking them to closure.
Writing testable, scalable, and efficient code.
Preparing necessary documentation as needed.
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence, and proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
#J-18808-Ljbffr