Python Engineer

Must Have Technical/Functional Skills • Design , build and maintain scalable pipelines using Python/Databricks • Leverage Spark and SQL to process and transform large scale datasets • Develop and optimize ELT/ETL processes for high volume of data workflows • ETL: Hands on experience of building data pipelines. Proficiency in data integration platforms such as Apache Spark or Talend • Big Data: Experience of big data platforms such as Hadoop, Hive or Snowflake for data storage and processing • Data Warehousing & Database Management: Understanding of Data Warehousing concepts, Relational Oracle database design • Data Modeling & Design: Good exposure to data modeling techniques; design, optimization and maintenance of data models and data structures • Languages: Proficient in any programming languages commonly used in data engineering such as Python or Scala • DevOps: Exposure to concepts and enablers - CI/CD platforms, version control systems (e.g. GIT), automated quality control management • Data Quality & Controls: Exposure to data validation, cleansing, enrichment and data controls Roles & Responsibilities • Design , build and maintain scalable pipelines using Python/Databricks • Leverage Spark and SQL to process and transform large scale datasets • Develop and optimize ELT/ETL processes for high volume of data workflows • ETL: Hands on experience of building data pipelines. Proficiency in data integration platforms such as Apache Spark or Talend • Design, develop, and maintain ETL processes to extract, transform, and load data from various sources into our data warehouse. • Write complex SQL queries and PL/SQL scripts to perform data manipulation, validation, and transformation. • Develop and maintain data pipelines using Python and related libraries. • Optimize ETL processes and data pipelines for performance and scalability. • Collaborate with data analysts and other stakeholders to understand data requirements and develop solutions to meet their needs. • Implement data quality checks and monitoring to ensure data accuracy and consistency. • Troubleshoot and resolve data-related issues. • Create and maintain technical documentation for ETL processes, data pipelines, and database solutions. • Stay up-to-date with the latest trends and technologies in data management and analytics.

Salary Range: $100,000 to $120,000 per year