TechDigital Group
Job Description:
The ideal candidate should possess excellent communication skills and have prior experience in a data lead or architect role, including hands-on work as an architect.
Key responsibilities include:
- Professional experience in Data Engineering using Google BigQuery (GBQ) and Google Cloud Platform (GCP), focusing on building data pipelines.
- Deep, hands-on experience with Google Data Products such as BigQuery, Dataflow, Dataproc, Dataprep, Cloud Composer, Airflow, and DAGs.
- Expertise in Python programming, with proficiency in PySpark and Pandas.
- Experience in Airflow, including creating DAGs, configuring variables, and scheduling.
- Knowledge of Big Data technologies and solutions like Spark, Hadoop, Hive, MapReduce, and scripting languages such as YAML and Python.
- Optional: Experience with DBT for creating lineage in GCP.
- Experience working in a DevSecOps environment, particularly with CI/CD pipelines.
- Design and development of ETL/ELT frameworks using BigQuery, with a strong understanding of BigQuery features like nested queries, clustering, and partitioning.
- Experience with data integration, transformation, quality, and lineage tools.
- Ability to automate data loads from BigQuery using APIs or scripting languages.
- Managing end-to-end data engineering lifecycle, including non-functional requirements and operations.
- Solution design skills, including prototyping, usability testing, and data visualization literacy.
- Proficiency with SQL and NoSQL modern data stores.