Senior Data Engineer

Required Skills and Experience: Strong programming skills in Python and Java. Hands-on experience with cloud platforms, preferably GCP. Experience working with Docker containers. Knowledge of environment management tools such as venv, pip, poetry, etc. Experience with orchestrators like Vertex AI pipelines, Airflow, etc. Data engineering and feature engineering techniques. Proficiency in Apache Spark, Apache Beam, or Apache Flink. Advanced SQL knowledge. Understanding of streaming concepts such as windowing, late arrival, triggers, etc. Hands-on experience with distributed computing. Experience in designing data architecture. Knowledge of storage and compute options and their appropriate use cases. Understanding of cluster and pipeline optimization strategies. Exposure to GCP tools for developing end-to-end data pipelines, including data ingestion from traditional databases and API sources. Business mindset to interpret data for BI and analytics. Experience with CI/CD pipelines, deployment methodologies, and Infrastructure as Code (e.g., Terraform). Good to have: Hands-on experience with Kubernetes. Good to have: Experience with vector databases like Qdrant. Experience with GCP tools: Ingestion:

Pub/Sub, Cloud Functions, App Engine, Kubernetes Engine, Kafka, Microservices. Processing:

Cloud Dataproc, Cloud Dataflow, Apache Spark, Apache Flink. CI/CD:

Bitbucket, Jenkins, GitLab.

#J-18808-Ljbffr