Lorven Technologies, Inc.
Google Cloud Platform Data Engineer
Lorven Technologies, Inc., Dallas, Texas, United States, 75215
Position: Google Cloud Platform Data Engineer
Location: Dallas, TX
Duration: Contract
Job description:
Design and implement real-time data ingestion pipelines using Apache NiFi, for structured and unstructured data.
Integrate data pipelines with the Google Cloud Platform ecosystem (BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, Composer, etc.).
Ensure low-latency, high-throughput data delivery to downstream systems for analytics and reporting.
Develop and optimize SQL queries for BigQuery and relational databases.
Write Python scripts for data transformation, automation, and custom NiFi processors when required.
Implement data quality, validation, and error-handling mechanisms within ingestion pipelines.
Collaborate with data analysts, data scientists, and platform engineers to deliver reliable datasets.
Ensure data security and compliance with Google Cloud Platform IAM, VPC Service Controls, and encryption standards.
Monitor and troubleshoot pipeline performance issues and provide operational support.
Document data flows, architecture, and best practices.
Required Skills & Qualifications
Proven experience with Apache NFL for real-time ingestion, routing, and transformation
Strong knowledge of the Google Cloud Platform ecosystem
#J-18808-Ljbffr
Location: Dallas, TX
Duration: Contract
Job description:
Design and implement real-time data ingestion pipelines using Apache NiFi, for structured and unstructured data.
Integrate data pipelines with the Google Cloud Platform ecosystem (BigQuery, Pub/Sub, Dataflow, Dataproc, Cloud Storage, Composer, etc.).
Ensure low-latency, high-throughput data delivery to downstream systems for analytics and reporting.
Develop and optimize SQL queries for BigQuery and relational databases.
Write Python scripts for data transformation, automation, and custom NiFi processors when required.
Implement data quality, validation, and error-handling mechanisms within ingestion pipelines.
Collaborate with data analysts, data scientists, and platform engineers to deliver reliable datasets.
Ensure data security and compliance with Google Cloud Platform IAM, VPC Service Controls, and encryption standards.
Monitor and troubleshoot pipeline performance issues and provide operational support.
Document data flows, architecture, and best practices.
Required Skills & Qualifications
Proven experience with Apache NFL for real-time ingestion, routing, and transformation
Strong knowledge of the Google Cloud Platform ecosystem
#J-18808-Ljbffr