Twine
Join a dynamic team as a data engineering expert, responsible for designing and building robust, scalable ETL/ELT pipelines to process both structured and unstructured data from diverse sources in real-time and batch environments. You will architect and optimize data lakes and warehouses on leading cloud platforms such as AWS, GCP, or Azure, ensuring efficiency, reliability, and cost-effectiveness. This role involves close collaboration with cross-functional teams to deliver data-driven products, machine learning feature stores, and analytical datasets, while leading initiatives in data governance, quality assurance, and observability. You will also mentor junior engineers and contribute to the evolution of technical best practices and platform enhancements.
Deliverables
Design, implement, and maintain scalable ETL/ELT pipelines for real-time and batch data processing
Architect and optimize cloud-based data lakes and warehouses for performance and cost efficiency
Develop and manage data models, transformation workflows, and orchestration systems (e.g., Airflow, Prefect, Dagster)
Build and maintain real-time streaming systems and internal/external APIs
Lead data governance, quality assurance, lineage tracking, and observability initiatives
Develop monitoring and alerting solutions using tools such as Datadog and Prometheus
Mentor junior engineers and contribute to technical design and best practices
Research and integrate new technologies to enhance the data platform
Requirements
6+ years of experience in data engineering or analytics infrastructure roles
Advanced proficiency in Python or Scala for data engineering tasks
Deep experience with cloud-native environments, preferably AWS
Expertise in data modeling, warehousing, orchestration, distributed computing, and CI/CD practices
Familiarity with containerization, NoSQL systems, streaming data platforms, and analytics tools
Experience with orchestration tools such as Airflow, Prefect, or Dagster
Strong understanding of data governance, quality assurance, and observability best practices
Excellent communication skills in English, with an accent similar to native speakers
Ability to work fully remotely and collaborate across time zones
Skills or interest in MLOps, infrastructure-as-code, and GenAI/LLM-based systems are a plus
About Twine Twine is a leading freelance marketplace connecting top freelancers, consultants, and contractors with companies needing creative and tech expertise. Trusted by Fortune 500 companies and innovative startups alike, Twine enables companies to scale their teams globally.
Our Mission Twine's mission is to empower creators and businesses to thrive in an AI-driven, freelance-first world.
#J-18808-Ljbffr
Deliverables
Design, implement, and maintain scalable ETL/ELT pipelines for real-time and batch data processing
Architect and optimize cloud-based data lakes and warehouses for performance and cost efficiency
Develop and manage data models, transformation workflows, and orchestration systems (e.g., Airflow, Prefect, Dagster)
Build and maintain real-time streaming systems and internal/external APIs
Lead data governance, quality assurance, lineage tracking, and observability initiatives
Develop monitoring and alerting solutions using tools such as Datadog and Prometheus
Mentor junior engineers and contribute to technical design and best practices
Research and integrate new technologies to enhance the data platform
Requirements
6+ years of experience in data engineering or analytics infrastructure roles
Advanced proficiency in Python or Scala for data engineering tasks
Deep experience with cloud-native environments, preferably AWS
Expertise in data modeling, warehousing, orchestration, distributed computing, and CI/CD practices
Familiarity with containerization, NoSQL systems, streaming data platforms, and analytics tools
Experience with orchestration tools such as Airflow, Prefect, or Dagster
Strong understanding of data governance, quality assurance, and observability best practices
Excellent communication skills in English, with an accent similar to native speakers
Ability to work fully remotely and collaborate across time zones
Skills or interest in MLOps, infrastructure-as-code, and GenAI/LLM-based systems are a plus
About Twine Twine is a leading freelance marketplace connecting top freelancers, consultants, and contractors with companies needing creative and tech expertise. Trusted by Fortune 500 companies and innovative startups alike, Twine enables companies to scale their teams globally.
Our Mission Twine's mission is to empower creators and businesses to thrive in an AI-driven, freelance-first world.
#J-18808-Ljbffr