Shop.ca
Overview
Join to apply for the
Staff Data Engineer
role at
SHOP.COM . Market America, a product brokerage and Internet marketing company that specializes in One-to-One Marketing, is seeking an experienced
Staff Data Engineer
for our IT team. As a senior member of the Data Engineering team, you will help millions of customers on our SHOP.COM and Market America Worldwide multi-country and multi-language global eCommerce websites find what they want across categories, merchant offers, products and taxonomy. We manage thousands of 3rd party affiliates/feeds through ETL and data ingestion pipelines before feeding our search systems. We operate multiple orchestration pipelines supporting data for products, store offers, analytics, customer behavioral profiles, segments, logs, and more. This is a highly visible role with the opportunity to make a significant impact. The data engineering team processes large amounts of data we import and collect, enrich content, integrate pricing and taxonomy, develop category classifier nodes and incorporate machine learning within the pipeline. Key Responsibilities
Must have minimum of 10-12 years of hands-on development experience implementing batch and event-driven applications using Java, Kafka, Spark, Scala, PySpark and Python Experience with Apache Kafka and Connectors, Java, Spring Boot in building event-driven services, Python in building ML pipelines Develop data pipelines responsible for ingesting large amounts of different kinds of data from various sources Evolve data architecture and work on next-generation real-time pipeline algorithms and architecture in addition to supporting and maintaining current pipelines and legacy systems Write code and develop worker nodes for business logic, ETL and orchestration processes Develop algorithms for better attribution rules and category classifiers Work with stakeholders to identify opportunities for leveraging company data to drive search, discovery, and recommendations Collaborate with architects, engineers, data analysts, data scientists, contractors/consultants and project managers to assess requirements, design, develop and support data ingestions and API services Work with data scientists to build feature engineering pipelines and integrate ML models during content enrichment Influence priorities with various partners including engineers, PMO and leadership Mentor junior team members, define architecture, perform code reviews, and deliver work in sprint cycles Participate in design discussions for new systems and re-engineering components Share architectural leadership and forward-thinking ideas when required Take a holistic approach to building solutions and move from legacy systems to next-generation architecture Take ownership from requirements through production launch and post-launch support; participate in code reviews and on-call rotations Apply best practices and design patterns and stay current with data engineering tools and technologies Required Skills & Experience
BS or MS in Computer Science (or related field) with 10+ years of hands-on software development experience in large-scale data processing pipelines Expertise in Apache Spark, Scala and PySpark with 2-4 years of production-grade batch pipelines At least 4+ years of Java and API/Microservices experience At least 2+ years of Python experience 2+ years of experience with SQL and stored procedures for processing raw data, ETL, data validation; databases include SQL Server, Redis and other NoSQL DBs Knowledge of Big Data technologies, Hadoop, HDFS Experience building event-driven pipelines with Kafka, Java/Spark, Apache Flink Experience with AWS (EMR, EC2, S3) Experience building/consuming APIs for data ingestion and business logic Experience deploying, maintaining, and debugging production systems and infrastructure Experience building fault-tolerant and resilient systems Experience building worker nodes; knowledge of REST, data engineering design patterns Strong knowledge of Java, Spring Boot, Spark, Scala, PySpark, Python, orchestration tools, ESB, SQL, stored procedures, Docker, RESTful services, Kubernetes, CI/CD, observability, Kafka, release processes, caching strategies, versioning, Bitbucket/Git and AWS Cloud ecosystem; NoSQL databases, Hazelcast Strong software development and problem-solving skills; effective communication and collaboration Nice to Have
Exposure to Machine Learning (ML), LLM models, and using AI during coding Knowledge of Elastic APM, ELK stack and search technologies such as Elasticsearch/SOLR Experience with workflow orchestration tools such as Airflow or Apache NiFi Location and Perks
Market America offers competitive salary and generous benefits, including health, dental, vision, life, short and long-term disability insurance, a 401(k) retirement plan with company match, and an on-site health clinic. Qualified candidates should apply online. This position can work remotely from our Greensboro, NC or Monterey, CA offices. Sponsorship is not available for this position. Market America is proud to be an equal opportunity employer. About Market America & SHOP.COM
Market America Worldwide | SHOP.COM is a global e-commerce and digital marketing company that specializes in one-to-one marketing. Headquartered in Greensboro, NC, it was founded in 1992 by Founder, Chairman & CEO JR Ridinger. The company provides a robust business system for entrepreneurs and a better way for consumers to shop via SHOP.COM. For more information, visit MarketAmerica.com and SHOP.COM. #J-18808-Ljbffr
Join to apply for the
Staff Data Engineer
role at
SHOP.COM . Market America, a product brokerage and Internet marketing company that specializes in One-to-One Marketing, is seeking an experienced
Staff Data Engineer
for our IT team. As a senior member of the Data Engineering team, you will help millions of customers on our SHOP.COM and Market America Worldwide multi-country and multi-language global eCommerce websites find what they want across categories, merchant offers, products and taxonomy. We manage thousands of 3rd party affiliates/feeds through ETL and data ingestion pipelines before feeding our search systems. We operate multiple orchestration pipelines supporting data for products, store offers, analytics, customer behavioral profiles, segments, logs, and more. This is a highly visible role with the opportunity to make a significant impact. The data engineering team processes large amounts of data we import and collect, enrich content, integrate pricing and taxonomy, develop category classifier nodes and incorporate machine learning within the pipeline. Key Responsibilities
Must have minimum of 10-12 years of hands-on development experience implementing batch and event-driven applications using Java, Kafka, Spark, Scala, PySpark and Python Experience with Apache Kafka and Connectors, Java, Spring Boot in building event-driven services, Python in building ML pipelines Develop data pipelines responsible for ingesting large amounts of different kinds of data from various sources Evolve data architecture and work on next-generation real-time pipeline algorithms and architecture in addition to supporting and maintaining current pipelines and legacy systems Write code and develop worker nodes for business logic, ETL and orchestration processes Develop algorithms for better attribution rules and category classifiers Work with stakeholders to identify opportunities for leveraging company data to drive search, discovery, and recommendations Collaborate with architects, engineers, data analysts, data scientists, contractors/consultants and project managers to assess requirements, design, develop and support data ingestions and API services Work with data scientists to build feature engineering pipelines and integrate ML models during content enrichment Influence priorities with various partners including engineers, PMO and leadership Mentor junior team members, define architecture, perform code reviews, and deliver work in sprint cycles Participate in design discussions for new systems and re-engineering components Share architectural leadership and forward-thinking ideas when required Take a holistic approach to building solutions and move from legacy systems to next-generation architecture Take ownership from requirements through production launch and post-launch support; participate in code reviews and on-call rotations Apply best practices and design patterns and stay current with data engineering tools and technologies Required Skills & Experience
BS or MS in Computer Science (or related field) with 10+ years of hands-on software development experience in large-scale data processing pipelines Expertise in Apache Spark, Scala and PySpark with 2-4 years of production-grade batch pipelines At least 4+ years of Java and API/Microservices experience At least 2+ years of Python experience 2+ years of experience with SQL and stored procedures for processing raw data, ETL, data validation; databases include SQL Server, Redis and other NoSQL DBs Knowledge of Big Data technologies, Hadoop, HDFS Experience building event-driven pipelines with Kafka, Java/Spark, Apache Flink Experience with AWS (EMR, EC2, S3) Experience building/consuming APIs for data ingestion and business logic Experience deploying, maintaining, and debugging production systems and infrastructure Experience building fault-tolerant and resilient systems Experience building worker nodes; knowledge of REST, data engineering design patterns Strong knowledge of Java, Spring Boot, Spark, Scala, PySpark, Python, orchestration tools, ESB, SQL, stored procedures, Docker, RESTful services, Kubernetes, CI/CD, observability, Kafka, release processes, caching strategies, versioning, Bitbucket/Git and AWS Cloud ecosystem; NoSQL databases, Hazelcast Strong software development and problem-solving skills; effective communication and collaboration Nice to Have
Exposure to Machine Learning (ML), LLM models, and using AI during coding Knowledge of Elastic APM, ELK stack and search technologies such as Elasticsearch/SOLR Experience with workflow orchestration tools such as Airflow or Apache NiFi Location and Perks
Market America offers competitive salary and generous benefits, including health, dental, vision, life, short and long-term disability insurance, a 401(k) retirement plan with company match, and an on-site health clinic. Qualified candidates should apply online. This position can work remotely from our Greensboro, NC or Monterey, CA offices. Sponsorship is not available for this position. Market America is proud to be an equal opportunity employer. About Market America & SHOP.COM
Market America Worldwide | SHOP.COM is a global e-commerce and digital marketing company that specializes in one-to-one marketing. Headquartered in Greensboro, NC, it was founded in 1992 by Founder, Chairman & CEO JR Ridinger. The company provides a robust business system for entrepreneurs and a better way for consumers to shop via SHOP.COM. For more information, visit MarketAmerica.com and SHOP.COM. #J-18808-Ljbffr