Director of Software Engineering (Node.js & Web Scraping Expert)

We are seeking a

Director of Software Engineering

with deep expertise in

Node.js development

and

large-scale web scraping . This role will lead the

engineering team , designing and optimizing

high-performance, distributed web scraping systems . The ideal candidate has

extensive experience in handling anti-bot measures, data pipeline optimization, and scalable cloud-based architectures . Key Responsibilities - Software Engineering & Web Scraping Leadership:

Architect, develop, and maintain

scalable and distributed web scraping systems

using

Node.js . Design and implement

data extraction pipelines

to process large volumes of structured and unstructured data. Develop solutions to

bypass anti-bot mechanisms , including

CAPTCHA handling, session management, fingerprinting, and IP rotation . Optimize scraping processes for

performance, reliability, and efficiency

while managing

proxy services

(residential, datacenter, rotating). Oversee

data storage and processing strategies , ensuring high availability and consistency. Collaborate with

Product, DevOps, and Data Science teams

to integrate extracted data into analytics and business applications. Implement

best practices for microservices, API integrations, and real-time data streaming . Key Responsibilities - Scalability, Security & DevOps:

Lead the transition to

cloud-native, containerized, and serverless architectures

for web scraping. Ensure compliance with

legal and ethical standards

(robots.txt, GDPR, CCPA, etc.). Optimize

cloud resources

(AWS, GCP, or Azure) to support high-throughput scraping. Manage

real-time monitoring and alerting

systems to detect

scraping failures, IP bans, or performance bottlenecks . Work closely with

DevOps teams

to optimize

CI/CD pipelines, automated deployments, and system scalability . Key Responsibilities - Engineering Team Management & Strategy:

Lead, mentor, and grow a

high-performance engineering team . Define and execute the

technology roadmap , aligning with business objectives. Foster a culture of

continuous learning, collaboration, and innovation . Implement

agile development methodologies (Scrum, Kanban)

to optimize project execution. Ensure

code quality, security, and best practices

across all engineering efforts. Qualifications & Experience - Technical Expertise:

10+ years

of experience in

software engineering , with at least

5+ years

in

web scraping and large-scale data extraction . Strong hands-on expertise in

Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation . Extensive experience in

handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques . Deep knowledge of

proxy management (residential, datacenter, rotating, and VPNs) . Experience with

NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.) . Familiarity with

data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.) . Strong experience with

CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure) . Qualifications & Experience - Leadership & Soft Skills:

Proven track record of

scaling engineering teams and leading complex projects . Strong

problem-solving and debugging

skills, especially for

scraping challenges and performance bottlenecks . Excellent

communication and stakeholder management

skills. Passion for

mentorship, team development, and continuous learning . Preferred Qualifications:

Experience with

machine learning for data extraction and NLP . Knowledge of

browser fingerprinting and bot detection mechanisms . Familiarity with

enterprise-scale web crawling frameworks

(Scrapy, Colly, Apify, etc.). Prior leadership experience in

data-driven businesses or web scraping startups .

#J-18808-Ljbffr