Get new jobs for this search by email

AI Infrastructure Engineer, Model Serving Platform, Mid-Level

Join to apply for the AI Infrastructure Engineer, Model Serving Platform, Mid-Level role at Jobright.ai

AI Infrastructure Engineer, Model Serving Platform, Mid-Level

2 days ago Be among the first 25 applicants

Join to apply for the AI Infrastructure Engineer, Model Serving Platform, Mid-Level role at Jobright.ai

Scale AI is a company transforming how organizations build and deploy AI, focusing on scalable and efficient serving of large language models (LLMs). The AI Infrastructure Engineer will design and build platforms for LLMs, ensuring high-performance and fault-tolerant systems while collaborating with researchers and engineers to optimize models for production and research use cases.

Responsibilities:

• Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale.

• Build an internal platform to empower LLM capability discovery.

• Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.

• Conduct architecture and design reviews to uphold best practices in system design and scalability.

• Develop monitoring and observability solutions to ensure system health and performance.

• Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment.

Qualifications:

Required:

• 4+ years of experience building large-scale, high-performance backend systems.

• Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++).

• Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.)

• Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc.

• Experience with containers and orchestration tools (e.g., Docker, Kubernetes).

• Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform).

• Proven ability to solve complex problems and work independently in fast-moving environments.

Preferred:

• Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference.

Company:

Scale AI provides a data-oriented platform that assists in the development of AI applications. Founded in 2016, the company is headquartered in San Francisco, California, USA, with a team of 501-1000 employees. The company is currently Late Stage. Scale AI has a track record of offering H1B sponsorships.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Industries

  Software Development

Referrals increase your chances of interviewing at Jobright.ai by 2x

Inferred from the description for this job

Medical insurance

Vision insurance

401(k)

Get notified about new Infrastructure Engineer jobs in San Francisco Bay Area .

San Francisco, CA $180,000.00-$250,000.00 9 hours ago

San Francisco, CA $150,000.00-$250,000.00 1 year ago

San Francisco, CA $150.00-$175.00 5 months ago

Fall 2025 Onboard Infrastructure Engineer

San Francisco, CA $130,000.00-$250,000.00 5 days ago

San Francisco, CA $140,000.00-$200,000.00 1 hour ago

Palo Alto, CA $180,000.00-$370,000.00 2 weeks ago

San Francisco, CA $150,000.00-$250,000.00 6 months ago

San Francisco, CA $135,000.00-$185,000.00 2 weeks ago

San Francisco, CA $100,949.33-$137,002.66 1 day ago

San Jose, CA $113,600.00-$170,400.00 2 days ago

San Jose, CA $125,000.00-$155,000.00 9 hours ago

Palo Alto, CA $144,000.00-$174,000.00 2 weeks ago

San Jose, CA $130,000.00-$182,000.00 5 months ago

Sunnyvale, CA $168,000.00-$276,000.00 1 week ago

Hayward, CA $100,000.00-$150,000.00 6 months ago

San Francisco, CA $110,000.00-$170,000.00 2 months ago

San Leandro, CA $135,000.00-$150,000.00 3 weeks ago

San Mateo, CA $157,000.00-$171,500.00 1 month ago

Fremont, CA $70,000.00-$100,000.00 3 weeks ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.