jobright.com
AI Infrastructure Engineer, Model Serving Platform, Mid-Level
jobright.com - San Francisco, California, United States, 94199
Work at jobright.com
Overview
- View job
Overview
AI Infrastructure Engineer, Model Serving Platform, Mid-Level
Join to apply for the
AI Infrastructure Engineer, Model Serving Platform, Mid-Level
role at
Jobright.ai AI Infrastructure Engineer, Model Serving Platform, Mid-Level
2 days ago Be among the first 25 applicants Join to apply for the
AI Infrastructure Engineer, Model Serving Platform, Mid-Level
role at
Jobright.ai Scale AI is a company transforming how organizations build and deploy AI, focusing on scalable and efficient serving of large language models (LLMs). The AI Infrastructure Engineer will design and build platforms for LLMs, ensuring high-performance and fault-tolerant systems while collaborating with researchers and engineers to optimize models for production and research use cases. Responsibilities: Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale. Build an internal platform to empower LLM capability discovery. Collaborate with researchers and engineers to integrate and optimize models for production and research use cases. Conduct architecture and design reviews to uphold best practices in system design and scalability. Develop monitoring and observability solutions to ensure system health and performance. Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment. Qualifications: Required: 4+ years of experience building large-scale, high-performance backend systems. Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++). Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.) Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc. Experience with containers and orchestration tools (e.g., Docker, Kubernetes). Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform). Proven ability to solve complex problems and work independently in fast-moving environments. Preferred: Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference. Company: Scale AI provides a data-oriented platform that assists in the development of AI applications. Founded in 2016, the company is headquartered in San Francisco, California, USA, with a team of 501-1000 employees. The company is currently Late Stage. Scale AI has a track record of offering H1B sponsorships. Seniority level
Seniority level
Mid-Senior level Employment type
Employment type
Full-time Job function
Industries
Software Development Referrals increase your chances of interviewing at Jobright.ai by 2x Inferred from the description for this job
Medical insurance Vision insurance 401(k) Get notified about new Infrastructure Engineer jobs in
San Francisco Bay Area . San Francisco, CA $180,000.00-$250,000.00 9 hours ago San Francisco, CA $150,000.00-$250,000.00 1 year ago San Francisco, CA $150.00-$175.00 5 months ago Fall 2025 Onboard Infrastructure Engineer
San Francisco, CA $130,000.00-$250,000.00 5 days ago San Francisco, CA $140,000.00-$200,000.00 1 hour ago Palo Alto, CA $180,000.00-$370,000.00 2 weeks ago San Francisco, CA $150,000.00-$250,000.00 6 months ago San Francisco, CA $135,000.00-$185,000.00 2 weeks ago San Francisco, CA $100,949.33-$137,002.66 1 day ago San Jose, CA $113,600.00-$170,400.00 2 days ago San Jose, CA $125,000.00-$155,000.00 9 hours ago Palo Alto, CA $144,000.00-$174,000.00 2 weeks ago San Jose, CA $130,000.00-$182,000.00 5 months ago Sunnyvale, CA $168,000.00-$276,000.00 1 week ago Hayward, CA $100,000.00-$150,000.00 6 months ago San Francisco, CA $110,000.00-$170,000.00 2 months ago San Leandro, CA $135,000.00-$150,000.00 3 weeks ago San Mateo, CA $157,000.00-$171,500.00 1 month ago Fremont, CA $70,000.00-$100,000.00 3 weeks ago Were unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr
Join to apply for the
AI Infrastructure Engineer, Model Serving Platform, Mid-Level
role at
Jobright.ai AI Infrastructure Engineer, Model Serving Platform, Mid-Level
2 days ago Be among the first 25 applicants Join to apply for the
AI Infrastructure Engineer, Model Serving Platform, Mid-Level
role at
Jobright.ai Scale AI is a company transforming how organizations build and deploy AI, focusing on scalable and efficient serving of large language models (LLMs). The AI Infrastructure Engineer will design and build platforms for LLMs, ensuring high-performance and fault-tolerant systems while collaborating with researchers and engineers to optimize models for production and research use cases. Responsibilities: Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale. Build an internal platform to empower LLM capability discovery. Collaborate with researchers and engineers to integrate and optimize models for production and research use cases. Conduct architecture and design reviews to uphold best practices in system design and scalability. Develop monitoring and observability solutions to ensure system health and performance. Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment. Qualifications: Required: 4+ years of experience building large-scale, high-performance backend systems. Strong programming skills in one or more languages (e.g., Python, Go, Rust, C++). Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.) Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc. Experience with containers and orchestration tools (e.g., Docker, Kubernetes). Familiarity with cloud infrastructure (AWS, GCP) and infrastructure as code (e.g., Terraform). Proven ability to solve complex problems and work independently in fast-moving environments. Preferred: Experience with modern LLM serving frameworks such as vLLM, SGLang, TensorRT-LLM, or text-generation-inference. Company: Scale AI provides a data-oriented platform that assists in the development of AI applications. Founded in 2016, the company is headquartered in San Francisco, California, USA, with a team of 501-1000 employees. The company is currently Late Stage. Scale AI has a track record of offering H1B sponsorships. Seniority level
Seniority level
Mid-Senior level Employment type
Employment type
Full-time Job function
Industries
Software Development Referrals increase your chances of interviewing at Jobright.ai by 2x Inferred from the description for this job
Medical insurance Vision insurance 401(k) Get notified about new Infrastructure Engineer jobs in
San Francisco Bay Area . San Francisco, CA $180,000.00-$250,000.00 9 hours ago San Francisco, CA $150,000.00-$250,000.00 1 year ago San Francisco, CA $150.00-$175.00 5 months ago Fall 2025 Onboard Infrastructure Engineer
San Francisco, CA $130,000.00-$250,000.00 5 days ago San Francisco, CA $140,000.00-$200,000.00 1 hour ago Palo Alto, CA $180,000.00-$370,000.00 2 weeks ago San Francisco, CA $150,000.00-$250,000.00 6 months ago San Francisco, CA $135,000.00-$185,000.00 2 weeks ago San Francisco, CA $100,949.33-$137,002.66 1 day ago San Jose, CA $113,600.00-$170,400.00 2 days ago San Jose, CA $125,000.00-$155,000.00 9 hours ago Palo Alto, CA $144,000.00-$174,000.00 2 weeks ago San Jose, CA $130,000.00-$182,000.00 5 months ago Sunnyvale, CA $168,000.00-$276,000.00 1 week ago Hayward, CA $100,000.00-$150,000.00 6 months ago San Francisco, CA $110,000.00-$170,000.00 2 months ago San Leandro, CA $135,000.00-$150,000.00 3 weeks ago San Mateo, CA $157,000.00-$171,500.00 1 month ago Fremont, CA $70,000.00-$100,000.00 3 weeks ago Were unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr