NVIDIA
Senior Software Engineer - Distributed Inference
NVIDIA, Oklahoma City, Oklahoma, United States
Senior Software Engineer - Distributed Inference
Join to apply for the
Senior Software Engineer - Distributed Inference
role at
NVIDIA
What You’ll Be Doing
Build and maintain distributed model management systems, including Rust-based runtime components, for large-scale AI inference workloads.
Implement inference scheduling and deployment solutions on Kubernetes and Slurm, while driving advances in scaling, orchestration, and resource management.
Collaborate with infrastructure engineers and researchers to develop scalable APIs, services, and end-to-end inference workflows.
Create monitoring, benchmarking, automation, and documentation processes to ensure low-latency, robust, and production-ready inference systems on GPU clusters.
What We Need To See
Bachelor’s, Master’s, or PhD in Computer Science, ECE, or related field (or equivalent experience).
6+ years of professional systems software development experience.
Strong programming expertise in Rust (with C++, Python as a plus).
Deep knowledge of distributed systems, runtime orchestration, and cluster-scale services.
Hands-on experience with Kubernetes, container-based microservices, and integration with Slurm.
Proven ability to excel in fast-paced R&D environments and collaborate across functions.
Ways To Stand Out From The Crowd
Experience with inference-serving frameworks (e.g., Dynamo Inference Server, TensorRT, ONNX Runtime) and deploying/managing LLM inference pipelines at scale.
Contributions to large-scale, low-latency distributed systems (open-source preferred) with proven expertise in high-availability infrastructure.
Strong background in GPU inference performance tuning, CUDA-based systems, and operating across cloud-native and hybrid environments (AWS, GCP, Azure).
NVIDIA has continuously reinvented itself over three decades. Our invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. We are widely considered to be the leader of AI computing, and one of the technology world’s most desirable employers. If you're creative and autonomous, we want to hear from you!
Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.
You will also be eligible for equity and benefits.
Applications for this job will be accepted at least until August 30, 2025.
NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.
JR2002881
Seniority level
Mid-Senior level
Employment type
Full-time
Job function
Software Development
Industries
Computer Hardware Manufacturing, Software Development, and Computers and Electronics Manufacturing
#J-18808-Ljbffr
Senior Software Engineer - Distributed Inference
role at
NVIDIA
What You’ll Be Doing
Build and maintain distributed model management systems, including Rust-based runtime components, for large-scale AI inference workloads.
Implement inference scheduling and deployment solutions on Kubernetes and Slurm, while driving advances in scaling, orchestration, and resource management.
Collaborate with infrastructure engineers and researchers to develop scalable APIs, services, and end-to-end inference workflows.
Create monitoring, benchmarking, automation, and documentation processes to ensure low-latency, robust, and production-ready inference systems on GPU clusters.
What We Need To See
Bachelor’s, Master’s, or PhD in Computer Science, ECE, or related field (or equivalent experience).
6+ years of professional systems software development experience.
Strong programming expertise in Rust (with C++, Python as a plus).
Deep knowledge of distributed systems, runtime orchestration, and cluster-scale services.
Hands-on experience with Kubernetes, container-based microservices, and integration with Slurm.
Proven ability to excel in fast-paced R&D environments and collaborate across functions.
Ways To Stand Out From The Crowd
Experience with inference-serving frameworks (e.g., Dynamo Inference Server, TensorRT, ONNX Runtime) and deploying/managing LLM inference pipelines at scale.
Contributions to large-scale, low-latency distributed systems (open-source preferred) with proven expertise in high-availability infrastructure.
Strong background in GPU inference performance tuning, CUDA-based systems, and operating across cloud-native and hybrid environments (AWS, GCP, Azure).
NVIDIA has continuously reinvented itself over three decades. Our invention of the GPU in 1999 sparked the growth of the PC gaming market, redefined modern computer graphics, and revolutionized parallel computing. We are widely considered to be the leader of AI computing, and one of the technology world’s most desirable employers. If you're creative and autonomous, we want to hear from you!
Your base salary will be determined based on your location, experience, and the pay of employees in similar positions. The base salary range is 184,000 USD - 287,500 USD for Level 4, and 224,000 USD - 356,500 USD for Level 5.
You will also be eligible for equity and benefits.
Applications for this job will be accepted at least until August 30, 2025.
NVIDIA is committed to fostering a diverse work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.
JR2002881
Seniority level
Mid-Senior level
Employment type
Full-time
Job function
Software Development
Industries
Computer Hardware Manufacturing, Software Development, and Computers and Electronics Manufacturing
#J-18808-Ljbffr