CloudBees
Data Scientist AI & Agentic Applications & Benchmarking
CloudBees, Los Angeles, California, United States, 90079
Description
About CloudBees
CloudBees is the leading software delivery platform for enterprise DevOps teams. As a high-growth startup, we empower developers to build, deploy, and manage software more efficiently. Now, were bringing
agentic intelligence
into our platform to supercharge developer workflowsand we need a data scientist who can both drive insights and tell the story behind the metrics. The Role
CloudBees is seeking a startup-savvy
Data Scientist
to help define, measure, and evangelize the impact of
Agentic Applications
across our platform. Youll work closely with engineers and product teams to
prototype and measure AI and Agentic experiences, using evals, telemetry, and AI benchmarks
to help the company drive the conversation in the market and with customers. Translating performance into clear, compelling narratives to our customers and internal teams. As a founding member of the team, you will lead the charge as equal parts
builder, evaluator, and communicator with the technical depth to prototype in Python notebooks, Claude Code, and other tools to drive clarity to write about what matters. Key Responsibilities
Partner with our platform team to develop and prototype
telemetry, eval frameworks, and benchmarks
for emerging agentic systems. Partner with product and engineering teams to
measure AI outcomes and usage across customers and teams . Help define KPIs and success metrics for AI and LLM-driven features and workflows. Use Python notebooks to
explore data, visualize insights , and test hypotheses rapidly and share insights. Tell the story behind the numbers: Write internal documentation, performance summaries, and thought leadership around outcomes. Enable engineering teams to
instrument, log, and evaluate
agent performance effectively. Stay up to date with evolving metrics and evaluation techniques in the LLM and agentic AI ecosystem. Required Qualifications
3+ years of experience in data science or ML analytics roles, ideally in startup or high-growth environments. Proficiency in
Python , including building and sharing analysis via
Jupyter notebooks . Experience working with evals, telemetry, A/B testing, and evaluating user-facing ML systems. Experience with AI/ML tools such as MLFlow, Hugging Face, or other Model / LLM tools. Ability to partner with technical teams to define meaningful metrics and benchmarks. Clear communication skillscapable of
writing about outcomes , sharing learnings, and influencing stakeholders. Comfort working in fast-paced, ambiguous environments where speed and clarity matter. Preferred Qualifications
Experience with
agentic or LLM-based applications
(e.g., evaluating AI copilots, autonomous workflows). Familiarity with tools like LangSmith, OpenInference, or custom eval stacks. Background in developer tools, DevOps, or platform engineering environments. Why Join CloudBees
Shape the future of AI-driven DevOps with real user impact. Join a nimble, passionate team at the forefront of agentic system development. Work in a flexible, remote-first culture built on trust and innovation. Competitive salary, startup equity, and excellent benefits. CloudBees is proud to be an Equal Opportunity Employer.
We value diverse voices, ideas, and experiences as essential to building great products. #J-18808-Ljbffr
agentic intelligence
into our platform to supercharge developer workflowsand we need a data scientist who can both drive insights and tell the story behind the metrics. The Role
CloudBees is seeking a startup-savvy
Data Scientist
to help define, measure, and evangelize the impact of
Agentic Applications
across our platform. Youll work closely with engineers and product teams to
prototype and measure AI and Agentic experiences, using evals, telemetry, and AI benchmarks
to help the company drive the conversation in the market and with customers. Translating performance into clear, compelling narratives to our customers and internal teams. As a founding member of the team, you will lead the charge as equal parts
builder, evaluator, and communicator with the technical depth to prototype in Python notebooks, Claude Code, and other tools to drive clarity to write about what matters. Key Responsibilities
Partner with our platform team to develop and prototype
telemetry, eval frameworks, and benchmarks
for emerging agentic systems. Partner with product and engineering teams to
measure AI outcomes and usage across customers and teams . Help define KPIs and success metrics for AI and LLM-driven features and workflows. Use Python notebooks to
explore data, visualize insights , and test hypotheses rapidly and share insights. Tell the story behind the numbers: Write internal documentation, performance summaries, and thought leadership around outcomes. Enable engineering teams to
instrument, log, and evaluate
agent performance effectively. Stay up to date with evolving metrics and evaluation techniques in the LLM and agentic AI ecosystem. Required Qualifications
3+ years of experience in data science or ML analytics roles, ideally in startup or high-growth environments. Proficiency in
Python , including building and sharing analysis via
Jupyter notebooks . Experience working with evals, telemetry, A/B testing, and evaluating user-facing ML systems. Experience with AI/ML tools such as MLFlow, Hugging Face, or other Model / LLM tools. Ability to partner with technical teams to define meaningful metrics and benchmarks. Clear communication skillscapable of
writing about outcomes , sharing learnings, and influencing stakeholders. Comfort working in fast-paced, ambiguous environments where speed and clarity matter. Preferred Qualifications
Experience with
agentic or LLM-based applications
(e.g., evaluating AI copilots, autonomous workflows). Familiarity with tools like LangSmith, OpenInference, or custom eval stacks. Background in developer tools, DevOps, or platform engineering environments. Why Join CloudBees
Shape the future of AI-driven DevOps with real user impact. Join a nimble, passionate team at the forefront of agentic system development. Work in a flexible, remote-first culture built on trust and innovation. Competitive salary, startup equity, and excellent benefits. CloudBees is proud to be an Equal Opportunity Employer.
We value diverse voices, ideas, and experiences as essential to building great products. #J-18808-Ljbffr