Scale AI
Tech Lead Manager, Machine Learning Research Scientist- LLM Evals
Scale AI, San Francisco, California, United States, 94199
Overview
Tech Lead Manager, Machine Learning Research Scientist- LLM Evals at Scale AI leads a team of research scientists and research engineers to develop and implement novel evaluation methodologies, metrics, and benchmarks for large language models. The role designs and executes a roadmap that defines best practices in data-driven AI development and accelerates the next generation of generative AI models in partnership with top foundational model labs. Responsibilities
Lead a team of research scientists and research engineers on LLM evals. Conduct research on the effectiveness and limitations of existing LLM evaluation techniques. Design and develop novel evaluation benchmarks for large language models, covering instruction following, factuality, robustness, and fairness. Communicate and collaborate with clients and peer teams to facilitate cross-functional projects. Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols. Implement scalable and reproducible evaluation pipelines using modern ML frameworks. Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives. Stay up-to-date on research developments, participate in design decisions, and contribute to the research community. Thrive in a high-energy, fast-paced startup environment and dedicate time and effort to drive impactful results. Qualifications
5+ years of hands-on experience in large language models, NLP, and Transformer modeling, in both research and engineering settings. Track record of landing major research impacts in fast-paced environments. Experience leading a team of research scientists and research engineers. Excellent written and verbal communication skills. Published research in machine learning at major conferences (e.g., NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR) and/or journals. Experience in customer-facing roles. Compensation and Benefits
Compensation packages include base salary, equity, and benefits. The stated salary range for this full-time role in San Francisco, New York, and Seattle is $260,000 – $350,000 USD. Scale employees may also receive equity-based compensation. Benefits include comprehensive health, dental, and vision coverage, retirement benefits, a learning and development stipend, and generous PTO. Some roles may include additional benefits such as a commuter stipend. About Scale and EEO
Scale AI is an equal opportunity employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability status, gender identity, or veteran status.
#J-18808-Ljbffr
Tech Lead Manager, Machine Learning Research Scientist- LLM Evals at Scale AI leads a team of research scientists and research engineers to develop and implement novel evaluation methodologies, metrics, and benchmarks for large language models. The role designs and executes a roadmap that defines best practices in data-driven AI development and accelerates the next generation of generative AI models in partnership with top foundational model labs. Responsibilities
Lead a team of research scientists and research engineers on LLM evals. Conduct research on the effectiveness and limitations of existing LLM evaluation techniques. Design and develop novel evaluation benchmarks for large language models, covering instruction following, factuality, robustness, and fairness. Communicate and collaborate with clients and peer teams to facilitate cross-functional projects. Collaborate with internal teams and external partners to refine metrics and create standardized evaluation protocols. Implement scalable and reproducible evaluation pipelines using modern ML frameworks. Publish research findings in top-tier AI conferences and contribute to open-source benchmarking initiatives. Stay up-to-date on research developments, participate in design decisions, and contribute to the research community. Thrive in a high-energy, fast-paced startup environment and dedicate time and effort to drive impactful results. Qualifications
5+ years of hands-on experience in large language models, NLP, and Transformer modeling, in both research and engineering settings. Track record of landing major research impacts in fast-paced environments. Experience leading a team of research scientists and research engineers. Excellent written and verbal communication skills. Published research in machine learning at major conferences (e.g., NeurIPS, ICML, ICLR, ACL, EMNLP, CVPR) and/or journals. Experience in customer-facing roles. Compensation and Benefits
Compensation packages include base salary, equity, and benefits. The stated salary range for this full-time role in San Francisco, New York, and Seattle is $260,000 – $350,000 USD. Scale employees may also receive equity-based compensation. Benefits include comprehensive health, dental, and vision coverage, retirement benefits, a learning and development stipend, and generous PTO. Some roles may include additional benefits such as a commuter stipend. About Scale and EEO
Scale AI is an equal opportunity employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability status, gender identity, or veteran status.
#J-18808-Ljbffr