Jobright.ai
Junior AI Data Scientist, Evaluation & Insights
Jobright.ai, San Francisco, California, United States, 94199
Junior AI Data Scientist, Evaluation & Insights
Join to apply for the
Junior AI Data Scientist, Evaluation & Insights
role at
Jobright.ai Junior AI Data Scientist, Evaluation & Insights
1 day ago Be among the first 25 applicants Join to apply for the
Junior AI Data Scientist, Evaluation & Insights
role at
Jobright.ai Get AI-powered advice on this job and more exclusive features. Ironclad is the leading AI-powered contract lifecycle management platform, processing billions of contracts every year. The AI Evaluation Engineer role focuses on unlocking insights from training data and ensuring continuous improvement of ML systems through data-driven evaluation and iteration. Responsibilities: • Analyze training and evaluation datasets to identify distributional gaps, labeling inconsistencies, and long-tail opportunities. • Design and execute labeling campaigns, including development of golden datasets and annotation guidelines. • Build and maintain dashboards that track model accuracy, regression trends, and product-specific KPIs like success rate or answer helpfulness. • Investigate failure modes via prompt clustering, error taxonomy development, and user intent classification. • Operationalize feedback loops: mine product telemetry and human-in-the-loop reviews for signal, then translate into data-driven model improvement strategies. • Partner with engineers and PMs to run structured A/B tests and human evaluations for new models or features. • Support the development of scalable data and evaluation infrastructure for LLMs and agents. • Work with product, engineering and legal to create clear & transparent processes for the handling of customer data in AI training, fine-tuning and evaluation. Qualifications: Required: • Bachelor's or Master's degree in a quantitative field (e.g., Statistics, Computer Science, Data Science, Applied Math). • 1–3 years of experience in applied ML or data science, preferably in NLP or LLM-based applications. • Strong SQL and Python skills; experience with Jupyter, Pandas, and experiment tracking tools. • Comfortable navigating ambiguity, slicing large datasets, and communicating insights clearly to cross-functional stakeholders. Preferred: • Experience with prompt analysis, clustering, or user behavior modeling is a plus. • Bonus: familiarity with LLM eval techniques, Reinforcement Learning from Human Feedback (RLHF), or agentic system design. • Experience with program management. Company: Ironclad is a contract lifecycle management platform used by companies to handle every type of contract workflow. Founded in 2014, the company is headquartered in San Francisco, California, USA, with a team of 501-1000 employees. The company is currently Late Stage. Ironclad has a track record of offering H1B sponsorships. Seniority level
Seniority level Entry level Employment type
Employment type Full-time Job function
Industries Software Development Referrals increase your chances of interviewing at Jobright.ai by 2x Inferred from the description for this job
Medical insurance Vision insurance 401(k) Get notified about new Data Scientist jobs in
San Francisco, CA . San Francisco, CA $140,000.00-$235,000.00 6 days ago Oakland, CA $10,279.00-$11,900.00 1 week ago San Francisco, CA $122,000.00-$167,000.00 1 day ago San Francisco, CA $121,900.00-$161,600.00 14 hours ago Machine Learning Engineer (I, II, or Sr.)
San Francisco, CA $140,670.00-$195,400.00 20 hours ago San Francisco, CA $190,000.00-$230,000.00 2 weeks ago San Francisco, CA $144,000.00-$192,000.00 3 weeks ago South San Francisco, CA $54.00-$97.00 2 weeks ago San Mateo, CA $120,000.00-$135,000.00 2 weeks ago San Bruno, CA $119,000.00-$169,000.00 2 months ago San Francisco, CA $115,000.00-$185,000.00 3 weeks ago San Francisco, CA $233,000.00-$282,000.00 2 weeks ago Data Scientist II – Experimentation & Measurement
San Mateo, CA $153,300.00-$229,900.00 2 weeks ago San Francisco, CA $85,000.00-$120,000.00 4 hours ago San Francisco, CA $149,998.00-$250,000.00 2 days ago San Francisco, CA $90,000.00-$150,000.00 1 month ago San Francisco, CA $100,000.00-$200,000.00 7 months ago San Francisco, CA $90,000.00-$150,000.00 1 month ago San Francisco, CA $180,000.00-$240,000.00 2 weeks ago We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr
Join to apply for the
Junior AI Data Scientist, Evaluation & Insights
role at
Jobright.ai Junior AI Data Scientist, Evaluation & Insights
1 day ago Be among the first 25 applicants Join to apply for the
Junior AI Data Scientist, Evaluation & Insights
role at
Jobright.ai Get AI-powered advice on this job and more exclusive features. Ironclad is the leading AI-powered contract lifecycle management platform, processing billions of contracts every year. The AI Evaluation Engineer role focuses on unlocking insights from training data and ensuring continuous improvement of ML systems through data-driven evaluation and iteration. Responsibilities: • Analyze training and evaluation datasets to identify distributional gaps, labeling inconsistencies, and long-tail opportunities. • Design and execute labeling campaigns, including development of golden datasets and annotation guidelines. • Build and maintain dashboards that track model accuracy, regression trends, and product-specific KPIs like success rate or answer helpfulness. • Investigate failure modes via prompt clustering, error taxonomy development, and user intent classification. • Operationalize feedback loops: mine product telemetry and human-in-the-loop reviews for signal, then translate into data-driven model improvement strategies. • Partner with engineers and PMs to run structured A/B tests and human evaluations for new models or features. • Support the development of scalable data and evaluation infrastructure for LLMs and agents. • Work with product, engineering and legal to create clear & transparent processes for the handling of customer data in AI training, fine-tuning and evaluation. Qualifications: Required: • Bachelor's or Master's degree in a quantitative field (e.g., Statistics, Computer Science, Data Science, Applied Math). • 1–3 years of experience in applied ML or data science, preferably in NLP or LLM-based applications. • Strong SQL and Python skills; experience with Jupyter, Pandas, and experiment tracking tools. • Comfortable navigating ambiguity, slicing large datasets, and communicating insights clearly to cross-functional stakeholders. Preferred: • Experience with prompt analysis, clustering, or user behavior modeling is a plus. • Bonus: familiarity with LLM eval techniques, Reinforcement Learning from Human Feedback (RLHF), or agentic system design. • Experience with program management. Company: Ironclad is a contract lifecycle management platform used by companies to handle every type of contract workflow. Founded in 2014, the company is headquartered in San Francisco, California, USA, with a team of 501-1000 employees. The company is currently Late Stage. Ironclad has a track record of offering H1B sponsorships. Seniority level
Seniority level Entry level Employment type
Employment type Full-time Job function
Industries Software Development Referrals increase your chances of interviewing at Jobright.ai by 2x Inferred from the description for this job
Medical insurance Vision insurance 401(k) Get notified about new Data Scientist jobs in
San Francisco, CA . San Francisco, CA $140,000.00-$235,000.00 6 days ago Oakland, CA $10,279.00-$11,900.00 1 week ago San Francisco, CA $122,000.00-$167,000.00 1 day ago San Francisco, CA $121,900.00-$161,600.00 14 hours ago Machine Learning Engineer (I, II, or Sr.)
San Francisco, CA $140,670.00-$195,400.00 20 hours ago San Francisco, CA $190,000.00-$230,000.00 2 weeks ago San Francisco, CA $144,000.00-$192,000.00 3 weeks ago South San Francisco, CA $54.00-$97.00 2 weeks ago San Mateo, CA $120,000.00-$135,000.00 2 weeks ago San Bruno, CA $119,000.00-$169,000.00 2 months ago San Francisco, CA $115,000.00-$185,000.00 3 weeks ago San Francisco, CA $233,000.00-$282,000.00 2 weeks ago Data Scientist II – Experimentation & Measurement
San Mateo, CA $153,300.00-$229,900.00 2 weeks ago San Francisco, CA $85,000.00-$120,000.00 4 hours ago San Francisco, CA $149,998.00-$250,000.00 2 days ago San Francisco, CA $90,000.00-$150,000.00 1 month ago San Francisco, CA $100,000.00-$200,000.00 7 months ago San Francisco, CA $90,000.00-$150,000.00 1 month ago San Francisco, CA $180,000.00-$240,000.00 2 weeks ago We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.
#J-18808-Ljbffr