STAND 8 Technology Consulting
AI/ML Prompt Scientist/Engineer
STAND 8 Technology Consulting, San Francisco, California, United States, 94199
Overview
STAND 8 Technology Consulting is seeking an AI/ML Prompt Scientist/Engineer to design, evaluate, and optimize prompt-based systems that drive real-world impact across cutting-edge AI applications. In this role, you'll combine expertise in prompt engineering, orchestration frameworks, retrieval pipelines, and evaluation tooling to build scalable, reliable, and safe AI systems. This position offers a base pay range and opportunities to work with enterprise partners across the United States and internationally. Base pay range $100,000.00/yr - $200,000.00/yr About the company STAND 8 provides end-to-end IT solutions to enterprise partners across the United States and globally with offices in Los Angeles, New York, New Jersey, Atlanta, Mexico, Japan, India, and more. STAND 8 focuses on the bleeding edge of technology and leverages automation, process, marketing, and over fifteen years of success and growth to provide a world-class experience for our customers, partners, and employees. Our mission is to impact the world positively by creating success through PEOPLE, PROCESS, and TECHNOLOGY. Responsibilities
Design, optimize, and evaluate prompts and orchestration pipelines to enhance LLM performance and scalability. Implement programmatic prompting using frameworks such as LangChain, LlamaIndex, DSPy, Guidance, and LangGraph. Define and manage evaluation frameworks, including A/B testing, human and automated scoring, and model quality assessments using tools such as Ragas, UpTrain, DeepEval, TruLens, Promptfoo, and OpenAI Evals. Engineer and maintain context management systems, including conversation memory, retrieval logic, and context compression strategies (chunking, summarization, prioritization). Design and optimize retrieval and RAG pipelines, integrating vector databases (e.g., Weaviate) and rerankers (e.g., Cohere) to enhance relevance and accuracy. Establish observability and traceability mechanisms (e.g., Langfuse) to monitor performance metrics, latency, and operational costs. Implement guardrails, safety mechanisms, and bias mitigation policies using frameworks such as Guardrails AI, NeMo Guardrails, Outlines, and Rebuff. Collaborate with cross-functional engineering and research teams to ensure high-quality, compliant, and well-documented system delivery. Qualifications
Proven experience in LLM and NLP model operations, including tokenization, context windows, and safety considerations. Strong technical expertise in prompt engineering, orchestration frameworks, and context management. Hands-on experience with retrieval-augmented generation, evaluation frameworks, and observability tooling. Understanding of key evaluation metrics such as faithfulness, groundedness, coherence, toxicity, latency, and cost. Experience with compliance and ethical AI practices, including bias and hallucination mitigation. Excellent analytical, documentation, and communication skills, with the ability to collaborate effectively across teams. Preferred Experience
Model fine-tuning and alignment (e.g., LoRA, DPO/RLHF, preference tuning). Experience developing AI agents and integrating tool-use capabilities. Medical coverage and Health Savings Account (HSA) through Anthem Dental/Vision/Various Ancillary coverages through Unum 401(k) retirement savings plan Paid-time-off options Company-paid Employee Assistance Program (EAP) Discount programs through ADP WorkforceNow Additional Details
The base range for this contract position is $100,000 - 200,000 per year, depending on experience. Our pay ranges are determined by role, level, and location. The range displayed on each job posting reflects the minimum and maximum target for new hires of this position across all US locations. Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training. Qualified applicants with arrest or conviction records will be considered. Employment information
Seniority level: Mid-Senior level Employment type: Full-time Job function: Information Technology Industries: Technology, Information and Media By applying to this position, your data will be processed in accordance with the STAND 8 Privacy Policy.
#J-18808-Ljbffr
STAND 8 Technology Consulting is seeking an AI/ML Prompt Scientist/Engineer to design, evaluate, and optimize prompt-based systems that drive real-world impact across cutting-edge AI applications. In this role, you'll combine expertise in prompt engineering, orchestration frameworks, retrieval pipelines, and evaluation tooling to build scalable, reliable, and safe AI systems. This position offers a base pay range and opportunities to work with enterprise partners across the United States and internationally. Base pay range $100,000.00/yr - $200,000.00/yr About the company STAND 8 provides end-to-end IT solutions to enterprise partners across the United States and globally with offices in Los Angeles, New York, New Jersey, Atlanta, Mexico, Japan, India, and more. STAND 8 focuses on the bleeding edge of technology and leverages automation, process, marketing, and over fifteen years of success and growth to provide a world-class experience for our customers, partners, and employees. Our mission is to impact the world positively by creating success through PEOPLE, PROCESS, and TECHNOLOGY. Responsibilities
Design, optimize, and evaluate prompts and orchestration pipelines to enhance LLM performance and scalability. Implement programmatic prompting using frameworks such as LangChain, LlamaIndex, DSPy, Guidance, and LangGraph. Define and manage evaluation frameworks, including A/B testing, human and automated scoring, and model quality assessments using tools such as Ragas, UpTrain, DeepEval, TruLens, Promptfoo, and OpenAI Evals. Engineer and maintain context management systems, including conversation memory, retrieval logic, and context compression strategies (chunking, summarization, prioritization). Design and optimize retrieval and RAG pipelines, integrating vector databases (e.g., Weaviate) and rerankers (e.g., Cohere) to enhance relevance and accuracy. Establish observability and traceability mechanisms (e.g., Langfuse) to monitor performance metrics, latency, and operational costs. Implement guardrails, safety mechanisms, and bias mitigation policies using frameworks such as Guardrails AI, NeMo Guardrails, Outlines, and Rebuff. Collaborate with cross-functional engineering and research teams to ensure high-quality, compliant, and well-documented system delivery. Qualifications
Proven experience in LLM and NLP model operations, including tokenization, context windows, and safety considerations. Strong technical expertise in prompt engineering, orchestration frameworks, and context management. Hands-on experience with retrieval-augmented generation, evaluation frameworks, and observability tooling. Understanding of key evaluation metrics such as faithfulness, groundedness, coherence, toxicity, latency, and cost. Experience with compliance and ethical AI practices, including bias and hallucination mitigation. Excellent analytical, documentation, and communication skills, with the ability to collaborate effectively across teams. Preferred Experience
Model fine-tuning and alignment (e.g., LoRA, DPO/RLHF, preference tuning). Experience developing AI agents and integrating tool-use capabilities. Medical coverage and Health Savings Account (HSA) through Anthem Dental/Vision/Various Ancillary coverages through Unum 401(k) retirement savings plan Paid-time-off options Company-paid Employee Assistance Program (EAP) Discount programs through ADP WorkforceNow Additional Details
The base range for this contract position is $100,000 - 200,000 per year, depending on experience. Our pay ranges are determined by role, level, and location. The range displayed on each job posting reflects the minimum and maximum target for new hires of this position across all US locations. Within the range, individual pay is determined by work location and additional factors, including job-related skills, experience, and relevant education or training. Qualified applicants with arrest or conviction records will be considered. Employment information
Seniority level: Mid-Senior level Employment type: Full-time Job function: Information Technology Industries: Technology, Information and Media By applying to this position, your data will be processed in accordance with the STAND 8 Privacy Policy.
#J-18808-Ljbffr