AI Solutions Analyst

Summary

Public Consulting Group LLC is a public sector solutions implementation and operations improvement firm serving health, education, and human services agencies. The AI Solutions Analyst will support the design, delivery, evaluation, tuning, and monitoring of AI-enabled solutions in state government environments by translating client needs into requirements, analyzing system performance, and improving AI workflow quality. The role also supports client engagement, solution design, evaluation testing, and broader program operations.

Responsibilities

• Support and lead client requirements sessions and discovery discussions
• Translate policy and operational workflows into structured requirements
• Serve as a bridge between stakeholders and technical teams
• Support design of AI-enabled solutions and define use cases
• Develop and document AI workflows
• Design evaluation test sets with attention to sample size and representativeness across case types (LTC, ADB, etc.)
• Analyze evaluation results using basic statistical reasoning — error rates, false positive/negative patterns, distribution of failure modes — to identify where and why the system underperforms
• Iterate on prompts and instructions to address root causes identified through evaluation analysis
• Identify tradeoffs related to performance, cost, token usage, and explainability
• Partner with technical leads on tuning cycles within established review and sign-off processes
• Support broader program operations as needed — process design, cross-workstream coordination, and other work that helps scale delivery beyond a single engagement

Skills

• Strong communication skills
• Bachelor's degree from an accredited institution
• 4+ years of experience as a Business Analyst, Solutions Analyst, or Consultant
• Experience translating complex needs into structured deliverables
• Client-facing experience in requirements sessions and workshops
• Working knowledge of basic statistics — sample sizes, error rates, false positive/negative distinction, and comfort reading distributions — sufficient to design representative evaluation test sets and interpret results
• Hands-on experience iterating on prompts/instructions for LLM-based systems in a production or near-production setting
• Working knowledge of AI systems, including behavior, limitations, and agentic workflows
• Practical understanding of model selection tradeoffs, context/cost constraints, and agentic workflow limitations beyond prompt-level interaction with frontier models
• Technical understanding of system architecture, data flows, APIs, and integrations
• Experience supporting public-sector or compliance-driven programs
• Familiarity with AI implementation patterns such as RAG and multi-agent workflows
• Exposure to cloud platforms (AWS Bedrock preferred, Azure acceptable)
• Familiarity with AI risks: bias, hallucination, explainability
• Familiarity with compliance-driven delivery environments, including responsible AI practices, federal or state regulatory requirements, or structured governance frameworks

Qualifications

Must Haves

• Strong communication skills
• Bachelor's degree from an accredited institution
• 4+ years of experience as a Business Analyst, Solutions Analyst, or Consultant
• Experience translating complex needs into structured deliverables
• Client-facing experience in requirements sessions and workshops
• Working knowledge of basic statistics — sample sizes, error rates, false positive/negative distinction, and comfort reading distributions — sufficient to design representative evaluation test sets and interpret results
• Hands-on experience iterating on prompts/instructions for LLM-based systems in a production or near-production setting
• Working knowledge of AI systems, including behavior, limitations, and agentic workflows
• Practical understanding of model selection tradeoffs, context/cost constraints, and agentic workflow limitations beyond prompt-level interaction with frontier models
• Technical understanding of system architecture, data flows, APIs, and integrations

Nice to Haves

• Experience supporting public-sector or compliance-driven programs
• Familiarity with AI implementation patterns such as RAG and multi-agent workflows
• Exposure to cloud platforms (AWS Bedrock preferred, Azure acceptable)
• Familiarity with AI risks: bias, hallucination, explainability
• Familiarity with compliance-driven delivery environments, including responsible AI practices, federal or state regulatory requirements, or structured governance frameworks

Benefits

• Medical and dental care benefits
• 401k
• PTO
• Parental leave
• Bereavement leave