AI Quality Engineer

Summary

PhoenixTeam supports federal customers with AI-enabled modernization and enterprise technology solutions. The AI Quality Engineer designs and integrates AI capabilities, establishes quality assurance frameworks, validates AI-generated code and prototypes, and ensures that AI solutions meet reliability, security, and compliance standards before production deployment.

Responsibilities

• Design and implement AI-enabled solutions for customer modernization use cases, developing and integrating AI capabilities into existing systems and workflows
• Establish and enforce a quality assurance framework — test plans, acceptance criteria, and regression suites — for validating AI-generated code, skills, and agents before they are promoted through the innovation pipeline
• Guide the development of reusable AI skills and agents for customer modernization tasks, with particular emphasis on automated testing, security scanning, data sensitivity, and scalability
• Collaborate with DevOps engineers to create and orchestrate long-running, multi-agent workflows, ensuring reliability, observability, and recoverability in production environments
• Support creation of the innovation pipeline that moves AI-generated (“vibe-coded”) prototypes from sandbox concepts into secure, enterprise-grade applications
• Support Value Engineers’ efforts by reviewing and quality-checking prototype outputs for accuracy, reliability, and adherence to standards prior to pipeline promotion
• Continuously monitor AI-assisted development outputs for accuracy, reliability, and compliance drift, feeding findings back to Value Engineers and AI Engineers to improve prompts, skills, and agents over time

Skills

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent professional experience
• Demonstrated experience in software quality assurance, test engineering, or a related discipline, including test plan design and defect management
• Hands-on experience testing or validating AI-generated code, prototypes, or “vibe-coded” solutions for accuracy and reliability
• Experience with automated testing tools and frameworks (e.g., Selenium) across web and enterprise applications
• Experience with one or more of the following: Salesforce, AWS (including Lambda and Redshift), Java, or PostgreSQL, or a comparable enterprise technology stack
• Familiarity with CI/CD pipelines and quality gates within modern software delivery practices
• Strong analytical skills, with the ability to identify security, data-sensitivity, and compliance risks in AI-assisted solutions
• Excellent written and verbal communication skills, with the ability to work across product, engineering, data, and business SME teams
• Must be a U.S. citizen and eligible to obtain and maintain a Public Trust clearance / favorable suitability determination
• Ability to use or teach tools such as ChatGPT, Claude, Gemini, Notion, Airtable, Zapier, LangChain, or similar
• Direct experience supporting Federal clients
• Experience with FedRAMP authorization processes
• Mortgage, fintech, or other regulated industry experience (preferred, not required — willingness to learn quickly is essential)
• Familiarity with AI safety, governance, and compliance
• Experience in startups, consulting, internal strategy, learning & development (L&D), or innovation teams

Qualifications

Must Haves

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent professional experience
• Demonstrated experience in software quality assurance, test engineering, or a related discipline, including test plan design and defect management
• Hands-on experience testing or validating AI-generated code, prototypes, or “vibe-coded” solutions for accuracy and reliability
• Experience with automated testing tools and frameworks (e.g., Selenium) across web and enterprise applications
• Experience with one or more of the following: Salesforce, AWS (including Lambda and Redshift), Java, or PostgreSQL, or a comparable enterprise technology stack
• Familiarity with CI/CD pipelines and quality gates within modern software delivery practices
• Strong analytical skills, with the ability to identify security, data-sensitivity, and compliance risks in AI-assisted solutions
• Excellent written and verbal communication skills, with the ability to work across product, engineering, data, and business SME teams
• Must be a U.S. citizen and eligible to obtain and maintain a Public Trust clearance / favorable suitability determination
• Ability to use or teach tools such as ChatGPT, Claude, Gemini, Notion, Airtable, Zapier, LangChain, or similar

Nice to Haves

• Direct experience supporting Federal clients
• Experience with FedRAMP authorization processes
• Mortgage, fintech, or other regulated industry experience (preferred, not required — willingness to learn quickly is essential)
• Familiarity with AI safety, governance, and compliance
• Experience in startups, consulting, internal strategy, learning & development (L&D), or innovation teams

Benefits

• 100% Remote (U.S.)