Remote Quality Assurance Engineer

Summary

Medical Review Institute of America is a clinically-driven, tech-enabled utilization management company offering expert clinical reviews and regulatory guidance. The Quality Assurance Engineer role focuses on ensuring the quality, reliability, and release-readiness of software products, particularly in testing applications and validating AI/ML features.

Responsibilities

• Primary role is testing software applications created or updated by the Application Development team
• Conduct structured peer and code reviews of test artifacts to ensure adherence to standards and best practices
• Design, create, and maintain realistic and compliant test data sets, including synthetic data for healthcare (PHI-safe) scenarios
• Design and execute validation strategies for AI/ML and Retrieval-Augmented Generation (RAG) features, including model quality, grounding, and safety evaluation
• Participate in the test management lifecycle, including test planning, execution tracking, defect management and reporting
• Provide input on release readiness criteria to Engineering and Product stakeholders
• Maintain the regression suite after every release
• Develop and maintain automated test suites in Python (e.g., pytest) and JavaScript/Node.js, integrated into CI/CD pipelines
• Write and review complex SQL queries, joins, and aggregations to validate data accuracy, completeness, and reconciliation across systems
• Validate data flows across source systems, transformation layers, and reporting/consumption layers
• Ensure test data supports both functional automation and AI/ML / RAG evaluation needs
• Design and execute test strategies for AI/ML and Retrieval-Augmented Generation (RAG) pipelines and AI-assisted features, including retrieval accuracy, grounding, hallucination, output quality, bias/safety, and drift (regression) checks
• Build and maintain automated evaluation harnesses and datasets for AI/ML features, including prompt/response scoring and metric tracking
• Validate applications, APIs, and test environments deployed on on-prem data center, Microsoft Azure, including Azure-hosted services, data stores, and CI/CD pipelines
• Read and interpret React / Node.js / Python / ColdFusion source code, pull requests, and technical designs to identify testable conditions, edge cases, and regression risk
• Write detailed test cases and test plans that trace directly to code paths, business rules, and acceptance criteria
• Partner with developers during code review to flag insufficient test coverage, untested branches, and high-risk changes before merge
• Perform post-deployment validation to confirm application, API, and AI feature stability in production
• Investigate production defects and data issues, perform root cause analysis, and drive resolution with Engineering

Skills

• 5+ years of experience in automated software testing
• 5+ years of experience in manual software testing
• 5+ years of experience in component-level testing
• 5+ years of experience in end-to-end testing
• At least 5 years of experience in SQL
• At least 3 years of experience in React
• At least 3 years of experience in Node.js
• At least 3 years of experience with Python source code in GitLab
• At least 2 years of experience testing AI/ML
• At least 2 years of experience evaluating AI/ML
• At least 2 years of experience testing Retrieval-Augmented Generation (RAG)
• At least 2 years of experience testing generative AI features
• At least 2 years of experience testing workloads on Microsoft Azure
• High School Diploma required
• Bachelor's degree (Advantage)

Qualifications

Must Haves

• 5+ years of experience in automated software testing
• 5+ years of experience in manual software testing
• 5+ years of experience in component-level testing
• 5+ years of experience in end-to-end testing
• At least 5 years of experience in SQL
• At least 3 years of experience in React
• At least 3 years of experience in Node.js
• At least 3 years of experience with Python source code in GitLab
• At least 2 years of experience testing AI/ML
• At least 2 years of experience evaluating AI/ML
• At least 2 years of experience testing Retrieval-Augmented Generation (RAG)
• At least 2 years of experience testing generative AI features
• At least 2 years of experience testing workloads on Microsoft Azure
• High School Diploma required

Nice to Haves

• Bachelor's degree (Advantage)

Benefits

• Benefits include healthcare, vision, and dental insurance
• A generous 401(k) match
• Paid vacation, PTO, and holidays
• Growth and training opportunities
• An award-winning remote work environment