QA Engineer: AI-Assisted Testing

Summary

ECFX is transforming the legal industry through technology that retrieves, categorizes, and delivers court notices and legal documents. The QA Engineer will perform hands-on exploratory, functional, regression, backend, security, integration, and AI feature testing, using AI tools to improve test design and investigation. The role may also involve targeted test automation and close collaboration with engineering and product teams.

Responsibilities

• Use AI tools (such as Claude, Claude Code, Cursor, or ChatGPT) to read tickets, merge requests, and code diffs; map out risks and edge cases; draft test charters and checklists; and decide where to focus. You bring the judgment about what matters
• Run risk-based exploratory, functional, and regression testing across our Vue dashboard, admin tools, email notifications, and integrations. Find defects before customers do
• Make sure documents are retrieved, parsed, categorized, stored, and delivered correctly across many external sources: court portals, 2FA/OTP logins, and integrations such as Clio, Filevine, iManage, and NetDocuments. Hunt for silent failures, where a document is dropped but the run reports success
• Much of our work has no screen to click. Verify backend and platform changes by hand, using API calls, SQL queries, logs, and dashboards (Grafana/Loki, Sentry), with AI helping you write the queries and make sense of the output
• Use AI to generate test data, sample court notice emails, and edge-case documents, and drive our simulators and test environments to reproduce hard-to-reach situations
• Evaluate our AI document categorization and our LLM-facing tools (an MCP server used by AI assistants). Build sample sets, check accuracy and consistency, try to break them with unusual or hostile inputs, and flag when a model or prompt change makes results worse
• Check logins (password, SAML SSO, OAuth), user roles and permissions, and that one firm can never see another firm's data. Verify security fixes without breaking what already works
• Investigate customer-reported and production issues, reproduce them reliably, and write bug reports clear enough that an engineer, or an AI agent, can act on them right away
• If automation is in your toolkit, use AI to create focused automated checks, such as Playwright scripts or API checks, for high-risk flows you find yourself retesting
• Work with engineers and product managers in sprint planning, backlog refinement, and retrospectives. Help write testable acceptance criteria and raise quality risks early

Skills

• **Professional Experience:** 3 to 7 years in software quality assurance for web or SaaS applications, with deep experience in exploratory and manual testing of complex, data-heavy systems
• ·You use AI assistants in your testing every day and can give concrete examples: generating test ideas and edge cases, building test data, writing SQL or log queries, summarizing logs or diffs, reproducing bugs, and drafting clear reports
• * Skill at prompting and giving context, such as feeding in the ticket, the diff, and the acceptance criteria to get useful output instead of generic answers
• * Healthy skepticism. You verify what AI tells you, recognize when it is confidently wrong, and never let it replace your own judgment about risk
• * An understanding of how to test systems that don't always give the same answer, such as AI features, by using repeated trials, sample sets, and "is this answer right?" checks rather than exact matches
• * Careful handling of data. You use synthetic or masked data with AI tools and never paste customer or court data into unapproved services
• ·     Experience with defect tracking and test management in Jira
• * **Analytical Mindset:** Sharp attention to detail, curiosity, and a methodical approach to breaking software and finding root causes
• * **Communication Excellence:** Strong written and verbal communication for bug reports, test notes, and status updates. Your reports are precise enough that another person, or an AI agent, can pick them up and act
• * **Ownership:** When something seems "untestable," you find a way, whether that's a query, a log search, a simulator, or a question to the right engineer, rather than skipping it
• * **Remote Flexibility:** A fully remote role open to every US time zone, Hawaii included
• * Experience writing or maintaining automated tests (for example Playwright, Selenium, or API test scripts), including building them with AI and reviewing and fixing what it produces
• * Experience testing AI or LLM features, such as checking classification accuracy, spotting hallucinations, or testing for prompt injection
• * SQL skills for checking data in PostgreSQL
• * Experience testing APIs directly with tools like Postman, curl, or Bruno, and an understanding of HTTP, REST, and JSON
• * Experience with web security testing, including authentication and sessions, roles and permissions, OAuth2/SAML, CSRF, and XSS
• * Experience using AI browser tools (for example Playwright MCP or browser agents) to speed up exploratory testing
• * Experience testing integrations with third-party systems, email-driven workflows, or browser automation against external sites
• * Experience with accessibility testing (WCAG 2.2 AA)
• * Knowledge of the legal domain, especially court e-filing, docketing, or legal document management
• * ISTQB or a similar QA certification
• * An understanding of or experience with the Lean Startup approach
• ·     Comfort with the command line, Git, and reading logs. Some experience with Docker or Kubernetes (such as port-forwarding or checking pod logs) is a plus
• * **Automate Where It Pays Off (Optional):** If automation is in your toolkit, use AI to create focused automated checks, such as Playwright scripts or API checks, for high-risk flows you find yourself retesting

Qualifications

Must Haves

• **Professional Experience:** 3 to 7 years in software quality assurance for web or SaaS applications, with deep experience in exploratory and manual testing of complex, data-heavy systems
• ·You use AI assistants in your testing every day and can give concrete examples: generating test ideas and edge cases, building test data, writing SQL or log queries, summarizing logs or diffs, reproducing bugs, and drafting clear reports
• * Skill at prompting and giving context, such as feeding in the ticket, the diff, and the acceptance criteria to get useful output instead of generic answers
• * Healthy skepticism. You verify what AI tells you, recognize when it is confidently wrong, and never let it replace your own judgment about risk
• * An understanding of how to test systems that don't always give the same answer, such as AI features, by using repeated trials, sample sets, and "is this answer right?" checks rather than exact matches
• * Careful handling of data. You use synthetic or masked data with AI tools and never paste customer or court data into unapproved services
• ·     Experience with defect tracking and test management in Jira
• * **Analytical Mindset:** Sharp attention to detail, curiosity, and a methodical approach to breaking software and finding root causes
• * **Communication Excellence:** Strong written and verbal communication for bug reports, test notes, and status updates. Your reports are precise enough that another person, or an AI agent, can pick them up and act
• * **Ownership:** When something seems "untestable," you find a way, whether that's a query, a log search, a simulator, or a question to the right engineer, rather than skipping it
• * **Remote Flexibility:** A fully remote role open to every US time zone, Hawaii included
• * Experience writing or maintaining automated tests (for example Playwright, Selenium, or API test scripts), including building them with AI and reviewing and fixing what it produces
• * Experience testing AI or LLM features, such as checking classification accuracy, spotting hallucinations, or testing for prompt injection
• * SQL skills for checking data in PostgreSQL
• * Experience testing APIs directly with tools like Postman, curl, or Bruno, and an understanding of HTTP, REST, and JSON
• * Experience with web security testing, including authentication and sessions, roles and permissions, OAuth2/SAML, CSRF, and XSS
• * Experience using AI browser tools (for example Playwright MCP or browser agents) to speed up exploratory testing
• * Experience testing integrations with third-party systems, email-driven workflows, or browser automation against external sites
• * Experience with accessibility testing (WCAG 2.2 AA)
• * Knowledge of the legal domain, especially court e-filing, docketing, or legal document management
• * ISTQB or a similar QA certification
• * An understanding of or experience with the Lean Startup approach

Nice to Haves

• ·     Comfort with the command line, Git, and reading logs. Some experience with Docker or Kubernetes (such as port-forwarding or checking pod logs) is a plus
• * **Automate Where It Pays Off (Optional):** If automation is in your toolkit, use AI to create focused automated checks, such as Playwright scripts or API checks, for high-risk flows you find yourself retesting

Benefits

• A fully remote role open to every US time zone, Hawaii included.
• Flexible working hours and a supportive approach to getting work done.
• A range of insurance options covering health, dental, vision, and life.
• Discretionary time off (DTO) in addition to scheduled vacations and holidays, with supervisor approval for unplanned time off.
• The opportunity to participate in the company stock option plan.
• Company-provided AI tools, with clear guidelines for using them safely.