The bottleneck moved
Something has changed in software delivery, and every QA leader we speak to has felt it in the same quarter.
Development teams ship features 2 to 3 times faster with AI coding tools. QA teams produce more test artifacts than at any point in the last decade. Release confidence, the outcome that actually matters, has fallen.
At Clixlogix we run QA engagements across healthcare, field service, ecommerce, and legal buyers, and we see the same story on almost every incoming project. AI writes the tests. Nobody trusts them.
One CTO described the state of his team to us like this.
My developers are outrunning my QA. The suite grows every sprint, the pass rate stays high, and the escape defect rate does not fall.
The QA teams responded to that pressure. AI drafted test cases. AI summarized tickets. AI opened bug reports. AI wrote release notes. AI even wrote the automation. On the surface, throughput went up on every measure.
A senior QA engineer described watching a teammate sit down at a workstation, connect Claude to Playwright MCP, point the agent at an internal tool, and give it a clear brief. Drive a test flow. Confirm the happy path. Log the steps.
The agent started confidently. It read the UI. It clicked. It waited for the response. Then it wandered.
The selectors on the internal tool had never been designed for automation. The DOM shifted between screens. The agent misread a modal and clicked past it. It retried. It hit a role check it had not been briefed on. It retried again. It fell into a loop trying to generate UI driven test instructions from a surface that was drifting under it every few seconds.
The teammate watched. The token meter kept moving.
30 minutes in, the teammate stopped the run. Roughly USD 120 in Claude tokens had gone through the wire. The transcript ran to thousands of lines. The release manager still had no test he could trust.
The story is useful because nobody in it was careless. The goal was reasonable. The tools were modern. Claude is a capable agent. Playwright MCP is a capable interface. The system around them lagged behind. The app lacked stable selectors. The framework lacked fixtures. The workflow lacked a review gate.
This is where AI QA stops feeling like automation and starts feeling like theater. An agent can click. Testability comes from the application, the framework, and the workflow, and an agent inherits whatever those provide.
The demo said the agent could test. The invoice said it tried. The release manager still had no confidence.
The bottleneck had not disappeared. It had moved to judgment.
Fig 1 – The QA output confidence gap
Why AI code review became the real QA problem
The scene above shows one workflow. The same shape shows up across incoming projects, and it points at a specific gap in most engineering orgs. AI code review and AI code generation are 2 different disciplines, and teams that treat them as one problem lose control of both.
AI code generation moves fast. Most engineering teams have adopted it. Copilot, Cursor, Claude, and other agents write meaningful volumes of code every sprint. The operating model then breaks at the next step. The pull request lands. Who reviews it? What rubric do they use? Which parts of the change carry the risk? What does the reviewer do when the author cannot fully defend the change in plain English?
Traditional code review assumed a human author who could speak for every line. AI code review starts from a different set of facts. The author is an agent. The defender is often a rushed engineer running down a sprint list. The reviewer is a peer with 4 other PRs to look at that afternoon. In that set of assumptions, the AI generated code enters main, the tests generated for it enter the suite, and the release goes out with a pass rate that proves less than it looks.
Clixlogix positions AI code review as its own governance surface, separate from AI code generation and from traditional peer review. The 5 practices below build that surface. The 5 rules for the dev team install the guardrails. The RACI table names the owners. Every part of this piece assumes that AI code review is now the QA problem in disguise.
What we see on incoming projects
The first 2 weeks of every Clixlogix engagement follow the same script. The team finds the same 5 failure modes. Sometimes 4 of the 5, sometimes all 5, sometimes with a local twist. They cluster like this.
Test slop reads as coverage. Agents produce Playwright specs that navigate to a page, wait for it to load, and assert that fields render. Those tests pass. They also prove nothing about the user journey. A team can double its test count in a month and lower its actual coverage of business promises.
Acceptance criteria are too thin. If the ticket says “user can filter results”, an agent will build a test that filters and asserts a non empty list. If the business rule is that filters must persist across pagination and reset on logout, and if that rule lives in a product manager’s head, no agent catches it. In regulated fields the risk becomes concrete. Implied acceptance criteria never make it into the test suite, and developers using the same tools code the same blind spot.
Development output outpaces QA capacity. Development teams now merge 4,500 line and 2,500 line pull requests inside a single week with no proper code review. QA absorbs the fallout. Leadership calls QA the bottleneck. QA calls it math.
Human judgment is the scarce resource. Exploratory testing, risk prioritization, and reading a bug report to decide if it ships are the parts of the QA job that agents genuinely cannot do. The senior people who own those calls now spend most of their day reviewing AI output.
The framework gap is the real blocker. Teams try AI test automation on top of a fragile Playwright setup with no fixtures, no reliable selectors, no test data strategy, and flaky CI. Agents amplify that fragility. On one client engagement we measured a 7 percent flake rate on hand written specs against the same codebase where AI generated specs flaked at 52 to 68 percent. The framework set the ceiling on what any agent could produce there.
Two smaller signals matter for enterprise buyers. Context, security, and token cost are real adoption blockers. Some organizations forbid sharing source code with any external AI client, which reduces QA use to walled question and answer. Ownership of AI produced tests is often unclear. When a test breaks in CI, no name is on it. It stays broken.
So the failure has shape. What produces it is a workflow so obvious most teams adopt it in the first month of AI use.
Why the obvious AI QA workflow fails
Every QA team we scope has already tried AI in testing before we arrive. The workflow they built looks almost identical across healthcare, ecommerce, field service, and legal engagements, and it fails for the same reasons every time. The team means well, the tooling works, and the engineer running it is often the strongest hands on tester on the team. The steps look like this.
- A QA engineer drops a ticket and a pull request into
Claude or Cursor. - The agent produces a test plan and a set of
Playwright specs. - The engineer runs the specs. Some pass. Some fail.
- The engineer marks the ticket done and moves to the next one.
Fig 2 – The obvious AI QA workflow and its 3 collapse points
This workflow collapses in 3 predictable ways.
Collapse #1. The agent works from what the ticket says, and most tickets are thin. The test plan looks thorough. It covers everything named in the ticket. It covers nothing implied. Bugs escape through the gap between what the product manager wrote and what the customer expects.
Collapse #2. The agent generates tests that assert page structure. The tests skip business outcomes. The test suite grows. The mean information content per test falls. Regression runs get slower and less useful with every sprint.
Collapse #3. Nobody owns the resulting suite. The agent wrote it. The engineer ran it once. When it fails 4 weeks later on a refactor, the engineer who added it is on a different project, and the engineer on duty skips the failure and marks the run green. Over 6 months, 15 percent of the suite becomes non actionable.
We see this shape on 8 of the last 12 QA engagements Clixlogix scoped. The teams are doing what the obvious workflow suggests, and the obvious workflow is under specified.
The fix is a different workflow, engineered in a fixed order, with governance in the loop.
Fig 3 – From AI output to release confidence
What we mean by AI augmented QA
Clixlogix runs QA engagements around 5 practices in a fixed order. The order matters. Every practice depends on the previous one working.
The 5 practices at a glance
Fig 4 – The 5 practices in sequence
Now walking through each in the order it runs on a real engagement.
1. Requirements hardening comes first. Before an agent writes a single test, a Clixlogix QA lead reviews the ticket with the product manager and the tech lead. Vague tickets become testable acceptance criteria, edge case inventories, and business promise statements. We write down what the feature must do, what it must refuse to do, and what a customer would notice if it broke. This step alone catches 40 to 60 percent of the bugs an AI generated test suite would miss on the same feature.
2. Automation foundation comes second. We build or refactor the test framework so agents can work reliably on top of it. That means a Playwright architecture with a working page object model, stable data-testid selectors on the UI, a fixtures library that sets up scenarios in seconds, seeded accounts, generated edge case data, mocks and service virtualization for dependencies still in development, contract checks and mocks generated from OpenAPI and Swagger specs so QA can test integrations before every dependency is live, and CI that runs the same on developer machines and on the pipeline. A QA team can safely delegate spec authoring to an agent once a solid framework is in place.
3. Governed AI test authoring comes third. With hardened requirements and a working framework in place, agents produce useful specs. Every AI produced test asset passes a review rubric before it enters the suite.
The AI code review rubric
The rubric checks 6 things.
- Does the test cover a business outcome?
- Are the assertions strong?
- Would this test fail if the business rule broke?
- Does the test read as a user journey?
- Is a human owner named?
- Did an independent agent review the test, separate from the model and context that produced the code under test?
A Clixlogix QA engineer approves entry, and the review record travels with the test through every future refactor.
Fig 5 – The AI code review rubric
We validate the “would this test fail” question with targeted mutation checks. A QA engineer flips a signed comparison, deletes a boundary condition, or breaks a permission check, and confirms that at least one test in the suite catches it. Suites that pass their mutation checks are suites that would catch a real regression.
Fig 6 – The mutation check loop
Bug reports get the same treatment. AI writes clean defect prose from evidence the tester hands it, and only the tester in the room can capture that evidence in the moment. Every defect that enters Jira carries the exact path clicked, the user and account state, the browser and device, the console and network capture, screenshots or video, and the Playwright trace file. A Clixlogix QA engineer captures the evidence before the agent writes the report.
4. Humans own the risk work. Clixlogix QA engineers own exploratory testing, risk prioritization, escape analysis, and the release go decision. Agents run the repeatable work of writing and running specs. The role of the senior QA engineer becomes higher in scope and higher in accountability, and smaller in headcount for the same delivery volume.
We build a calibration loop into this practice. Every quarter, a senior QA engineer writes an exploratory charter for a chosen feature by hand, works through it, and compares the resulting findings against what the AI generated coverage caught for the same feature. Gaps get logged. The team keeps its testing muscle, and Clixlogix keeps a running record of where AI still misses.
Fig 7 – The human calibration loop
5. Release readiness becomes a metric. We report 6 numbers on every release cycle.
- Test suite flake rate
- Coverage of business acceptance criteria
- AI produced tests reviewed and approved
- Escape defects logged
- Time from commit to release confidence
- Cost per release in AI tokens
A CTO who reads those 6 numbers knows the state of QA in under 2 minutes. Governed QA also checks whether the feature can be safely observed, flagged, rolled back, and diagnosed after release. A test suite that passes on a feature nobody can turn off in production is not release ready.
What governed QA includes beyond test generation
The 5 practices define how Clixlogix governs functional test authoring on a web application. Real QA covers more ground. We build 4 additional modules into the same operating model.
QA for AI products. When the product itself uses AI, test cases will not prove it works. Clixlogix builds evaluation sets, prompt regression suites, factual accuracy checks for RAG answers, safety probes, and red team scenarios, using Braintrust, promptfoo, DeepEval, Ragas, garak, and PyRIT across engagements. Every model swap, every prompt edit, and every context change moves the output, and the eval set is what tells the team whether the change shipped a fix or a regression.
Visual, accessibility, and design QA. Functional flows can pass while the interface drifts. Clixlogix runs screenshot comparison against a versioned baseline, WCAG accessibility scans, and design token audits against the Figma source. Design drift, color contrast failures, and broken keyboard navigation get caught before customers do.
QA workflow integration. Requirements live in Jira, ADO, or Confluence. Test cases live in TestRail, Zephyr, QASE, or Rovo. Automation lives in GitHub or GitLab. Evidence lives in Slack threads and Playwright traces. Clixlogix builds the connective tissue so a ticket, its acceptance criteria, its tests, its runs, its evidence, its defects, and its release notes carry the same identifier across every tool. Without that connection, audit becomes impossible.
Agent independence. The model writing the test must not be the model that wrote the code. Same model, same context, same blind spot. Clixlogix runs code and tests through separate agent identities with different system prompts, and pairs adversarial review agents with authoring agents on high risk features.
Fig 8 – Independent AI code review agents
We apply the same governed model beyond web Playwright, on iOS and Android through Appium, on APIs through contract tests, and on browser and device labs through BrowserStack. Operational QA paperwork like release notes, rollback notices, and pipeline triage runs through the same review rubric as tests, with a named human approver on every AI generated artifact.
So far this is the QA leader’s operating model. The next 5 sections speak to the people whose choices land upstream of QA, the engineering leads and the dev teams.
What engineering leads should change this sprint
Engineering leads can make 5 changes in the next 2 week cycle without waiting on Clixlogix or anyone else.
Keep AI generated pull requests small enough to review. A 4,500 line PR does not get a real review in one sitting. Enforce a size ceiling that reflects your team’s actual review capacity, and split larger changes into landed slices.
Require every developer to explain AI written code in review. If the author cannot defend a block of code in plain English, the block does not merge. This one rule catches most of the “I asked Claude and shipped what it wrote” failures we see on incoming projects.
Add automation coverage expectations to the pull request template. The template asks 3 questions before submit. Which business behavior does this change cover? What is the test intent? Which failure mode does the new test catch?
Require reproducibility evidence for “tested by AI” claims. A developer who says the change is tested attaches the run log, the test file, and the assertion set. Screenshots of chat windows do not count.
Block large AI generated PRs without a risk note and rollout plan. The risk note names the affected surfaces, the rollback trigger, and the observability signal that would catch a regression. Without those 3 items, the PR waits.
What dev leaders should own, and not push to QA
The failure mode we see on incoming projects starts with the dev team pushing quality work onto QA. AI accelerates the trend. Dev leaders can reclaim these 5 responsibilities inside their own team this quarter, with no external help.
- Unit, contract, and integration tests stay dev owned.
- Feature flags and rollback plans stay engineering owned.
- Observability and logs stay engineering owned.
- Testability lives in the design of the code and belongs to engineering.
- Unreviewed AI code is not QA’s cleanup job under any conditions.
The dev leader who owns those 5 stays out of the escape defect briefing. The dev leader who pushes them onto QA ends up in the room explaining last quarter’s outage.
AI code review rules for the dev team
Any dev team can adopt a short set of AI code review rules this week. 5 rules cover most of the ground, and Clixlogix uses this same set on client engagements.
AI code goes through review at the level of code from a junior engineer. Same rubric, same rigor.
No AI generated code enters main without a named human who can defend it. Every commit includes an attribution trailer, and reviewers see who wrote what and who approved what.
Authoring and review run on separate agents with different system prompts. Same rule as in QA. Same reason.
No massive AI generated pull requests unless the sprint plan approved a refactor. Ad hoc large PRs get returned to the author.
Every AI assisted change carries a test intent note. The note names what the change is meant to do, what it must not break, and which existing test proves the second half.
Ownership across Product, Engineering, QA, and Leadership
Governance works when the 4 roles agree on the split. Clixlogix uses one page.
Every incoming Clixlogix engagement starts with a review of this page against how the team actually operates. The gaps usually name the source of the escape defects that got the CTO to call us.
Fig 9 – The 4 role ownership map
A team lead scorecard
The CTO reads 6 numbers per release. The engineering lead reads a different 8 numbers each sprint, and Clixlogix generates all of them from CI, the code hosting platform, and the test framework.
- Pull request size trend
- AI generated pull request percentage
- Review depth measured by reviewer comment count
- Defects found before QA
- Escape defects by source team
- Flaky test count
- Tests added by test type across unit, contract, integration, and end to end
- Rollback readiness for each shipped feature
An engineering lead who reads those 8 numbers weekly can act on trends before they reach the escape defect briefing. Clixlogix builds the scorecard as part of the readiness audit and keeps it running through the engagement.
Fig 10 – The dual dashboard
Now for the outcomes. What actually changes for the people in the seats when a company runs the governed model for 6 to 9 months.
What changes for the Product leader
Product leaders own the business promise. In the AI delivery era, vague tickets become production risk faster than at any point in the last decade, and the escaped defects that reach the customer usually trace to a business rule that nobody wrote down.
Product’s contract with QA is 3 things. Make acceptance criteria explicit. Name the edge cases that matter. Stay available when QA finds a gap between the ticket and the customer promise.
The Product leader who takes on that contract sees a return that reaches beyond QA. Requirements written for a governed test process are also better requirements for the humans building the feature, for the account manager selling it, and for the support team answering questions about it after launch. One good ticket saves 3 escape defects and a Slack thread.
Requirements quality is the input with the largest downstream effect. Product owns it.
Fig 11 – Product's contract with QA
What changes for the QA engineer
The QA role at Clixlogix client sites in the AI era looks different from 2023. Fewer engineers on the roster. Higher seniority per seat. Broader scope per person.
Agents now do the junior work of writing rote Selenium scripts against a fresh UI, generating the first pass of test cases from a ticket, and drafting bug prose from captured evidence. That work used to fill the calendars of manual testers and junior SDETs at every client we scoped. It is gone.
The senior work has become more valuable. Choosing what to test on a feature nobody has seen before. Reading a bug report from an unfamiliar user and deciding if it ships. Running an exploratory session and reporting findings that the acceptance criteria never named. Owning the release go call on a Friday afternoon. Those decisions belong to a person, and a good QA engineer can make 5 of them on the same day an AI is generating 500 test artifacts.
We see the shift in the role descriptions we help clients write. Clients replace QA engineer titles with senior QA engineer, staff SDET, and QA lead. The manual testers of 2023 who moved into risk work now command higher salaries in 2026 than they did 3 years ago. Companies let go of the manual testers who stayed in scripted execution first.
The role compresses. The people who own it are worth more.
What changes for the CTO
3 numbers on the CTO dashboard move within 2 quarters of a governed rollout.
Release quality becomes predictable. With hardened requirements and a governed review process, escape defects fall by 30 to 50 percent on Clixlogix engagements. The CTO gets fewer 2 AM calls. The board slide showing outages per quarter starts to bend down. Support ticket volume tied to escaped defects follows the same trend, which shifts customer success capacity back toward retention and expansion work.
AI cost becomes controlled. A monthly budget bounds token spend on QA workflows. Clixlogix tracks it per feature and reports it alongside developer AI spend. Wasted tokens on failed agent runs stop showing up as surprise line items in the FP&A close. The CTO can point at a feature and name what QA on that feature cost in agent time. Finance stops asking why the AI line grows every month with no attributable output.
Audit trail becomes defensible. Every AI produced test asset carries its review record. The compliance team can trace a released feature to the tests that cleared it, and every test back to the acceptance criteria it proved. This matters for regulated buyers in healthcare, finance, and industrial. It matters for security questionnaires, SOC 2 evidence collection, and any customer that asks how AI was used to test their data flow before it went live.
The CTO stops firefighting AI costs and starts governing them.
Where to start
The Clixlogix QA readiness audit runs over 2 weeks and scores 6 dimensions.
The output is a 1 page scorecard, a 90 day plan, and a decision on which of the 5 practices your team runs first.
Fig 12 – The readiness audit timeline
Week 1
A senior Clixlogix QA lead sits with the team, reviews the last 3 sprints of tickets, tests, PRs, and escape defects, and interviews the QA, Engineering, and Product leads. We do not touch the codebase in week 1. We are listening.
Week 2
Clixlogix builds the scorecard, drafts the 90 day plan, and walks the client through the findings. The plan names the first practice to install, the team members who own it, and the metrics that will show it working within 60 days.
Clixlogix takes on 4 audits per quarter. Most clients start with Practice 1 and Practice 2 in the first 90 days, add Practice 3 in the following quarter, and mature into Practices 4 and 5 across the following 6 to 9 months. The audit itself pays for itself inside 1 release cycle for teams already spending on AI QA tooling.
Frequently asked questions
