Three months later, a redesign of the cart page breaks fourteen of them. Each test carried its own copy of the cart selectors and login steps, even though the team already had shared versions of both, so the fix takes days instead of minutes.

AI-driven quality engineering services are testing services that use AI to design, write, and maintain software tests. Teams also do this work themselves, with commercial tools or general-purpose models.

Whichever route a team takes, generating tests is fast. Whether those tests still work, and still make sense, a year later depends on how well they fit the codebase they land in. This article covers why generated tests drift out of fit, how to spot it, what it costs, why migrations raise the stakes, what to ask before relying on any AI approach, and the tradeoffs involved.

Why Do AI-Generated Tests Stop Fitting?

A mature test suite is more than a pile of tests, because over the years teams build shared parts that every new test is supposed to use.

Two of the most common are fixtures and page objects. A fixture is shared setup, such as a logged-in user or a seeded database, that many tests reuse. A page object is shared test code for one screen of the app. It holds the selectors, the instructions a test uses to find buttons and fields, so each test can simply say "add an item to the cart" instead of spelling out where every button sits. Playwright's documentation presents page objects as a way to reduce duplication and make tests easier to update when the interface changes.

A general-purpose model sees none of this unless you show it. It also doesn't know which flows have broken before, which naming rules the team follows, or which selectors are stable. So it fills the gaps with reasonable guesses: inline selectors, fresh login steps, and fixed waits. Each guess is harmless on its own, but together they create a second, parallel way of writing tests inside the same repository.

Review rarely catches this, because reviewers ask whether a test makes sense, not whether a helper for the same job already exists three folders away.

How Can You Spot Generated Tests That Won't Last?

Poor fit shows up in pull requests long before it shows up in a broken build, usually as one of these patterns.

• New Helpers That Copy Old Ones. A pull request adds its own "log in as an admin" steps when the suite already has shared setup for that.

• Selectors Tied To Layout. A test finds a button by its position on the page, such as "the button inside the third box," so it breaks as soon as a designer moves things around. Playwright's locator guidance recommends finding elements the way a user would, by their role or label, instead.

• Fixed Waits. A test pauses for a set time, such as three seconds, and hopes the page is ready, which makes it slow and still unreliable. Modern frameworks wait for elements automatically before acting on them.

• Early Retries And Skips. A generated test that picks up a retry setting or a skip within weeks of merging is a strong sign of poor fit.

A useful measure is the share of generated tests that needed edits, retries, or skips within two months of merging.

What Does Poor Fit Cost?

The first cost is maintenance that grows with every copy. When cart selectors live in one page object, a redesign means one fix; when they live in twenty tests, it means twenty. For an engineering leader, that shows up as release time spent repairing tests instead of adding coverage.

The second is flakiness, meaning a test that passes and fails on the same code. Google reported that about 1.5% of its test runs produced flaky results and that almost 16% of its tests showed some level of flakiness. An empirical study of flaky tests found that asynchronous waiting was the most common root cause, which is exactly the weakness fixed waits and fragile selectors invite.

The third is trust. Once red builds are often noise, people rerun failures instead of reading them, and real defects slip through.

Poor fit also makes later change harder. A framework upgrade or migration has to deal with every one-off pattern the suite has collected.

What Changes During a Framework Migration?

Context matters most when a suite moves between frameworks, such as from Selenium to Playwright. A line-by-line translation copies old habits, including fixed waits the new framework no longer needs.

A safer order is to map the old helpers and page objects to new ones first, then migrate tests onto that shared layer. Run old and new suites side by side for a few release cycles, and retire the old one only when the new one catches the same failures.

How Should You Evaluate AI-Driven Quality Engineering Services?

Whether the job is a migration or new coverage, the same questions apply when you weigh an outside provider, a commercial tool, or a model your own team runs.

1. What context does generation see? Find out whether it reads your existing helpers, fixtures, page objects, and conventions, or only the application and a prompt. If you run generation yourself, keep a short testing style guide and helper list in the repository and include both in every request.

2. Who checks fit before merge? Generated tests need a reviewer who knows the suite and asks one standing question: does this reuse what we already have?

3. Can automated checks catch the obvious problems? Lint rules for tests, such as those in eslint-plugin-playwright, can flag fixed waits and weak assertions before a person looks.

4. Are new tests run repeatedly before merge? Running each new test several times, for example with Playwright's repeat option, exposes flakiness while it is still cheap to fix.

5. How is maintenance handled? Ask who updates tests when the UI or an API changes, and who checks those updates.

6. Where does your code go? Ask whether source code or test data is sent to external models, and whether it is used for training.

7. Who owns the output? Tests should be committed to your repository, in your framework and language, so they keep running if the tool or provider changes.

What Are the Limits and Tradeoffs?

Giving generation enough context takes work, because someone has to write down the conventions and keep the helper list current.

Fit is not the same as strength: a test can reuse every helper and still check the wrong thing, so review for intent stays necessary.

Repeat runs and lint checks add pipeline time, so limit them to new and changed tests.

Finally, generation copies the codebase it learns from, so if the existing framework is inconsistent, a well-informed model may spread that inconsistency faster. Sometimes the right first step is cleaning up the framework itself.

How QASource Approaches This

Several checklist questions map to how QASource delivers this work. On context, test design, automation, and maintenance are generated by QASource's internal platform, which works from a client's requirements, code diffs, and defect history and matches the framework a team already uses. On review, a QASource engineer checks every generated case and script for correctness and intent before it is committed.

On maintenance, affected scripts are updated and re-verified when UI or API drift is detected, and duplicate tests are consolidated. On data, no client data is used to train or prompt external models. On ownership, every script is committed to the client's repository. QASource also used this internal platform for its own Selenium to Playwright migration, and a scoped assessment includes a generated-and-verified sample suite for one service.