How do you run only the regression tests affected by a change?
Running only the affected tests needs a way to map a change to the tests that cover it. Playwright's --only-changed flag does part of that: it runs only the test files that changed between HEAD and a git reference you name, and with no reference it uses your uncommitted changes.
BASH
# Test files changed in this branch compared with main
npx playwright test --only-changed=origin/main
# Rerun only the tests that failed in the last run
npx playwright test --last-failed
In our run, --only-changed also picked up a spec when we edited only a helper file that the spec imports. It does not know about your application code, though. If a developer changes the checkout service and touches no test file, --only-changed selects nothing, so use it to check the tests a pull request edits, not as your whole regression scope.
To select tests from application changes, you need a map from code areas to tests. The simplest version is a tag per feature area, such as @checkout, plus a rule that a change under the checkout code runs --grep @checkout. The fuller version is test impact analysis, which scores each changed file and picks the tests that cover it. Either way, keep a full run on a schedule, because no selection method is perfect.
How do you run regression tests in CI?
Run the smoke set on every pull request and the full regression set on a schedule, from the same workflow. This GitHub Actions file is based on the workflow in Playwright's CI guide, with a nightly schedule and tag filters added:
YAML
name: Regression tests
on:
pull_request:
branches: [ main ]
schedule:
- cron: '0 2 * * *'
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: actions/setup-node@v6
with:
node-version: lts/*
- name: Install dependencies
run: npm ci
- name: Install Playwright Browsers
run: npx playwright install --with-deps
- name: Smoke tests
if: github.event_name == 'pull_request'
run: npx playwright test --grep @smoke
- name: Full regression
if: github.event_name == 'schedule'
run: npx playwright test --grep @regression
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: playwright-report
path: playwright-report/
retention-days: 30
The report upload keeps the HTML report and traces for 30 days, so a nightly failure still has evidence the next morning. Add your test account's credentials as repository secrets and pass them to the test steps as environment variables, never as literals in the file.
Running the suite is half the job. If you want a failed run to block a merge, the next step is to integrate test management with CI/CD quality gates, so a regression failure stops the pipeline instead of posting a warning that people learn to scroll past.
How do you keep regression test maintenance low?

Keep maintenance low by writing tests that survive normal UI changes, catching flaky tests before they erode trust, reviewing every repair, and deleting tests that no longer protect anything.
Flaky tests are the biggest drain. Google reported that about 1.5% of all its test runs reported a flaky result, that almost 16% of its tests had some level of flakiness, and that about 84% of the pass-to-fail transitions it observed involved a flaky test. When most red runs are noise, people stop reading them, and a real regression slips through with the noise.
Playwright separates flaky results from real failures. With retries on, a test that fails and then passes on retry is reported as flaky, not passed, and --fail-on-flaky-tests makes the run fail when that happens:
BASH
npx playwright test --retries=2 --fail-on-flaky-tests
In our run, a test that failed on its first attempt and passed on retry was reported as "1 flaky", and with --fail-on-flaky-tests the command exited with a failure. That turns flakiness into work someone owns, instead of a green run that hides it.
What drives upkeep | What to do about it |
|---|
Fragile selectors | Use role, label, and text locators instead of CSS paths, so a markup change does not break the test |
Shared test data | Have each test create and clean up its own data, so tests pass alone and in any order |
Fixed waits | Replace sleeps with assertions that wait for a condition, which are faster and less flaky |
Silent repairs | Review every test fix as a diff, and reject any fix that makes an assertion check less |
Flaky tests left in place | Quarantine each one with an owner and a date, and fix or delete it before the date |
Duplicate and dead tests | Review the suite each quarter and retire tests that repeat others or cover removed features |
The silent-repair row deserves emphasis. The quickest way to turn a failing test green is to make it check less, and a fix that hides a real regression is worse than no test at all.
What should you measure in an automated regression suite?
Measure whether the suite catches regressions and what it costs to keep, not how many tests it has. A test count rises easily and tells you little.
Metric | What it tells you |
|---|
Escaped defects in covered areas | Whether the suite catches what it is meant to catch |
Time from merge to result | Whether results arrive while the change is still fresh |
Flaky rate | The share of runs with a flaky result, and whether trust in the suite is rising or falling |
Maintenance hours per sprint | What the suite costs to keep, counted in hands-on hours, not run time |
Coverage of release-blocking flows | Whether the flows from your ranked list are actually automated |
Track these from the first month. Without a starting value, you cannot show whether automation reduced escaped defects or just added a new kind of work.
How do you choose automated regression testing tools?
Choose automated regression testing tools by what they do after the first test is written: how they pick scope, how they report failures, and how repairs are reviewed. Most tools can run a test, so execution rarely separates them.
Ask each vendor of automated regression testing software these questions:
Scope. Can it select tests from what changed in the application, or only by file or tag?
Evidence. Does every failure carry a screenshot, trace, logs, and the change that caused it?
Repairs. When the UI changes, who fixes the test, and does a person approve each fix?
Sets. Can one test belong to a smoke set and a regression set without being copied?
Cost as the suite grows. Does the price rise with every run, or stay flat as the suite gets larger?
A side-by-side comparison of regression testing tools belongs in its own guide. Two neighboring topics come up in most shortlists: visual regression testing, which compares screenshots to catch layout changes that functional tests miss, and automated UI testing, which covers the browser-driven tests most regression suites are built from.
How does Test Studio handle regression testing?
Test Studio, our AI test automation platform, is built around the upkeep problems above, from choosing scope to reviewing repairs.
Sets from shared steps. A set groups cases, so a smoke set and a regression set draw on the same reusable steps. Each set can run on a schedule or from a CI webhook.
Scope from what changed. The Test Plan picks regression scope from changed files, scored by impact, change risk, reach, and coverage. That is the application-code selection that --only-changed leaves to you.
Release branches. Tests are versioned with the release branch, so a run for a release branch tests that branch's own cases.
Evidence. Every failure names the commit that broke it, with a screenshot, trace, and logs.
Repairs. When a UI change breaks a test, Test Studio proposes the repair as a diff your team approves. Nothing is repaired silently.
Cost. Test Studio is $220 per seat per month with no execution metering, so a growing regression suite does not raise the bill.
See Test Studio pick regression scope from your own changes. Book a demo to run it against your build, or download the solution brief for the detail.
Which option is right for your team?
Which option is right for your team depends on who will own the suite day to day:
You have engineers who will own a framework. Use the steps, tags, and CI workflow above, and budget for test maintenance and optimization from the start.
You have tooling and a mandate but no bandwidth, or a hard environment. Our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one. Talk to the team.
You want the function run for you. If you want the function run for you, our managed QA runs all six steps while the suite and sign-off stay with you. It is the service behind outsourced QA at ZeuZ.
Key takeaways
Tests are versioned with the release branch, so a run for a release branch tests that branch's own cases.
The Test Plan picks regression scope from changed files, scored by impact, change risk, reach, and coverage.
Every failure names the commit that broke it.
Repairs are proposed as a diff your team approves.
If you want the function run for you, our managed QA runs all six steps while the suite and sign-off stay with you.