ZeuZ

How to Automate Regression Testing and Keep Upkeep Low

Key Takeaways

  • Tests are versioned with the release branch, so a run for a release branch tests that branch's own cases.
  • The Test Plan picks regression scope from changed files, scored by impact, change risk, reach, and coverage.
  • Every failure names the commit that broke it.
  • Repairs are proposed as a diff your team approves.
  • If you want the function run for you, our managed QA runs all six steps while the suite and sign-off stay with you.
Blog

Share with your community!

How to Automate Regression Testing and Keep Upkeep Low


Regression testing in software development is the practice of rerunning existing tests after every change to confirm that features which already worked still work, and you automate it by ranking the flows that matter most, writing stable tests for them, running them in CI on every change, and budgeting time to keep them current.

The ISTQB glossary defines regression testing as a type of change-related testing that detects whether defects have been introduced or uncovered in unchanged areas. Moving that check from a release-week chore to every pull request is what automated regression testing is for. Writing the first hundred tests is the easy part. Keeping them passing for the right reasons as the product changes is where most suites lose their value, so this guide spends as much time on upkeep as on setup.

We ran the Playwright commands in this guide on Playwright 1.56 against a small demo app. The CI workflow follows Playwright's own CI guide, and we did not run it in GitHub Actions.

What is regression test automation, and when is it worth it?

Regression test automation means scripting your regression checks so a machine runs them on every change, and it is worth it once the same checks repeat often enough that running them by hand slows releases or gets skipped.

Four signals usually mean it is time:

  • Releases are frequent. If you ship weekly or faster, a manual regression pass either delays the release or gets cut short.

  • The same checks repeat. Testers run the same sign-in, checkout, or report flows before every release.

  • Old features break. Defects reach production in areas nobody touched in that release.

  • The flows are stable. The screens and APIs you want to check change less often than you release.

It is not worth it for a feature that is still changing every week, for a check you will run once, or for exploratory testing, where a person's judgment is the point. Regression testing automation pays back on repetition, so start where repetition is highest.

What should you automate first in a regression suite?

Automate the flows that block a release when they fail, then the areas that have broken before, and leave fast-changing and judgment-heavy checks with people until they settle.

Candidate

Why automate it early

Watch out for

Release-blocking flows

Sign-in, checkout, payments, and core reports are the failures customers notice first

Shared test accounts and data that tests overwrite

Areas with past defects

A defect that shipped once tends to come back when nearby code changes

Tests that only re-check the original bug, not the behavior around it

Code that changes often

High change risk means a higher chance of breaking something that worked

UI that is still being redesigned, which turns tests into rework

APIs and integrations

API checks run fast and break less often than UI checks

Third-party services you do not control, which you should mock

New features still in flux

Usually leave manual for now

Automating before the design settles, then rewriting the tests

Visual polish and exploratory checks

Usually leave manual

Expecting a script to judge whether a screen looks right


A short list of well-chosen tests beats a long list of fragile ones. Twenty tests on release-blocking flows that pass for the right reasons are worth more than five hundred that the team has learned to ignore.


How do you automate regression testing, step by step?

what-to-automate-first-regression-suite

To automate regression testing, work through six steps in order: test design, test case creation, test automation, test execution, QA reporting and analysis, and test maintenance and optimization. Each step has its own output, and skipping one is usually why a suite decays.

  1. Test design. List the flows from the table above and rank them by impact, how often the code behind them changes, and how many users they reach. The ranked list is your scope.

  2. Test case creation. For each flow, write down the steps, the test data, and the expected result before anyone writes code. A case a product owner can read is a case you can review later.

  3. Test automation. Turn each case into a test with user-facing locators, such as roles and labels, and with test data the test creates for itself. Tag each test so you can run it as part of a smaller or larger set.

  4. Test execution. Run a fast smoke set on every pull request and the full regression set on a schedule or before a release, in CI, in parallel.

  5. QA reporting and analysis. Every failure needs evidence: the step, the error, a screenshot or trace, and the change that caused it. Triage each failure as a product bug, a test bug, or a flaky test.

  6. Test maintenance and optimization. Repair tests when the product changes on purpose, review each repair, and retire tests that duplicate others or no longer protect anything.

Steps 01, 05, and 06 are the ones teams underestimate. A framework helps most with test automation and test execution, and the rest is process.

What should you automate first in a regression suite?

Automate the flows that block a release when they fail, then the areas that have broken before, and leave fast-changing and judgment-heavy checks with people until they settle.

Candidate

Why automate it early

Watch out for

Release-blocking flows

Sign-in, checkout, payments, and core reports are the failures customers notice first

Shared test accounts and data that tests overwrite

Areas with past defects

A defect that shipped once tends to come back when nearby code changes

Tests that only re-check the original bug, not the behavior around it

Code that changes often

High change risk means a higher chance of breaking something that worked

UI that is still being redesigned, which turns tests into rework

APIs and integrations

API checks run fast and break less often than UI checks

Third-party services you do not control, which you should mock

New features still in flux

Usually leave manual for now

Automating before the design settles, then rewriting the tests

Visual polish and exploratory checks

Usually leave manual

Expecting a script to judge whether a screen looks right


A short list of well-chosen tests beats a long list of fragile ones. Twenty tests on release-blocking flows that pass for the right reasons are worth more than five hundred that the team has learned to ignore.

How do you automate regression testing, step by step?

To automate regression testing, work through six steps in order: test design, test case creation, test automation, test execution, QA reporting and analysis, and test maintenance and optimization. Each step has its own output, and skipping one is usually why a suite decays.

  1. Test design. List the flows from the table above and rank them by impact, how often the code behind them changes, and how many users they reach. The ranked list is your scope.

  2. Test case creation. For each flow, write down the steps, the test data, and the expected result before anyone writes code. A case a product owner can read is a case you can review later.

  3. Test automation. Turn each case into a test with user-facing locators, such as roles and labels, and with test data the test creates for itself. Tag each test so you can run it as part of a smaller or larger set.

  4. Test execution. Run a fast smoke set on every pull request and the full regression set on a schedule or before a release, in CI, in parallel.

  5. QA reporting and analysis. Every failure needs evidence: the step, the error, a screenshot or trace, and the change that caused it. Triage each failure as a product bug, a test bug, or a flaky test.

  6. Test maintenance and optimization. Repair tests when the product changes on purpose, review each repair, and retire tests that duplicate others or no longer protect anything.

Steps 01, 05, and 06 are the ones teams underestimate. A framework helps most with test automation and test execution, and the rest is process.

How do you split a regression suite into smoke and full runs?

smoke-and-full-regression-runs-in-ci

Split the suite with tags: mark a small set of release-blocking tests as smoke, mark everything in scope as regression, and run each set by tag. In Playwright, a tag is an @-prefixed string you pass in the test's details object.

TYPESCRIPT

import { test, expect } from '@playwright/test';

const both = { tag: ['@smoke', '@regression'] };

test('signs in with valid credentials', both, async ({ page }) => {

  await page.goto('/login');

  await page.getByLabel('Email').fill(process.env.E2E_EMAIL!);

  await page.getByLabel('Password').fill(process.env.E2E_PASSWORD!);

  await page.getByRole('button', { name: 'Sign in' }).click();

  await expect(page.getByRole('heading', { name: 'Welcome back' })).toBeVisible();

});

 

test('rejects a wrong password', { tag: '@regression' }, async ({ page }) => {

  await page.goto('/login');

  await page.getByLabel('Email').fill(process.env.E2E_EMAIL!);

  await page.getByLabel('Password').fill('not-the-password');

  await page.getByRole('button', { name: 'Sign in' }).click();

  await expect(page.getByRole('alert')).toHaveText('Invalid email or password');

});

Then run each set with --grep, and exclude a tag with --grep-invert:

BASH

npx playwright test --grep @smoke

npx playwright test --grep @regression

npx playwright test --grep @regression --grep-invert @smoke

In our demo suite of three tests, the first command ran one test, the second ran all three, and the third ran the two regression tests that are not also smoke tests.

Teams often ask "is smoke and regression testing the same?", and the short answer is no. A smoke run asks whether the build works well enough to test further, and a regression run asks whether anything that used to work has broken. Tags let one suite serve both questions without keeping two copies of the same steps.



How do you run only the regression tests affected by a change?

Running only the affected tests needs a way to map a change to the tests that cover it. Playwright's --only-changed flag does part of that: it runs only the test files that changed between HEAD and a git reference you name, and with no reference it uses your uncommitted changes.

BASH

# Test files changed in this branch compared with main

npx playwright test --only-changed=origin/main

 

# Rerun only the tests that failed in the last run

npx playwright test --last-failed

In our run, --only-changed also picked up a spec when we edited only a helper file that the spec imports. It does not know about your application code, though. If a developer changes the checkout service and touches no test file, --only-changed selects nothing, so use it to check the tests a pull request edits, not as your whole regression scope.

To select tests from application changes, you need a map from code areas to tests. The simplest version is a tag per feature area, such as @checkout, plus a rule that a change under the checkout code runs --grep @checkout. The fuller version is test impact analysis, which scores each changed file and picks the tests that cover it. Either way, keep a full run on a schedule, because no selection method is perfect.

How do you run regression tests in CI?

Run the smoke set on every pull request and the full regression set on a schedule, from the same workflow. This GitHub Actions file is based on the workflow in Playwright's CI guide, with a nightly schedule and tag filters added:

YAML

name: Regression tests

on:

  pull_request:

    branches: [ main ]

  schedule:

    - cron: '0 2 * * *'

jobs:

  test:

    runs-on: ubuntu-latest

    steps:

      - uses: actions/checkout@v6

      - uses: actions/setup-node@v6

        with:

          node-version: lts/*

      - name: Install dependencies

        run: npm ci

      - name: Install Playwright Browsers

        run: npx playwright install --with-deps

      - name: Smoke tests

        if: github.event_name == 'pull_request'

        run: npx playwright test --grep @smoke

      - name: Full regression

        if: github.event_name == 'schedule'

        run: npx playwright test --grep @regression

      - uses: actions/upload-artifact@v4

        if: ${{ !cancelled() }}

        with:

          name: playwright-report

          path: playwright-report/

          retention-days: 30

The report upload keeps the HTML report and traces for 30 days, so a nightly failure still has evidence the next morning. Add your test account's credentials as repository secrets and pass them to the test steps as environment variables, never as literals in the file.

Running the suite is half the job. If you want a failed run to block a merge, the next step is to integrate test management with CI/CD quality gates, so a regression failure stops the pipeline instead of posting a warning that people learn to scroll past.

How do you keep regression test maintenance low?

keep-regression-test-maintenance-low

Keep maintenance low by writing tests that survive normal UI changes, catching flaky tests before they erode trust, reviewing every repair, and deleting tests that no longer protect anything.

Flaky tests are the biggest drain. Google reported that about 1.5% of all its test runs reported a flaky result, that almost 16% of its tests had some level of flakiness, and that about 84% of the pass-to-fail transitions it observed involved a flaky test. When most red runs are noise, people stop reading them, and a real regression slips through with the noise.

Playwright separates flaky results from real failures. With retries on, a test that fails and then passes on retry is reported as flaky, not passed, and --fail-on-flaky-tests makes the run fail when that happens:

BASH

npx playwright test --retries=2 --fail-on-flaky-tests

In our run, a test that failed on its first attempt and passed on retry was reported as "1 flaky", and with --fail-on-flaky-tests the command exited with a failure. That turns flakiness into work someone owns, instead of a green run that hides it.

What drives upkeep

What to do about it

Fragile selectors

Use role, label, and text locators instead of CSS paths, so a markup change does not break the test

Shared test data

Have each test create and clean up its own data, so tests pass alone and in any order

Fixed waits

Replace sleeps with assertions that wait for a condition, which are faster and less flaky

Silent repairs

Review every test fix as a diff, and reject any fix that makes an assertion check less

Flaky tests left in place

Quarantine each one with an owner and a date, and fix or delete it before the date

Duplicate and dead tests

Review the suite each quarter and retire tests that repeat others or cover removed features


The silent-repair row deserves emphasis. The quickest way to turn a failing test green is to make it check less, and a fix that hides a real regression is worse than no test at all.

What should you measure in an automated regression suite?

Measure whether the suite catches regressions and what it costs to keep, not how many tests it has. A test count rises easily and tells you little.

Metric

What it tells you

Escaped defects in covered areas

Whether the suite catches what it is meant to catch

Time from merge to result

Whether results arrive while the change is still fresh

Flaky rate

The share of runs with a flaky result, and whether trust in the suite is rising or falling

Maintenance hours per sprint

What the suite costs to keep, counted in hands-on hours, not run time

Coverage of release-blocking flows

Whether the flows from your ranked list are actually automated


Track these from the first month. Without a starting value, you cannot show whether automation reduced escaped defects or just added a new kind of work.

How do you choose automated regression testing tools?

Choose automated regression testing tools by what they do after the first test is written: how they pick scope, how they report failures, and how repairs are reviewed. Most tools can run a test, so execution rarely separates them.

Ask each vendor of automated regression testing software these questions:

  • Scope. Can it select tests from what changed in the application, or only by file or tag?

  • Evidence. Does every failure carry a screenshot, trace, logs, and the change that caused it?

  • Repairs. When the UI changes, who fixes the test, and does a person approve each fix?

  • Sets. Can one test belong to a smoke set and a regression set without being copied?

  • Cost as the suite grows. Does the price rise with every run, or stay flat as the suite gets larger?

A side-by-side comparison of regression testing tools belongs in its own guide. Two neighboring topics come up in most shortlists: visual regression testing, which compares screenshots to catch layout changes that functional tests miss, and automated UI testing, which covers the browser-driven tests most regression suites are built from.

How does Test Studio handle regression testing?

Test Studio, our AI test automation platform, is built around the upkeep problems above, from choosing scope to reviewing repairs.

  • Sets from shared steps. A set groups cases, so a smoke set and a regression set draw on the same reusable steps. Each set can run on a schedule or from a CI webhook.

  • Scope from what changed. The Test Plan picks regression scope from changed files, scored by impact, change risk, reach, and coverage. That is the application-code selection that --only-changed leaves to you.

  • Release branches. Tests are versioned with the release branch, so a run for a release branch tests that branch's own cases.

  • Evidence. Every failure names the commit that broke it, with a screenshot, trace, and logs.

  • Repairs. When a UI change breaks a test, Test Studio proposes the repair as a diff your team approves. Nothing is repaired silently.

  • Cost. Test Studio is $220 per seat per month with no execution metering, so a growing regression suite does not raise the bill.

See Test Studio pick regression scope from your own changes. Book a demo to run it against your build, or download the solution brief for the detail.

Which option is right for your team?

Which option is right for your team depends on who will own the suite day to day:

  • You have engineers who will own a framework. Use the steps, tags, and CI workflow above, and budget for test maintenance and optimization from the start.

  • You have tooling and a mandate but no bandwidth, or a hard environment. Our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one. Talk to the team.

  • You want the function run for you. If you want the function run for you, our managed QA runs all six steps while the suite and sign-off stay with you. It is the service behind outsourced QA at ZeuZ.

Key takeaways

  • Tests are versioned with the release branch, so a run for a release branch tests that branch's own cases.

  • The Test Plan picks regression scope from changed files, scored by impact, change risk, reach, and coverage.

  • Every failure names the commit that broke it.

  • Repairs are proposed as a diff your team approves.

  • If you want the function run for you, our managed QA runs all six steps while the suite and sign-off stay with you.

Frequently asked questions

What is regression testing in agile?

In agile, regression testing runs every sprint, or on every merge, rather than once before a big release. Because each sprint changes working software, the team reruns its regression checks to confirm that earlier stories still work. That frequency is why agile teams automate the suite, keep a fast smoke set for each pull request, and run the full set on a schedule.

Book a demo to see regression sets run from a CI webhook →

How do you maintain 5000+ regression test scenarios?

At that size, select instead of running everything on every change. Tag tests by feature area, choose scope from what changed, and keep a full run on a schedule. Track flaky tests with an owner, review each repair as a diff, and retire duplicate scenarios every quarter. Reusable steps matter most, because one UI change should mean one fix, not hundreds.

Talk to the team about taming a large regression suite →

Can regression testing be fully automated?

Most regression checks can be automated, but not all of them, and not the judgment around them. Stable, repeatable flows automate well. Exploratory checks, brand-new features, and visual judgment usually stay with people. Someone also has to decide what to test, triage failures, and approve repairs, so plan for people to own the suite even when machines run it.

Book a demo to see which steps Test Studio automates →

How often should you run automated regression tests?

Run a small smoke set on every pull request and the full regression set at least nightly, plus once before each release. If the full set takes too long to run nightly, split it across machines or select tests by what changed, but keep a regular full run, because selection methods miss some dependencies.

Download the solution brief to see scheduled and CI-triggered runs →

What is the difference between regression testing and retesting?

Retesting, which ISTQB calls confirmation testing, checks that a specific fixed defect is really fixed. Regression testing checks that the change did not break anything else, in areas nobody meant to touch. Both follow a change, so teams often run them together: the retest for the fix, and the regression set for everything around it.

Book a demo to see failures linked to the commit that caused them →

How do you deal with flaky regression tests?

Treat a flaky test as a defect in the suite, not as noise. Turn on retries so flaky results are reported separately from real failures, fail the run when a test is flaky, and quarantine each one with an owner and a fix-by date. Common causes are fixed waits, shared test data, and fragile selectors, so check those first.

Talk to the team about stabilizing a flaky suite →


Related Blogs