ZeuZ

Autonomous Software Testing: What the AI Agent Should Decide

Key Takeaways

  • Test Studio's agent shows its plan while it works and records where it clicked.
  • New cases land on an isolated branch with the plan and changed files ready for review.
  • Test design produces a prioritized Test Plan before any test is written.
  • Nothing repairs itself silently. A person approves every diff.
  • Our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one.
Blog

Share with your community!

Autonomous Software Testing: What the AI Agent Should Decide

Autonomous software testing is testing in which an AI agent plans, writes, runs, and repairs tests with little step-by-step direction, exploring the application, choosing what to check, and acting on the results, while people set the goals and approve any decision that changes what the product is expected to do.

That last part is where most teams get stuck. An agent that can do every step can also make every decision, including ones that hide a real bug. This guide sorts the decisions in testing into three groups: what the agent can decide alone, what it should propose for a person to approve, and what stays with people. For a hands-on setup with one agent, see our guide on how to use Claude Code for QA automation.

We build Test Studio, an AI test automation platform with its own agent, so we have a view on where the lines sit. To ground that view, we installed Playwright's test agents on Playwright 1.63 and read the instructions they ship with.

What is autonomous testing, and how is it different from test automation?

Autonomous testing differs from test automation in who makes the decisions. In classic test automation, people decide what to test and write every step, and the tool only runs it. In autonomous testing, an agent makes many of those decisions itself.

Approach

Who decides what to test

Who writes the steps

What happens when a test fails

Scripted test automation

People

People, in code or a recorder

A person investigates and fixes it

AI-assisted test automation

People

AI drafts, people edit

AI suggests a cause, a person fixes it

Autonomous software testing

The agent, within goals people set

The agent

The agent investigates and proposes or applies a fix

Scripted, AI-assisted and autonomous testing compared by who decides what to test and how to handle failures

The jump from the second row to the third is not only about speed. It moves judgment from people to the agent. That is useful for routine judgment, such as finding a renamed button, and risky for judgment that defines correct behavior, such as what a discount should total.

Which decisions does an autonomous testing agent make?

An autonomous testing agent makes decisions in every step of the testing work, not only in writing code. Listing them is the first move toward deciding which ones to hand over.

  • What to test. Which pages, flows, and states to explore, and which risks matter most.

  • What counts as correct. The expected values and messages each test checks against.

  • How to find each element. Which locator to use for each button, field, and message.

  • When and where to run. Which tests run on which change, browser, and environment.

  • What a failure means. Whether a failure is a real bug, a broken test, or an environmental problem.

  • What to do about it. Whether to repair the test, retry it, skip it, or report a defect.

  • What goes into the suite. Whether a new or changed test is merged into the shared suite.

Some of these are mechanical. Others decide whether a bug reaches your users. The next two sections split them.

What should the AI agent decide on its own?

An AI agent can safely decide anything that does not change what the suite treats as correct, and anything a person can check quickly afterward. These decisions are frequent, mechanical, and cheap to undo.

  • Exploring the application. Crawling pages and flows to build a map of what exists.

  • Drafting a test plan. Listing candidate scenarios ranked by risk, as a draft for review.

  • Choosing locators. Picking stable ways to find elements, such as role and visible name.

  • Running tests. Choosing run order, parallel workers, and which tests to run first on a change.

  • Collecting evidence. Saving screenshots, DOM snapshots, console logs, and network requests from every run.

  • Grouping failures. Clustering failures that share a cause, so a person triages one issue instead of twenty.

  • Writing a first diagnosis. Stating the likely cause of a failure, with the evidence attached.

Each of these saves hours, and none of them can turn a failing product into a passing suite.

What should always need a person's approval?

Any decision that can make a real bug look like a pass should need a person's approval. These decisions are rarer, and each one changes what the suite promises about the product.

  • Changing an expected value. If the total used to be $90 and is now $100, only someone who knows the pricing rules can say which is right.

  • Skipping or quarantining a test. A skipped test cannot fail, so it cannot catch the bug it was written for.

  • Deleting a test. Removing coverage is a product decision, not a repair.

  • Merging new or changed tests. New cases and repairs should land on a branch and be reviewed like any other code.

  • Changing shared test data or environments. An agent that edits shared data can break other teams' runs.

  • Calling a failure a bug or an intended change. The agent can suggest which it is. A person who knows the requirement should confirm it.

The release decision stays with people. An agent can report what passed, what failed, and what it changed, but whether to ship is a business call.

How should decisions split across each step of the testing work?

The table below applies those two lists to each step, from test design to test maintenance and optimization. Use it as a starting policy and adjust it to your risk.

Testing decisions split into what the agent decides alone, what it proposes for approval, and what people decide

Step

Agent decides

Agent proposes, person approves

Person decides

01 Test design

Explores the app and maps flows

A test plan ranked by risk

Which risks and requirements matter most

02 Test case creation

Drafts cases from the plan

Expected results for each case

What correct behavior is

03 Test automation

Locators, waits, and code structure

New tests, merged through review

Coding standards for the suite

04 Test execution

Run order, parallel runs, and selection on each change

Changes to shared environments or data

Which environments are in scope

05 QA reporting and analysis

Evidence capture, grouping, and first diagnosis

Bug or intended change, for each failure

The release decision

06 Test maintenance and optimization

Detecting broken and duplicate tests

Repairs, skips, and deletions, as a diff

Coverage you are willing to give up

The middle column is where autonomy pays off. The agent does the work, and a person spends a minute approving or rejecting it instead of an hour doing it.

What do autonomous testing tools do by default?

Autonomous testing tools differ most in the defaults they ship with, so read the instructions behind any agent before you connect it to your suite. Playwright's open-source test agents are a useful example, because their instructions are plain text in your repository.

Playwright ships three test agents: a planner that explores the app and writes a Markdown test plan, a generator that turns the plan into test files, and a healer that runs the suite and repairs failing tests. We ran npx playwright init-agents --loop=claude on Playwright 1.63, which added the three agent definitions, a seed test, and an MCP configuration to the project.

The healer's instructions show why defaults matter. Among other things, they tell the agent to:

  • Edit test code, including "fixing assertions and expected values."

  • If it is confident the test is correct but the error persists, "mark this test as test.fixme() so that it is skipped during the execution," with a comment explaining what happened.

  • "Do not ask user questions," and "do the most reasonable thing possible to pass the test."

Those are sensible defaults for an agent working alone on a developer's machine. They are risky defaults for an agent with merge rights on a shared suite, because two of them (changing expected values and skipping tests) are the decisions from the approval list above. Because the definitions are plain files, you can edit them, for example to stop at a proposed diff instead of applying it. Playwright's docs say the definitions should be regenerated whenever Playwright is updated, so recheck your edits after each upgrade.

Commercial platforms, from Tricentis Tosca to newer AI-first tools, make the same choices inside the product. Ask each vendor which decisions its agent takes alone and where a person approves.

How does AI test case generation work, and where does it go wrong?

AI test case generation works in two ways: the agent explores the running application and writes tests for what it finds, or it reads requirements and writes tests for what should happen. The two fail differently.

Source

What it does well

Where it goes wrong

Exploring the running app

Finds pages and flows quickly, with working locators

Writes down current behavior, including current bugs, as the expected result

Reading requirements or stories

Tests what the product is supposed to do

Misses flows the requirements never mention, and may guess at unclear rules

Both, with a person reviewing

Covers what exists and what should exist

Takes review time, which is where the real value sits

The first row is the trap. A test generated from a buggy page will pass on the bug and fail on the fix. Generated tests also tend to cover the happy path and miss permissions, unusual data, and error states, so ask the agent for those cases by name.

How do you set guardrails for an autonomous testing agent?

Set guardrails by controlling where the agent can write, what it can change without review, and what record it leaves behind.

  • Work on an isolated branch. New and repaired tests land on a branch and reach the shared suite only through review.

  • Protect expected values. Treat any change to an assertion's expected value as a change that needs approval, even inside a larger repair.

  • Block silent skips. Fail the build, or flag the pull request, when a test is newly skipped or marked to fix later.

  • Show the plan and the trail. Require the agent to state its plan before acting and to record what it clicked and changed.

  • Limit environments. Give the agent its own test environment and data, not shared staging accounts.

  • Roll out in stages. Widen autonomy only when the numbers support it.

A staged rollout might look like this:

Staged rollout of an autonomous testing agent from observing to proposing to applying low-risk changes, with guardrails

Stage

What the agent may do

Move on when

1. Observe

Run tests, collect evidence, and diagnose failures

Its diagnoses match your triage most of the time

2. Propose

Draft new cases and repairs as diffs for review

Most proposed diffs are approved with few edits

3. Apply low-risk changes

Apply locator and wait repairs on a branch, with review before merge

No approved change has hidden a real bug

4. Wider autonomy

Generate and run new cases for new features, still reviewed before merge

Escaped defects stay flat or fall

Most teams find stage 2 or 3 is where the value is. Expected values, skips, and deletions stay behind approval at every stage.

How do you measure whether autonomous testing is working?

Measure autonomous testing by what reaches your users and by how much review time it saves, not by how many tests the agent wrote.

  • Approval rate. The share of proposed tests and repairs your team accepts without edits.

  • Hidden-bug catches. Proposed changes rejected in review because they would have hidden a real defect. Track these closely.

  • Skipped and deleted tests. A rising count is a warning sign, whoever made the change.

  • Time to triage. The time from a failed run to a confirmed cause.

  • Escaped defects. Bugs found in production that the suite should have caught.

If your product itself includes an AI feature, the agent's own output needs testing too. Checking non-deterministic answers needs different methods, which our guide on how to test generative AI applications covers.

When do autonomous testing services make sense?

Autonomous testing services make sense when you want the speed of an agent but lack the people to set its policy, review its diffs, and own the results. An agent does not remove that work. It moves it from writing tests to reviewing them.

Teams in that position usually choose between hiring for the review role, bringing in engineers who already run an agent day to day, or handing the testing function to a provider. The right choice depends on how hard your environment is and how much of the decision-making you want to keep in-house.

How does Test Studio decide, and what does it leave to you?

Test Studio, our AI test automation platform, is built on the split in this guide: the agent does the work, and people approve every decision that changes what counts as correct.

  • Design comes first. Test design produces a prioritized Test Plan before any test is written, scoring changed files by impact, change risk, reach, and coverage.

  • Changes arrive for review. New cases land on an isolated branch with the plan and changed files ready for review.

  • You can see what it did. Test Studio's agent shows its plan while it works and records where it clicked, and you can take over the browser at any point.

  • No silent repairs. Nothing repairs itself silently. A person approves every diff.

  • Readable tests. Tests are readable Python generated from plain-English instructions, so a reviewer can check them without learning a new format.

  • Price. Test Studio is $220 per seat per month, flat, with no metering on test runs.

See Test Studio's agent plan, write, and propose a repair, with every change waiting for your approval. Book a demo to try it on your app, or download the solution brief for the detail.

Which option is right for your team?

Which option is right for your team depends on who will set the agent's policy and review its work:

  • You have engineers who will own the agent. Use the decision table and staged rollout above, and start at stage 1 with your own environment and data.

  • You have the mandate but not the bandwidth, or a hard environment. Our forward deployed engineers work inside your team, each carrying a Test Studio license, and are measured on a baseline agreed in writing in week one. Talk to the team.

  • You want to change how the whole organization delivers. For an org-wide rebuild, our AI SDLC transformation maps where the hours go, rebuilds the steps that delay the release, and transfers the capability back to your team.

Frequently asked questions

What can autonomous testing not do?

It cannot decide what your product is supposed to do. An agent can explore, write, run, and repair tests, but it learns expected behavior from the app or the requirements it is given. If the app has a bug, or a rule is unclear, the agent may treat the wrong result as correct. People still own requirements, risk, and the release decision.

Talk to the team about where an agent fits in your QA →

How is autonomous testing different from codeless test automation?

Codeless test automation changes how tests are written: people build them with recorders or plain language instead of code. Autonomous testing changes who decides: an agent chooses what to test, writes the tests, and acts on failures. A tool can be both, for example an agent that writes readable tests from plain-English goals and asks for approval.

Book a demo to see tests written from plain-English instructions →

Can an AI agent write test cases from requirements?

Yes, if the requirements are clear. An agent can turn user stories and acceptance criteria into test cases with steps and expected results. It works best when a person reviews the expected results, because unclear or missing rules lead the agent to guess. Ask it to cover permissions, error states, and unusual data, not only the happy path.

Download the solution brief to see how Test Studio plans tests →

How do you know an AI-generated test is correct?

Review it like code from a new team member. Check that each expected value comes from a requirement, not from what the page shows today. Run it against a known bug or an older build to confirm it fails when it should. A test that has never failed has not yet proved it can catch anything.

Talk to the team about reviewing AI-generated tests →

Should an AI agent be allowed to skip a failing test?

Not without approval. A skipped test cannot fail, so it stops protecting the feature it covers. An agent can propose a skip with its reasons and evidence, and a person should decide. Track newly skipped tests in every pull request, so a skip never reaches the shared suite unnoticed.

Book a demo to see every repair arrive as a diff for review →

Where should a team start with autonomous testing?

Start with the agent observing, not changing. Let it run your existing suite, collect evidence, and diagnose failures in its own environment, then compare its diagnoses with your team's. Once those match, let it propose new tests and repairs as diffs. Widen its rights only when review shows its changes are not hiding bugs.

Talk to the team about a staged rollout →

Related Blogs