User Guide

How it works

The one idea that explains every other behaviour in the tool.

You describe steps the way you'd tell a careful human tester. The agent sees a screenshot and the current URL — never your DOM, selectors or page source. As it works, the tool watches where it clicked and derives a stable Playwright locator from the element that was there. That locator is an output of the run, used later by Replay; it never influences a decision.

The seven tabs

Where everything lives.

Signing in

Protect AgentQA itself with a login gate.

AgentQA itself can require a login, configured in appsettings.json under an AppAuth section. Off by default.

Fields:

• Enabled (true/false)
• SessionHours (default 12)
• Users — a list of { Username, Password, DisplayName }

Passwords are plaintext, which is why appsettings.json is gitignored. appsettings.example.json is the committed template.

If Enabled is true but no users are configured, sign-in is disabled and a warning is logged, rather than locking everyone out.

Signed-in user's name and a sign-out button appear at the bottom of the sidebar.

Getting started

Project, variables, page, test — in that order.

1. Create a project

Test Builder → Project selector (sidebar) → +

2. Add variables

Test Builder → Project tab → Variables. One KEY=value per line. Use them in steps as {{KEY}}.

TEST_USER=qa@example.com
TEST_PASS=your-password

Secrets never reach git. A variable whose name contains PASS, SECRET, TOKEN or KEY commits as a declaration with no value; the value lives in DataStorage/variables.local.json on your machine. Values are masked as •••••• in reports, model reasoning, judge evidence and error text.

Built-ins, re-expanded every run so they stay unique across replays: {{RANDOM_EMAIL}}, {{RANDOM_NAME}}, {{RANDOM_PHONE}}, {{TIMESTAMP}}.

3. Add a page

Test Builder → Pages → +

4. Write a test

Test Builder → pick a page → New test. Each row is one step: Action or Assert.

• One objective per step. Click the 'Log In' button, not Enter credentials and submit.
• Name what's on screen, exactly as printed. The model sees pixels, not your HTML.
• Always end with an assertion. An action-only test passes as long as the clicks happen — it stays green while the app is broken.
• Assertions are claims, not commands. The dashboard heading is visible, not Check the dashboard.
• Negative tests need two assertions — that the error showed and that the user wasn't also let in.

5. Import from Azure DevOps CSV

Test Builder → pick a page → Import CSV.

Accepts an Azure DevOps test case export. Columns are matched by name, so column order doesn't matter and extra columns (Area Path, Assigned To, State, Work Item Type) are ignored.

Required columns: Title, Test Step, Step Action, Step Expected.

A preview shows what was found before anything is written, with per-test tick boxes and expandable steps. Tests whose name already exists on that page are flagged and unticked by default, so re-importing doesn't silently duplicate.

Imported tests arrive unrecorded and need one Discover run.

Sessions & login

How a test reaches a page that requires signing in.

Every scenario gets a fresh browser context: no cookies, no session, no local storage. So a test cannot rely on a previous one having logged in.

Set the login up once in Project tab → Shared sign-in: switch it on, give it the login path, and write the login steps. They're ordinary steps — recorded and replayed like any other.

Why signed-out is the default: "an anonymous visitor is redirected away from /account" only means something in a browser that has never signed in.

When to use own login: a test that signs out, changes the password, or waits for a session to expire can invalidate the shared session server-side. The shared session renews at most once per run, so such a test poisons the ones after it — and the failures land on innocent tests.

Setup & teardown

A fresh context resets cookies for free. It does not reset your server.

Edit test → Add setup. The teardown editor is currently hidden because no test in the suite used it; if your committed test files already contain teardown steps, they still run.

setup     1. Click 'Empty cart'
          2. [assert] The cart shows 0 items
steps     1. Click 'Add to Cart' on the Wireless Headphones card
          2. [assert] The cart counter shows 1
teardown  1. Click 'Empty cart'

Copy steps from another test

Edit test → Add setup → "Copy steps from another test." Pick any test in the project, tick the steps you want, and they're appended to setup. Action steps are pre-ticked; assertions aren't, since setup exists to reach a state rather than re-verify what the source test already checks.

Run another test first

Edit test → Add setup → "Run another test first." You can make a test reference another test's steps instead of copying them. A reference stores a link to the source test, not a frozen copy. When the login flow gains a field, every test that referenced it automatically gets the new field, whereas every test that copied it must be edited one by one.

Tick tests in the picker. They run in the order you tick them — shown as a numbered chip list you can reorder with up/down arrows.

Only the actions run. Assertions in the referenced tests are skipped, because setup exists to reach a state; asserting inside one reports a prerequisite's failure against a claim this test never made. Verifying the login flow is the login test's job. It also saves the judge call, which is the expensive part of a step.

Referenced tests run before the inline setup steps, so inline steps can build on the state they reach — e.g. "log in" (referenced) then "empty the cart" (inline).

Resolution is transitive: if a referenced test references a login itself, that login runs too. Deduplicated: if two references both reach the same test, it runs once, not twice. So referencing "log in" and "open dashboard" (which itself references "log in") logs in once.

A referenced test contributes its own setup steps plus its main steps, but not its teardown. You reference a test to reach the state it reaches; its teardown is how it undoes itself, so including it would create a record and immediately delete it again.

Cycles are rejected with the chain named, e.g. "Setup references form a loop: A → B → A. A test cannot be its own precondition." The test fails rather than running without its preconditions.

A deleted referenced test is skipped, not fatal — the rest of the run continues — and the editor shows a warning chip so a silently-skipped precondition is not mistaken for a passing test.

Borrowed steps arrive unrecorded and are recorded against this test on its first Discover run, same rule as copied steps: a locator captured while the source ran as its own test resolves against whatever screen this test is on.

In reports, a borrowed step is labelled with the test it came from, e.g. "setup 1 (from Login)", so a red row points at the test that owns the step.

Trade-off: referencing couples tests. Break the login test and every test referencing it goes red at once. That is still much better than an order-dependent suite — order is not load-bearing and each test still runs on its own — but it is real coupling, and the reason borrowed steps are labelled with their source in reports.

Grouping tests

Organize tests by feature for easier navigation and bulk operations.

Each test can belong to one named group — e.g. "Save", "Update", "Search". Set it in the test editor's Group field, or leave blank.

Blank is a real value meaning "not sorted yet"; those collect under an Ungrouped section, sorted last, never hidden.

Groups render as collapsible sections in both Test Builder and Live Runner, each with a test count and a Run group button.

Bulk assign: tick several tests in Test Builder and a bar appears with a group name box (autocompleting from existing groups), Assign and Ungroup. This is how you sort a large import.

Shared window

Opt-in, per page. Read the trade before switching it on.

Edit page → "Run this page's tests in one browser window." Normally every test gets a fresh browser and opens the page itself. With this on, the page's tests run top to bottom in one window, each carrying on from wherever the previous one left the screen.

Prefer setup steps. Copying an earlier test's steps into setup buys the same convenience with none of this. Use shared window only when re-reaching a state is genuinely impractical — a long wizard, a one-shot token, a flow the app won't let you restart.

The first test on a shared-window page still gets a clean context and navigates normally, so the chain has a defined starting point. Order follows the page file's list order, not names.

Running tests in parallel

Speed up execution with multiple concurrent browsers.

Settings → Run behaviour → Parallel browsers, 1–8, default 4. Each worker runs a scenario in its own browser, pulling from one shared queue.

A shared-window page always forces serial, since those tests must hand one browser between them.

The Browser pane shows a live tile per worker while running in parallel.

Sign-in still happens exactly once per run regardless of worker count.

Live Runner

Run tests, watch the browser, diagnose failures without leaving the tab.

The rail

Every test, grouped by page, with its state. Pages are collapsible with Collapse all / Expand all buttons. A page auto-expands while one of its tests is running.

Replay all covers every recorded test and skips unrecorded ones rather than quietly starting hours of discovery — it tells you how many it skipped. Next takes the first test with no recording.

Execution log

A step that didn't pass is clickable: expand it for the error, the agent's blocking question, judge evidence, the locator trace, its reasoning, and the screenshot at the point of failure. Failures expand automatically. Every genuinely running test highlights — with parallel workers several rows can be live at once.

Click a test name in the rail to see the steps from its last saved run without going to Reports. Click again, or Close, to return to the live log. A live run always takes priority over a selection. A pencil icon on every test jumps straight to that item in Test Builder.

Browser panel

Streams the headless browser at your configured fps. A Headless / Visible segmented control in the topbar (next to the Discover / Replay toggle) switches between them for the next run. It's disabled while a run is in progress since it only applies to the next run.

Reports

Every run saved with screenshots, locators and reasoning.

• History on the left — click any run. The History header has HTML, CSV, and Delete buttons. HTML and CSV download the WHOLE run history (not just one run). HTML omits screenshots deliberately — embedding them for every run would make the file enormous. CSV is one row per scenario result, which is the grain that makes a pivot table useful. There is also a /history.json endpoint. Delete enters selection mode with per-row checkboxes, Select all / Select none, a count chip, and a Delete button. A confirmation dialog states exactly what is removed, e.g. "This permanently deletes these 3 runs and their screenshots, videos and traces. Downloaded report files are not affected. This cannot be undone." The confirmation only names videos and traces when the selected runs actually have them. Deleting a run removes its screenshots, videos and traces from disk — they all live in one per-report folder. Already-downloaded HTML/JSON report files are unaffected. Reports are automatically pruned to retain only the 50 most recent runs; older ones have their artifacts deleted.
• Scoreboard — passed, failed, duration, model calls. 0 model calls means a pure replay.
• Failures only filter for long runs.
• Failing scenarios expand by default; passing ones stay folded.
• Open in a new tab, Download self-contained HTML with screenshots embedded, or JSON.

Settings

Model connection and run behaviour.

Model connection

Test connection verifies it before you spend a run finding out.

Run behaviour

Project settings

Performance testing

Run Lighthouse scans to measure Core Web Vitals and performance metrics.

What it is

The Performance tab runs Google Lighthouse — the same engine behind PageSpeed Insights — against a page or a custom URL, and reports the score alongside this tool's own network analysis. Pick a project page or type a URL, choose a device profile, press Run.

What it measures

A run is one clean, cold visit in a fresh, isolated browser context — no cache, no warm-up — so the numbers reflect a first-time visitor rather than a primed machine. From that single load it captures:

• Core Web Vitals — LCP (largest contentful paint, load speed), CLS (cumulative layout shift), and INP (interaction to next paint, responsiveness).
• Lab timings — FCP (first contentful paint), TTFB (time to first byte), Total Blocking Time, Speed Index, DOMContentLoaded, Load event, and Time to Interactive.
• A Lighthouse performance score — 0–100, from the same engine PageSpeed Insights uses, when Lighthouse is installed. When it is not, a clearly-labelled heuristic score that is not comparable with PageSpeed.
• Network weight — total request count and bytes transferred, broken down by JavaScript, CSS, images, fonts, API, and third-party.
• Real-user field data (CrUX) — roughly 28 days of real Chrome visitors (LCP, INP, CLS, FCP, TTFB) with a good / needs-improvement / poor verdict, shown only when the site has enough Chrome traffic. Otherwise the report clearly says there is no field data.

The score

When Lighthouse runs, the score is Lighthouse's own, so it is directly comparable with PageSpeed Insights for the same URL and device. The report labels it "Lighthouse <version> · <device> · comparable with PageSpeed Insights". If Lighthouse cannot run, the report falls back to this tool's own heuristic score and says so explicitly — that score is not comparable with PageSpeed, and the report states why Lighthouse was unavailable.

Device profiles — this is the single most common reason a score seems wrong

Mobile and desktop are scored on different curves, so the same page legitimately scores differently on each. That is why PageSpeed has two tabs. Comparing a desktop run here against a mobile PageSpeed result is a false mismatch. Profiles available: Desktop (no throttling), Mobile 4G (4G network + 4x CPU slowdown), Mobile Slow 3G (slow 3G + 4x CPU slowdown).

Iterations

Lighthouse is run N times (1–5, default 3) and the median run is reported, with the spread shown. One run is not a measurement: a single cold run varies by several points on a busy machine, which is the usual reason two runs of the same page disagree. Each iteration costs roughly 20–45 seconds.

Metrics

Lab metrics shown: FCP, LCP, Total Blocking Time, CLS, Speed Index, Time to Interactive, TTFB, DOM Ready, Load Event. Each is colour-coded against Google's thresholds.

What Lighthouse found

The report lists every audit the page did not pass, severity-ordered, with estimated savings in ms and KB — for example "Reduce unused JavaScript", "Use efficient cache lifetimes", "Legacy JavaScript". Roughly 50 audits run; only the failing ones are listed.

Network analysis

Kept as a separate section, because it is the part Lighthouse does not provide: per-request timings, a breakdown by resource type (JS, CSS, images, fonts, API), third-party weight, and slow or failed requests. Findings are labelled by which engine produced them.

Real-user data (CrUX)

Optional. With a CrUX API key set in Project settings, the report also shows the Chrome UX Report section PageSpeed displays above its lab results: the 75th percentile of real Chrome users over the trailing 28 days, and whether the page passes the Core Web Vitals assessment. This is a different kind of number from the lab metrics — 28 days of real users versus one controlled load on this machine — and the two routinely disagree without either being wrong. Most internal, pre-release or low-traffic URLs have no field data at all; the report says so rather than showing zeros.

Requirements

Lighthouse needs Node.js. On a developer machine it is resolved via npx automatically; the Docker image installs it into the image so no network call is needed at run time. If Node is missing, the Performance tab says so and disables the Lighthouse option rather than silently producing a heuristic score.

Troubleshooting

Design consistency

Crawl a site and report where its design contradicts itself — or where it departs from a design system you declare.

The case that motivated this: a login page that shows field validation errors inline, under the field, and shows the "wrong password" error as a toast that disappears after three seconds. Each is defensible on its own. Together they teach the user two different rules for the same thing, and no single-page test will ever say so, because neither page is broken. That exact case is what Message placement checks, and it is reported without any declaration at all.

The Consistency tab crawls a site, reopens each page, and reads the styles the browser actually resolved on the controls it finds. By default it compares the site against itself: most findings are counts of distinct values, not opinions, and it does not judge taste. The exception is where messages appear, which is compared across the run whether or not anything has been declared.

It does not know your design system unless you declare one. Declaring value scales replaces the counting for those properties with a check against the scale; declaring roles or message placements adds checks that have no counting equivalent. See Declaring a design system.

The four phases

Pages are deduplicated by kind, not URL. /order/1842 and /order/1843 collapse into one entry, /order/:id, and the map shows how many URLs it stands for — "×312 URLs" — so you can see they were represented rather than missed. Without this, one template with a lot of rows would eat the whole page budget.

Why the run stops at the checkpoint. A crawler cannot tell forty distinct pages from one template rendered forty times; a person looking at forty thumbnails can tell instantly. Unticking a page removes it from capture, so it contributes no elements and no findings.

Target

The tab asks what to crawl, the same way the Performance tab does:

A custom URL replaces both the base URL and the start path. Pasting https://preview.example.com/pricing audits that page on that host: keeping the project's start path would quietly audit a different page, and keeping the project's origin would audit the right path on the wrong site.

A URL typed without a scheme is read as https://. A URL that cannot be parsed falls back to the project rather than failing the run, and every run records the base URL it actually crawled — so a stored result never leaves you guessing which site it looked at. The crawl stays on the target's own origin, as it does for a project.

Sign-in

Three modes, in place of the Session select the tab used to have:

Configurations saved before this existed still work. A blank sign-in mode falls back to what the old Session field said: a config that asked for a signed-in session uses the project's steps, anything else crawls anonymously.

What it reports

Every finding lists the variants it counted, most common first, with the pages they came from. The most common variant is treated as the incumbent; the rest are marked as outliers. That is a frequency ordering, not a verdict about which one is right — the tool has no way to know which one is right until you tell it.

These four checks are what runs when nothing is declared, and they keep running for every property you have not declared a scale for. Placement is the one that can have an opinion without a scale: it does not need to know which place is correct to see that the answer changes from page to page.

Declaring a design system

The counting checks are a guess about what normal looks like. They cannot say which of nine corner radii is the correct one, and they cannot tell a deliberate exception from a mistake.

The Design system card on the Consistency tab is where a project says what its scales actually are. Turn on Check against a declared design system and fill in whatever you have settled:

Every scale is optional and independent. A property left empty keeps its counting heuristic. Declaring your radii and nothing else is a supported way to use this — you do not have to describe the whole system before you get value from part of it. With the master switch off, every check behaves exactly as it did before.

The same card holds two rule types that are not scales: Roles, which bind a name to the markup that wears it and the colours it must have, and Message placement, which says where each kind of message belongs.

What changes for a property once its scale is declared:

• The count is replaced, not added to. An off-scale value becomes a finding that names the nearest declared value: "6px is not on your scale; the nearest declared value is 4px."
• A page that conforms reports nothing. That is what makes filling the card in worth doing: it converts "six findings, all judgement calls" into "zero findings, and that is a fact".
• Severity reflects blast radius, not novelty — 3 or more distinct off-scale values, or 10 or more affected elements, is high. One stray value on one control is a slip; the same slip on thirty controls is a pattern that has already been copied.
• A declared verb is checked even when every label agrees with every other. A product whose buttons all say "Delete" is still wrong if the declared word is "Remove", and the counting check cannot see that case at all, because there is no disagreement to count.

The declared spec applies to both the standalone Consistency crawl and the ride-along audit a page opts into from Test Builder. A standard with two different answers is not a standard.

Values are normalised before they are compared, so you can write your palette the way you already have it written down:

A value the tool cannot parse is left exactly as written and simply fails to match. That is visible and correctable, which is better than silently matching the wrong thing.

Importing a design document

A team that has a design system has usually already written it down. Retyping forty values into a form is work with a typo rate attached, so the Design system card takes the file instead: .md, .markdown, .txt or .css, read up to 512 KB.

It understands three shapes, listed in the order they can be trusted:

Canonical verbs are read from prose as well, in two phrasings: use Remove, not delete and delete → Remove.

The two rule types that are not scales are read too, so a document can declare the whole system rather than only its numbers.

A row carrying a selector is read as a role, not as a scale. A Roles table with the columns Role, Selector and Background — | Primary action | .btn-primary | #005AC8 | — says which colour one named thing must have, which is a different claim from "this colour is allowed anywhere"; putting #005AC8 on the palette instead would lose it. The row is recognised by the shape of its cells rather than by the column headings, because a header row is optional in markdown and its wording is not something a document has to agree with.

Message placement is read from prose. "Errors belong inline, under the field they are about. Never as a toast." is imported as a declared inline placement for the error tone, exactly as if you had chosen it in Message placement. The line is read one sentence at a time, which is what makes that phrasing safe: the first sentence names a tone and a place, and the second is a negation with no tone in it, so it is ignored rather than read as a declaration of toast. A sentence needs both a tone and a placement word to count — "errors are red" names a tone and no place, and reading it as a rule would declare a constraint nobody wrote.

What it does with what it finds:

• A variable that points at another variable is skipped. --radius-alias: var(--radius-sm) carries no value of its own, and importing it would put the text var(--radius-sm) on a scale where a length belongs.
• Colours are decided by the value, not the name. --brand-500: #005AC8 is imported as a colour because #005AC8 is one. Most teams name a colour after the colour, not after the word "colour".
• Values are normalised on the way in, exactly as they are when you type them: hex and rgb() become the same value, and a bare 8 means 8px.
• Nothing is applied silently. The import shows a preview of everything it recognised, and separately what looked like a token but could not be classified. Apply merges with whatever is already in the fields rather than overwriting them, and a canonical verb you have already typed is never replaced. The same holds for the two rule types: a role whose name is already on screen is left alone, and a tone you have already pinned to a place is not overwritten. Importing the same document twice therefore changes nothing the second time, rather than doubling a scale.
• If nothing recognisable is found it says so and offers no Apply, rather than appearing to succeed with an empty result.

A DESIGN.md mixing all three shapes imported as 4 corner radii, 1 font size, 2 control heights, 2 text colours, and the verbs delete → Remove and create → New.

Roles

A scale says which values are allowed anywhere. A role says which value one named thing must have. Roles sit in the same Design system card, with the scales rather than after them, because that is the difference between a palette and a design system.

The selector is what makes a role checkable at all. A scale can be checked without knowing what anything is: every radius either is or is not on the list. A role cannot. "The primary button is #005AC8" is unverifiable until something identifies which buttons are primary, and no amount of looking at pixels will reveal it. A role with a blank selector is inert — it checks nothing — so the editor flags that row instead of letting it sit there looking like a rule.

What a role reports:

• Every element the selector matches is checked against whichever colours were declared. A mismatch names the expected value against the actual values found, with counts: declared #005AC8, found rgb(27, 97, 201) on 6 of 9 matching elements — the near miss you get when a colour was eyedropped off a screenshot rather than taken from the token.
• A selector that matches nothing anywhere the audit reached is itself reported, at medium severity. Silence would let a renamed class leave a rule looking enforced forever while checking nothing at all. The finding distinguishes the two causes: either the crawl never got to a page that uses the class, or the class was renamed and the rule has stopped checking anything.
• Hex and rgb() are the same value here, as everywhere else in the spec, and an element that does not express the property at all is skipped rather than counted as wrong.

Message placement

This is the check the whole feature was originally asked for, and it is the one no value-based check can perform. The login page from the top of this section shows "enter a password" inline under the field, and "wrong password" as a toast at the top that clears itself after three seconds. Both messages are the right colour, the right size, correctly worded. Every value-based check passes. What is wrong is that the product answers the question "where do I look to find out what went wrong" in two different places, and a user who learns the first answer misses the second.

The four placements:

Tones are error, success, warning and info. A message is detected three ways, because no single one is reliable: role="alert" or role="status", an aria-live region, or a telling class name — error, success, toast, notification and similar. The tone is read off the same markup. A message whose tone cannot be determined is grouped on its own rather than guessed into a bucket, because calling an error a success would invert the finding.

How placement is decided. Anything fixed or sticky is a toast wherever it sits, because being out of flow is what makes it one. Otherwise an element in the top fifth of the viewport is a banner, and anything below that is inline. It is judged as a fraction of the viewport rather than in pixels: the same y of 140 is a top banner on a laptop and mid-page on a phone.

There are two modes, and they do not need each other:

• Declared. Set a placement for a tone, and messages of that tone appearing anywhere else are reported, with the declared placement named against the places they were actually found. An optional selector narrows the rule to messages matching a class or id.
• Self-consistency. A tone appearing in more than one place across the run is reported, with no rules written at all. This is the Report a message tone that appears in more than one place checkbox, and it is on by default.

Two behaviours here are deliberate:

• A tone with an explicit rule is judged against that rule only, not additionally against itself. It has already been told where it belongs; adding "and they also disagree with each other" would be a second finding about one fact.
• Two different tones in two different places is not reported. An error inline and a success toast is a legitimate choice, not a contradiction. Only one tone landing in two places is.

Environment

You declare the environment; the tool never infers it from the URL.

Two independent layers

A blocklist decides what may be clicked. A read-only guard decides what a click is allowed to do. They are separate mechanisms, and neither is sufficient on its own.

The blocklist — what gets clicked. Four built-in categories, each switchable:

Payment matches three ways — by label, by CSS selector, and by path — because a button labelled "Continue to confirmation" contains no payment word but still walks into checkout. Path rules catch what wording misses.

Short ambiguous terms like pay, order, send and clear match whole word only, so "Display settings" and "Reorder columns" are still clicked rather than silently skipped. Longer, unambiguous terms match anywhere in the label.

The read-only guard — what a click may do. Enforced at the network layer, not by trusting the page:

Sub-resources — fonts, images, analytics — are still allowed off-origin, so the page renders the way it really renders. Only document navigations are held to the origin.

Custom rules

Add your own by label text, CSS selector, or path glob. Text rules have a whole-word option; with it off, the term also matches inside longer words. Selector rules match on tokens, so .btn does not also match .btn-primary. Path globs are anchored, with * staying inside one segment and ** crossing them.

What every run reports

The run records what it stopped, not just what it found:

• Mutating requests blocked
• Off-origin navigations refused, with the URLs
• Controls skipped, each with the rule that skipped it — e.g. text:pay (whole word) or path:/checkout/**

If any guard category was switched off, the run is stamped with that. A clean audit performed with the payment guard disabled must not be mistakable for a clean audit.

After a run finishes

The result is what the tab is for, so it is shown first. When a run finishes — or stops at the checkpoint for you to choose pages — the configuration gets out of the way: at the checkpoint it is hidden entirely, and on a finished run it collapses to a one-line summary of what produced the result — target, page budget, environment, sign-in mode, and whether a declared design system was checked against — with an Edit settings toggle and Run again beside it. The page scrolls to the result once, on that transition only, rather than on every render.

Previously the findings rendered below the configuration and you had to scroll past the form to reach them.

Where the configuration lives

Audit configuration — environment, depth, budget, target URL, sign-in mode, the audit's own sign-in steps, disabled categories, custom rules — belongs to the project, not to your local settings, so a loosened blocklist appears in a diff and gets reviewed like any other change. It is written to tests/<project>/project.json as a DesignAudit block, with the declared design system beside it as a DesignSpec block — so the scales a team builds to are diffable source, like the code that has to follow them. The audit's own sign-in steps live in that block too, deliberately apart from the project's shared sign-in. Completed runs are stored under DataStorage/audits.json with their screenshots.

Save on the Consistency tab writes both without starting a run. Starting an audit saves them too. Configuration used to be written only when a run began, so configuring now and running later lost the edits.

Headless, for CI

dotnet run -- --audit <url> [production|uat|dev] [--spec <file>]
dotnet run -- --audit https://staging.example.com/ uat
dotnet run -- --audit https://staging.example.com/ uat --spec design-system.json
dotnet run -- --audit https://staging.example.com/ uat --spec DESIGN.md

The same crawl without the UI. Exits 0 when the run completes, non-zero when it fails or is cancelled.

No operator is present, so there is nobody to prune the map at the checkpoint: it is auto-approved and the page budget decides what gets audited instead of a person. The output says so — "every mapped page was audited — no operator selected a subset" — so a headless result is never read as a curated one. The headless run always crawls anonymously and uses the default depth and budget.

--spec <file> declares the design system for a target that has no saved project. It takes JSON, using the same shape as the DesignSpec block in project.json:

{
  "Enabled": true,
  "Radii": ["0px", "4px", "8px"],
  "ControlHeights": ["32px", "40px"],
  "CanonicalVerbs": { "delete": "Remove" }
}

It also takes a markdown or CSS file, read by the same importer as Importing a design document. A team can declare their system once in DESIGN.md and use that file in CI, instead of maintaining a second JSON copy of the same scales — two copies of a standard is how a standard drifts. The run prints what it read, line by line, before the crawl starts.

This is the CI shape of the feature: the scales live in a file next to the code, the audit runs against a dev deployment on merge, and a value drifting off the scale is reported the same way a broken test is. Properties the file does not declare keep the counting heuristic, so a partial spec is fine — with the caveat above about half-declared scales.

The shop fixture, and proving the tab in a browser

Everything the audit computes is already covered by --selftest and by --audit, and neither of them can press a button. So the arithmetic was tested and the tab was not: the custom URL, the design-document upload, the Save button, the checkpoint, the results view and the audit's own sign-in steps had never once been exercised by a click. A path nobody has clicked is not a path that works, which is what this command is for.

dotnet run -- --uiproof [outDir]

It drives the Consistency tab through a real browser against the app you already have running on http://localhost:5078 — it does not start one, and it will fail immediately if nothing is there. Every step writes a screenshot into outDir, or into a temporary directory if you name none, so the result can be looked at and not only read. It prints N passed, M failed and exits non-zero on a failure, so it can sit on a gate beside --run replay. It is the sibling of --shots, --uilive and --selftest — the same idea one layer up, where what is under test is the interface rather than the wire contract.

What it clicks through:

• a project created through the New project button, then the custom-URL target replacing the start path with a URL field
• a design document uploaded, previewed, and applied — then imported a second time, to prove that the same document twice duplicates neither a scale nor a role
• Save, followed by a reload, because a form rebuilt from project.json is the only real test of persistence
• the crawl checkpoint: the page tiles and their screenshots, the selected count reacting to an unticked page, and Audit selected pages
• the findings view, including that findings render above the crawl settings rather than below them — read off the rendered geometry rather than off the CSS, because a flex order is exactly the kind of thing that regresses silently
• the custom sign-in step editor, a real signed-in run, and Cancel

The target is the app's own fixture, served by the app at /testbed/shop/, so no external site is involved and the findings are known in advance rather than being whatever the internet looked like that morning:

Because the app serves the fixture itself, the same seven findings can be had from the command line alone, with nothing to install and nothing to configure:

dotnet run -- --audit http://localhost:5078/testbed/shop/home.html --spec wwwroot/testbed/shop/DESIGN.md

That is the shortest honest demonstration of the feature: a design system declared in a document, a site contradicting it in seven specific places, and seven findings each naming exactly one element. It is also what the audit is developed against — a change that stops reporting one of the seven has broken something, and a change that reports an eighth has started guessing.

Security scanning

Run an autonomous penetration test with Strix and read the findings.

What it is

The Security tab runs Strix, an open-source autonomous penetration-testing tool, against a target and shows what it found. Strix does all the actual testing inside its own Docker sandbox — its agents drive an HTTP intercepting proxy, a headless browser and real exploit attempts. OpenAgentQA itself does no security testing: it starts Strix as a local subprocess, streams the log live into the tab as the scan runs, then parses Strix's result files and renders the findings in the app's normal report design — a hero verdict, stat tiles, severity chips, and a card per finding with impact, technical analysis, proof-of-concept, reproduction and remediation.

The point of a tool like this over a static scanner is that every finding carries a working proof-of-concept and reproduction steps. A static scanner flags patterns and produces false positives; Strix only reports a vulnerability it managed to demonstrate, so a finding is something you can reproduce rather than something to triage.

What it tests

Strix does dynamic testing: it does not read your source code — it attacks the running site the way an external attacker would, and only reports vulnerabilities it can actually demonstrate. Every finding carries a working proof-of-concept and reproduction steps, so a finding is an exploitable fact rather than a static-analysis guess. That is the key difference from a SAST tool or a linter. It probes the OWASP Top 10 classes:

• Injection — SQL injection, command injection, and XXE (XML external entity).
• Cross-site scripting (XSS) — reflected, stored, and DOM-based.
• Broken access control / IDOR — reaching data or actions that should be denied, for example by changing an id in a URL.
• Authentication and session weaknesses.
• CSRF (cross-site request forgery) and clickjacking.
• Business-logic flaws — abusing a legitimate flow, such as skipping a step or tampering with a value.

Each finding records a CWE identifier, a CVSS severity score, the endpoint and method, the exact proof-of-concept, and remediation steps. Strix produces these by driving a real headless browser, an intercepting HTTP proxy — to inspect, replay and tamper with requests — and a Python exploit sandbox, all inside its Docker container.

Running a scan

Pick a project or type a custom URL, then choose a scan depth — quick, standard or deep. Quick is the default and the fastest. Two settings are optional: max turns per agent bounds how long the run can go (0 means Strix's own default of 500), and a free-text instruction steers it, for example "Focus on IDOR and XSS".

A locally-served target — anything on localhost or 127.0.0.1 — is reached over host.docker.internal automatically, because the scan runs inside a Docker container and would otherwise not see your machine.

Authorization — this cannot be skipped

Prerequisites

The scan depends on tooling outside the app, and each piece must be in place before the first run:

• Docker Desktop, installed and running. Strix runs its sandbox inside Docker; the ~5.9 GB sandbox image is pulled once on the first scan.
• Strix, installed with pipx install strix-agent. It needs Python 3.12 or newer.
• A model. Strix uses the same model endpoint the app already uses, configured under Settings / config.json (BaseUrl, ApiKey). The model id is auto-discovered from the endpoint's /models if you have not set one. Any OpenAI-compatible endpoint works — LM Studio, llama.cpp, or a cloud provider. Strix is entirely tool-call driven, so the served model must emit native tool calls; a weak or mis-configured model may find little or stall.

When Docker or Strix is absent the scan fails with a clear message rather than crashing.

Where findings are stored

Past scans are listed in the tab and can be reopened. Each run is stored to DataStorage/security.json (the most recent 20 runs are kept), and the raw Strix output is left under DataStorage/security-runs/.

Headless, for CI

dotnet run -- --strix <url|projectId> [quick|standard|deep] --yes [--instruction "..."]

This runs a scan without the UI. --yes is the command-line form of the authorization checkbox and is required — without it the scan is refused, not prompted. The exit code follows Strix's own convention so a CI job can branch on it: 0 — no vulnerabilities, 2 — vulnerabilities found, 1 — error or cancelled.

Run speed

Why a step takes minutes, and what actually helps.

The loop is screenshot → ask the model → act → screenshot → ask again. After the browser finishes typing, the step isn't done — it screenshots again and asks "did that work?" That round trip is the wait you're watching. The typing itself is milliseconds.

A typical step is 2–3 rounds, so 1.5–4 minutes per step in Discover. Budget ~2 min/step and run a few tests at a time with ▶ rather than "Run all".

To go faster without losing accuracy

1. Replay, not Discover. Zero model calls, ~0.2s/step. This dwarfs every other option and is more deterministic, not less.
2. Turn on "Fresh context per step." ~27% faster at step 3 of 4, and the gap widens with scenario length. Each step is its own objective and still gets a recap of what previous steps achieved. Keeps all 3 screenshots.
3. Write steps that ground in 2 rounds, not 4. At ~83s a round, one wasted round costs more than any setting. The round count shows in the failure panel.
4. Shorter scenarios. Two 4-step tests beat one 8-step test, and they diagnose better.
5. Lower max rounds to 3–4. Correct steps terminate on their own; this only caps the damage on broken ones.

Self-healing

What happens when the app changes underneath a recorded test.

The first Discover run records a stable locator beside every click, most-stable-first: data-testid → id → ARIA role+name → text → CSS path.

If a locator stops resolving, Replay walks the fallback list. Only when all fail does it re-invoke the model for that one step, then saves the new locator. Renaming a data-testid typically costs zero model calls — the ARIA fallback catches it.

Healed steps are marked healed in reports, so a silent change shows up as a reviewable diff rather than a mystery.

Where tests live

Tests are source. Commit, review and diff them like code.

tests/                          ← commit this
  my-project/
    project.json                base URL, variable NAMES, shared sign-in steps
    pages/
      login-page.json           the page, its scenarios, and their recordings

DataStorage/                    ← gitignored
  variables.local.json          secret variable VALUES
  config.json                   endpoint, viewport, headless
  reports.json  audits.json  screenshots/  auth/

One file per page, so recording a test touches only that file and two people working on different pages never conflict.

Command line

The CI gate and the diagnostics.

dotnet run                              # start the app
dotnet run -- --run replay              # CI gate: exits non-zero if anything fails
dotnet run -- --run discover            # headless discovery
dotnet run -- --run replay <scenarioId> # one scenario
dotnet run -- --audit <url> [env]       # design consistency audit, no UI — see below
dotnet run -- --uiproof [outDir]        # click through the Consistency tab in a real browser
dotnet run -- --strix <url|projectId> [quick|standard|deep] --yes  # autonomous pentest, no UI — see Security scanning
dotnet run -- --selftest                # check the model wire contract
dotnet run -- --authtest                # check shared-session plumbing, no model needed

--uiproof needs the app already running on http://localhost:5078; every other command starts what it needs. See The shop fixture, and proving the tab in a browser.

From a clean checkout:

git clone <repo> && cd <repo>
export AGENTQA_TEST_PASS=...            # PowerShell: $env:AGENTQA_TEST_PASS = "..."
dotnet run -- --run replay

DataStorage is recreated with defaults on first run.

When a step fails

Reading the report.

Start with the step's screenshot and its Show reasoning toggle — in the Live Runner log or in Reports.

Changelog

Features and changes that affect what a user sees or does.