One suite, four shards, and everything it left on disk

A browser suite that fails in CI at three in the morning gives you one line of text and a stack trace taken after the page had moved on. You reproduce it locally, it passes, and you move on too. I have done that more times than I would like to admit.

Everything I needed to stop doing that shipped in Motus 1.0.15, so I built a suite around it. The one rule I gave myself was that a run has to leave enough on disk to answer what happened without rerunning anything. Here's what that looks like.

What I pointed it at

Two public pages the Motus samples already drive: https://example.com and https://www.iana.org/domains/reserved. Neither of them belongs to me, and that turned out to be the useful part of the exercise.

When you test a page you control, you can be sloppy about the difference between a finding that should stop a build and a finding you only want written down, because you can always go fix the page. When the page is somebody else's, you have to decide up front. That decision shows up in almost every section below.

Getting the project set up

Start with the tool and a browser.

dotnet tool install --global Motus.Cli
motus install

motus install downloads a Chrome for Testing build into ~/.motus/browsers. After that, the project is ordinary MSTest.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.11.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="3.5.2" />
    <PackageReference Include="MSTest.TestFramework" Version="3.5.2" />
    <PackageReference Include="Motus" Version="1.0.15" />
    <PackageReference Include="Motus.Testing.MSTest" Version="1.0.15" />
    <PackageReference Include="Motus.Analyzers" Version="1.0.15" />
  </ItemGroup>
</Project>

Motus.Analyzers is optional and I would take it every time. It's seven Roslyn diagnostics that catch an unawaited call, an undisposed browser and a navigation with no wait after it, at compile time instead of at three in the morning. One heads-up when you go looking for it: an unrelated motion-planning library shares the name on nuget.org, so use the package page link and you'll land on the right one.

Usings.cs:

global using Microsoft.VisualStudio.TestTools.UnitTesting;
global using Motus.Abstractions;
global using Motus.Assertions;
global using Motus.Testing.MSTest;

AssemblySetup.cs launches one browser for the whole assembly. Each test still gets its own context and page, created and closed by the base class.

namespace PublicPages.Tests;

[TestClass]
public class AssemblySetup
{
    [AssemblyInitialize]
    public static async Task InitializeAsync(TestContext _) =>
        await MotusTestBase.LaunchBrowserAsync(new LaunchOptions { Headless = true });

    [AssemblyCleanup]
    public static async Task CleanupAsync() =>
        await MotusTestBase.CloseBrowserAsync();
}

You'll notice the accessibility, performance and coverage options are not set anywhere in that code. That's on purpose. Leaving them unset is what lets the config file and the MOTUS_* variables fill them in. Configuration layers file, then environment, then code, so whatever you hardcode wins and whatever you leave alone stays steerable from CI.

{
  "motus": "1.0",
  "launch": { "headless": true, "timeout": 30000 },
  "context": { "viewport": { "width": 1280, "height": 720 } },
  "assertions": { "timeout": 10000 },
  "accessibility": {
    "auditAfterNavigation": true,
    "includeWarnings": true,
    "skipRules": ["a11y-missing-landmark"]
  },
  "flaky": {
    "retryPolicy": "flake",
    "retries": 2,
    "historyPath": "artifacts/flake-history.json"
  }
}

I spelled out includeWarnings even though its default is already true, because that default is the one people trip over: a warning, not just an error, is enough to fail an audit assertion. Both of these pages trip a11y-missing-landmark, so the file skips it for the hook and the assertion below skips it for itself.

The tests

Four tests on the example domain, four on the reserved-domains page, and one that exists only to capture evidence.

namespace PublicPages.Tests;

[TestClass]
[PerformanceBudget(Lcp = 4000, Fcp = 3000, Ttfb = 2000, Cls = 0.25)]
public class ExampleDomainTests : MotusTestBase
{
    private const string Url = "https://example.com";

    private async Task OpenAsync()
    {
        await Page.GotoAsync(Url);
        await Page.WaitForLoadStateAsync(LoadState.Load);
    }

    [TestMethod]
    public async Task Page_HasItsTitleAndHeading()
    {
        await OpenAsync();

        await Expect.That(Page).ToHaveTitleAsync("Example Domain");
        await Expect.That(Page.GetByRole("heading", "Example Domain")).ToBeVisibleAsync();
    }

    [TestMethod]
    public async Task Page_LinksToTheIanaRecord()
    {
        await OpenAsync();

        var link = Page.GetByRole("link", "Learn more");
        await Expect.That(link).ToBeAttachedAsync();
        await Expect.That(link).ToHaveAttributeAsync("href", "https://iana.org/domains/example");
    }

    [TestMethod]
    public async Task Page_MeetsItsBudget()
    {
        await OpenAsync();

        await Expect.That(Page).ToMeetPerformanceBudgetAsync();
    }

    [TestMethod]
    public async Task Page_PassesTheRulesWeGateOn()
    {
        await OpenAsync();

        await Expect.That(Page).ToPassAccessibilityAuditAsync(
            o => o.SkipRules("a11y-missing-landmark"));
    }
}

The ToBeAttachedAsync on the link is not decoration, and if you take one thing from this post I would like it to be this one. A locator assertion polls for state once the element is attached. It does not wait for one to appear. So ToBeVisibleAsync on a selector matching nothing fails almost at once rather than after its timeout. Fast failures are usually good news, but not that one. ToBeAttachedAsync, ToBeDetachedAsync and ToHaveCountAsync are the three that are safe before the element exists, so one of them goes first.

The second class covers the response, the content, a network failure and a test I do not trust.

namespace PublicPages.Tests;

[TestClass]
public class ReservedDomainsTests : MotusTestBase
{
    private const string Url = "https://www.iana.org/domains/reserved";

    private async Task<IResponse?> OpenAsync()
    {
        var response = await Page.GotoAsync(Url);
        await Page.WaitForLoadStateAsync(LoadState.Load);
        return response;
    }

    [TestMethod]
    public async Task Page_RespondsWithTwoHundred()
    {
        var response = await OpenAsync();

        Assert.IsNotNull(response);
        await Expect.That(response).ToHaveStatusAsync(200);
    }

    [TestMethod]
    public async Task Page_HasAnExampleDomainsSection()
    {
        await OpenAsync();

        await Expect.That(Page.GetByRole("heading", "Example domains")).ToBeVisibleAsync();
    }

    [TestMethod]
    public async Task Page_RendersItsHeadingWithoutItsStylesheet()
    {
        await Page.RouteAsync("**/*.css", route => route.AbortAsync());

        await OpenAsync();

        await Expect.That(Page.GetByRole("heading", "IANA-managed Reserved Domains"))
            .ToBeVisibleAsync();
    }

    [TestMethod]
    [Quarantine(Reason = "Section headings follow IANA policy, not our code")]
    public async Task Page_ListsTheTestIdnTopLevelDomains()
    {
        await OpenAsync();

        await Expect.That(Page.GetByRole("heading", "Test IDN top-level domains"))
            .ToBeVisibleAsync();
    }
}

RouteAsync with AbortAsync is the cheapest way I know to test a failure you cannot otherwise arrange. I want to know the heading still renders when the stylesheet never shows up, and there is no way for me to make that happen on a server I do not run.

The last one is where [Quarantine] earns itself. That test's subject is somebody else's editorial decision, so it will change when IANA decides it should and not when I do. Quarantine keeps the test rather than deleting it: it runs, and it reports, in its own bucket, without gating the run. If you would rather not put the attribute in the source, a --quarantine list file does the same job from outside.

What actually gates the build

Two of the gates are the assertions you just read. The rest divide the same way all the way through: the flag decides whether a collector runs at all, and the file or the attribute supplies the numbers.

--perf-budget sets no numbers of its own, and that catches people out. With no attribute and no performance section there is nothing to enforce, and ToMeetPerformanceBudgetAsync throws rather than passing on an empty budget. I would rather it throw. A green test that measured nothing is worse than a red one.

Be clear-eyed about the accessibility side too. The nine rules are Motus's own, read from the browser's accessibility tree over the protocol, and they are not an axe-core replacement. Both gates are covered in more detail in Accessibility and performance are assertions.

Running it locally:

dotnet build -c Release
motus run bin/Release/net8.0/PublicPages.Tests.dll \
  --a11y warn \
  --perf-budget \
  --coverage console \
  --reporter console \
  --reporter html:artifacts/report.html

--a11y warn audits after every navigation and prints what it finds without failing anything, which is the right setting for pages you do not own. On this suite it reports the missing main landmark on both pages, and a missing document language attribute on the reserved-domains page. Neither one gates the build. They are real findings on markup I did not write, and I would rather see them and decide for myself than have a build tell me what to do about them.

What one test leaves behind

This is the test I am a little proud of. It records the same short scenario three ways and stops each recorder in a finally, so the artifacts exist on disk whether the assertion held or not. That second part matters more than the recording does. The run you want to look at afterward is the one that failed.

namespace PublicPages.Tests;

[TestClass]
public class EvidenceTests : MotusTestBase
{
    [TestMethod]
    public async Task FollowingTheLink_LeavesATraceAHarAndAVideo()
    {
        Directory.CreateDirectory("artifacts");

        await Context.Tracing.StartAsync(new TracingStartOptions
        {
            Screenshots = true,
            Snapshots = true,
        });
        await Page.StartHarRecordingAsync();
        await Page.StartVideoRecordingAsync("artifacts/follow-the-link.avi");

        try
        {
            await Page.GotoAsync("https://example.com");
            await Page.WaitForLoadStateAsync(LoadState.Load);

            await Page.GetByRole("link", "Learn more").ClickAsync();

            await Expect.That(Page).ToHaveTitleAsync("Example Domains");
        }
        finally
        {
            await Page.StopVideoRecordingAsync();
            await Page.StopHarRecordingAsync("artifacts/follow-the-link.har");
            await Context.Tracing.StopAsync(new TracingStopOptions
            {
                Path = "artifacts/follow-the-link.zip",
            });
        }
    }
}

Three details there are easy to get wrong, so here they are together. Tracing is on the context and browser-wide underneath, so concurrent starts serialize behind a gate, and one tracing test per suite is the comfortable number. HAR recording is per page, and it is the stop call that takes the path. Video is MJPEG frames in an AVI container, at viewport size, with no pointer drawn and no transcoding, so reach for ffmpeg when you need another format.

Open it with the CLI:

motus trace show artifacts/follow-the-link.zip --port 5200

That starts the visual runner in trace mode and opens your default browser. The archive is Motus's own layout, and this is its viewer.

That redirect is the reason I record all three instead of picking one. The click reads as a single action in the trace. The video shows a page that changes. Only the HAR shows the href returning a 301 before the browser reached the page the assertion checks. Three views of the same short scenario, and each one knows something the other two do not.

Coverage

Coverage is per test. It's collected over the protocol when a page closes, and remapped through source maps when a script has one.

motus run bin/Release/net8.0/PublicPages.Tests.dll \
  --coverage console \
  --coverage html:artifacts/coverage

The console reporter prints a per-file table and an overall line. The HTML reporter writes index.html plus a page per file. Thresholds are deliberately not a command-line option: they live in coverage.js.lines and coverage.css.rules, and a run below one exits non-zero.

This suite sets neither, and I want to be honest about why. The code being measured is somebody else's, so a threshold here would be a number about their scripts, not about my tests. On your own app it's the other way around, and that's where I'd set one.

Splitting it four ways

motus run --shard <index>/<total> sorts the discovered tests by assembly path and fully qualified name, then deals them out round-robin. That's the whole algorithm, and it works because every agent computes the same partition without talking to any other one. The sort makes it reproducible across machines. The round-robin stops one slow class landing on one agent.

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - run: |
          motus run bin/Release/net8.0/PublicPages.Tests.dll \
            --shard ${{ matrix.shard }}/4 \
            --a11y warn --perf-budget \
            --retries 2 --retry-policy flake \
            --flaky-history artifacts/flake-history.json \
            --reporter junit:results.shard-${{ matrix.shard }}.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: artifacts-${{ matrix.shard }}
          path: |
            results.shard-*.xml
            artifacts/

  merge:
    needs: test
    if: always()
    steps:
      - uses: actions/download-artifact@v4
      - run: motus shard merge **/results.shard-*.xml --output junit:results.xml --expect 4

fail-fast: false lets every shard finish after one of them fails, and if: always() runs the merge even when a shard job failed. Those two go together. The run where a shard blew up is exactly the run where you want the combined report.

--expect 4 guards the failure that sharding introduces, and it is the flag I would not leave off. Each shard stamps motus.shard.index and motus.shard.total into its result file. The merge reads them back and fails when one is missing or when an index turns up twice. Without it, an agent that died before writing anything produces a smaller report that is entirely green and looks fine, and you will believe it, because there is nothing in it to disbelieve.

Two limits are worth knowing before you add agents. Shards balance by count, not by duration, so a suite dominated by one slow test gains nothing from more of them. And the index is 1-based, so --shard 0/4 is rejected rather than guessed at.

Flaky, quarantined, or broken

Live pages over a shared network are a classic source of a failure that means nothing at all. --retry-policy decides which failures earn another attempt, and the two policies are further apart than the names suggest.

transient is the default and re-runs only a lost browser, which cannot mask a bug: a browser that vanished never told you anything about your code in the first place. flake re-runs any failure, assertion failures included, and labels a test that then passes as flaky rather than green. That label is the half that matters. Every attempt rebuilds the test instance and its context, and data from a failed attempt is discarded.

The console reporter shows you the outcome by name:

  [PASS] PublicPages.Tests.ExampleDomainTests.Page_HasItsTitleAndHeading
  [FLAKY] PublicPages.Tests.ExampleDomainTests.Page_MeetsItsBudget (after 2 attempts)
  [QUARANTINE] PublicPages.Tests.ReservedDomainsTests.Page_ListsTheTestIdnTopLevelDomains

Results: 8 passed, 0 failed, 1 flaky, 1 quarantined, 9 total

That run exits zero. --fail-on-flaky makes it exit non-zero, and it is what stops a suite drifting into needing retries everywhere while still reporting success. I would not switch it on the first day, though. Let --flaky-history accumulate per-test counters first:

{
  "PublicPages.Tests.ExampleDomainTests.Page_MeetsItsBudget": {
    "runs": 128,
    "failures": 3,
    "flakyPasses": 17,
    "lastSeenUtc": "2026-08-30T14:02:11Z"
  }
}

A flake rate is (failures + flakyPasses) / runs, and that's the number to sort by when you're deciding what to fix first. Persist the file between runs, as a CI cache or a committed artifact. If you don't, every run starts from nothing and you have a file that only ever describes the last few minutes.

Pinning the browser

A run that passed last week and fails this week with no commit behind it is usually a browser that moved. --channel takes whatever is installed. Pinning replaces that with an exact binary:

motus install --channel chromium --revision 149.0.7827.156
export MOTUS_EXECUTABLE_PATH="$(cat ~/.motus/browsers/.installed.chromium)"

--revision takes a Chrome for Testing version in dotted form. The marker file the installer writes holds the resolved executable path, so the install step can export it and nothing later in the job has to go hunting for where it landed. A path set in code still wins over the variable, which is the same layering from earlier showing up again.

Where this one falls short

It cannot run on Firefox as written, and I would rather say that here than have you find out from a failing job. Tracing, RouteAsync interception and code coverage all go through Chrome DevTools Protocol domains that a Firefox session driven over WebDriver BiDi does not have. The capability guard says so by name rather than failing obscurely, which is the best I can do about it right now. Locators, actions, assertions and script evaluation all work there. Those three do not.

Every artifact, and what it answers

None of that is a separate tool, a second browser session or another package. It is one suite, one browser, and a run you can ask what it saw.

Every flag and key above is documented at motustesting.com/docs, nearest the sharding, flaky tests and CLI pages. Motus is MIT licensed and the source is on GitHub. Build something like this, kill a shard on purpose, and tell me whether the merged report told you what happened.