Test · TRX, nextest & JUnit

Separate flaky tests from broken code and from a starved CI runner.

Nothing erodes trust in CI faster than intermittent red builds. Developers click “Re-run failed jobs” three times, waste 40 minutes waiting for runners, and eventually start ignoring test failures altogether.

Codebase Pulse ingests VSTest TRX, cargo nextest / libtest JSON, and JUnit XML alongside live runner hardware telemetry. It separates real code regressions from same-commit flaky tests and from infrastructure outages — posting a clean verdict on the pull request.

  • End blind CI reruns

    Know immediately if a test failed because of your commit, a known flaky race condition, or a starved CI VM.

  • Protect clean baselines

    Mass failures (DB container crash) and noisy runs skip candidate and baseline updates so outages do not flood the flaky list.

  • Actionable root-cause hints

    See whether a flaky test fails alongside its class (shared fixture) or, when timing data exists, during parallel execution overlap (race hint).

PR-first workflow

Run and Guard: CI output and pull requests.

When a test step finishes, Pulse posts the pass rate, normalized duration delta, and a categorized breakdown of failures. Developers see right on the PR whether they broke a test or hit an existing flaky candidate.

  • Cross-commit transitions (Passed → Failed on a new SHA) are flagged as New Failures, not flaky.
  • Same-commit flips and pass-on-retry runs are classified as Flaky with a confidence score.
  • Resolved flaky or slow tests are automatically evicted after enough consecutive clean runs.
  • On pull requests with --guard, the job fails when any test fails, the runner exits non-zero, or total suite duration regresses vs baseline; classification explains why, not whether the gate fires.

Rollout without friction

Start passively or pass --warn-only to post the PR report with exit code 0. Enable --guard when your team is ready to block merges on regressions.

pulse-agent summary · AcmePlatform · Guard

✖ Test Guard Check: FAILED
Framework: xUnit | Job: AcmePlatform
Passed: 616/617 | Pass Rate: 99.8% (baseline 100.0%, -0.2 pts) | Duration: 68.40s (baseline 64.00s, +6.9%, gate +20.0%)
┌──────────────────────────────────────────────────────────────────────────────┐
│ Regression detected: 1 of 617 test(s) failed; test process exited with code 1│
└──────────────────────────────────────────────────────────────────────────────┘
┌─────────┬───────┐
│ Metric  │ Count │
├─────────┼───────┤
│ Total   │  617  │
│ Passed  │  616  │
│ Failed  │   1   │
│ Skipped │   0   │
│ Flaky   │   1   │
│ Slow    │   2   │
└─────────┴───────┘
Failed tests
┌──────────────────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────┐
│ Test                                                                             │ Error                                          │
├──────────────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────┤
│ AcmePlatform.Backend.Payments.Tests.WebhookDispatcherTests › Should_Sign_Payload │ Expected HMAC signature to match payload bytes │
└──────────────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────────────┘
Slow tests
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─────────────────────────────┐
│ Test                                                                                                                                                                                                                 │ Duration                    │
├──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼─────────────────────────────┤
│ AcmePlatform.Agent.Application.Tests.Features.Run.SystemMetricsMonitorTests › AcmePlatform.Agent.Application.Tests.Features.Run.SystemMetricsMonitorTests.MonitorSession_ShouldAccumulateSnapshotsAndProduceValidDTO │ 12.19s (+22.0% vs baseline) │
│ AcmePlatform.Agent.Infra.Tests.Services.AgentEnvironmentDetectorTests › AcmePlatform.Agent.Infra.Tests.Services.AgentEnvironmentDetectorTests.Detect_ShouldReturnHardwareMetricsAndSignature                         │ 5.69s (+22.0% vs baseline)  │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴─────────────────────────────┘
Flaky tests
┌──────────────────────────────────────────────────────────────────────────────────────┬────────────┐
│ Test                                                                                 │ Confidence │
├──────────────────────────────────────────────────────────────────────────────────────┼────────────┤
│ AcmePlatform.Backend.Payments.Tests.StripeRetryTests › Concurrent_Charge_Idempotency │    80%     │
└──────────────────────────────────────────────────────────────────────────────────────┴────────────┘
Full timeline, attempts, and history in CodebasePulse.

Under the hood

How the tests engine works.

Tests: PR guard failed — 1 failing test, 1 in-run flaky, duration +6.9% vs baseline (within +20% gate).

  1. 01

    3-layer environment & outage protection

    First, NoiseDetector flags runs with CPU saturation, disk I/O surges vs hardware baseline, RAM pressure, or VM core mismatches — often together with a duration spike. Second, IsMassFailure detects suite-wide infrastructure crashes (many failures at once when a shared dependency dies). Third, selector-change detection spots when the effective test filter changed between runs.

  2. 02

    Commit-aware flaky vs bug classification

    Pulse tracks commit SHAs across a sliding window of recent runs. A test that flips between pass and fail on the same commit or passes on retry is scored as Flaky. A test that breaks on a new commit and stays red is classified as a real bug (NewFailure).

  3. 03

    Forensic hints: fixtures vs race conditions

    For every flaky candidate, FlakyDiagnosticsCalculator computes co-failed counts in the same test class/group (shared fixtures or DB state), concurrent timeline overlap when start/end timestamps exist (race or contention hints), and normalized failure signatures.

Engineering questions answered

What your team can answer in 10 seconds

  • “Did my PR break this test, or is it just flaky?”

    Pulse checks whether the test flipped on the same commit SHA or passed on retry versus transitioning from green to red across commits.

  • “Why did 300 integration tests fail at once when Redis timed out?”

    Mass-failure protection recognizes infrastructure outages and prevents hundreds of tests from polluting your flaky candidate list or duration baselines.

  • “How do we actually fix a flaky test once Pulse finds it?”

    Inspect whether the test fails alone under high parallel overlap (race condition / resource contention) or always fails together with sibling tests in the same fixture.

  • “Do we have to manually clean up old flaky tests after we fix them?”

    No. After enough consecutive clean passes — fewer for a fixed failure, more for a long-lived flaky candidate — Pulse evicts it automatically.

Forensic web UI

When the CI output or PR headline is not enough.

Most runs end with CI output or a PR comment. When you need to investigate a structural bottleneck, audit a release branch, or compare historical runs, open the full Tests views in Codebase Pulse.

01 · Flaky Diagnostics

Confidence scores, commit flips, and root-cause indicators

Prioritize the flaky tests that hurt the team most. Each candidate shows its flip rate, whether it passed on retry, how many distinct error signatures it produces (stripped of random GUIDs and timestamps), and whether it correlates with parallel execution.

  • Flaky confidence score from clean-run history
  • Shared-fixture indicator (co-failed tests in the same group)
  • Concurrency overlap when TRX or other reports include per-test timestamps

Zero-code on the runner

Wrap the commands you already run.

No SDK in your application. Pulse enables and reads the artifacts your toolchain already emits and uploads metadata to the product backend.

.NET

Artifacts: VSTest TRX (auto-enabled or imported via -f)

.NET

# Pulse automatically injects --logger:trx$pulse-agent test -- dotnet test$pulse-agent summary --vcs-token ${GITHUB_TOKEN}

Rust

Artifacts: cargo nextest JSON (auto-enabled); cargo test libtest JSON on nightly or with -Z flags

Rust

# nextest: libtest JSON enabled automatically$pulse-agent test -- cargo nextest run$pulse-agent summary --vcs-token ${GITHUB_TOKEN}