Performance · Benchmarks, traces & load tests

Catch benchmark, trace, and load-test regressions before merge — not in production.

Production APMs tell you that latency spiked after customers are already affected and the developer has moved on to another task. Running benchmarks or k6 in CI without automated comparison just leaves JSON in build artifacts that nobody reads.

Codebase Pulse turns BenchmarkDotNet, Criterion.rs, OpenTelemetry traces, and load-test reports (Grafana k6, NBomber, Goose, and oha) into a pre-production regression gate. It diffs every benchmark method, trace operation, and load-test scenario against branch baselines — flagging slower hot paths, allocation spikes, N+1 calls, and p95 or error-rate drift directly on the pull request.

  • Catch regressions pre-deploy

    Spot allocation spikes, a slower checkout scenario, or an accidental N+1 database query on the PR while the code is still in review.

  • Method, span & scenario granularity

    Never just “performance degraded” — see the exact benchmark method, OTel operation, or k6/NBomber scenario that shifted.

  • Not another production APM

    Zero production agents or per-host fees. Pulse compares deterministic CI benchmarks, integration test traces, and load-test summaries.

PR-first workflow

Run and Guard: CI output and pull requests.

Whether you run micro-benchmarks, collect OpenTelemetry spans during integration tests, or export k6/NBomber load-test reports, Pulse summarizes the top regressions in the PR comment. Each perf step is a separate pipeline job — enable --guard on pull requests after baselines look right.

  • Matches benchmarks by full identity (namespace, type, method, and parameter set).
  • Ingests k6 summary JSON, NBomber JSON/HTML, Goose report JSON, and oha JSON — scenarios, thresholds, checks, p95 latency, throughput, and error rate vs baseline.
  • Groups OTel spans into operations and compares duration, errors, and external call volume vs baseline.
  • On pull requests with --guard, pulse-agent perf bdn, perf otel, perf k6, perf nbomber, perf goose, or perf oha fails when configured suite- or scenario-level regressions are detected.

Rollout without friction

Start passively or pass --warn-only to post the PR report with exit code 0. Enable --guard when your team is ready to block merges on regressions.

pulse-agent summary · Catalog.Benchmarks · Guard

✖ Benchmark Guard Check: FAILED
Engine: BenchmarkDotNet | Execution Time: 14.45s | Benchmarks: 5 | Total Mean Time: 3.53 µs | Allocated: 513.3 KiB | Status: Failed
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Regression detected: 2 of 5 benchmark(s) regressed in execution time or memory allocation beyond +5.0%. Total benchmark mean time increased by +26.8% (+746.32 ns), exceeding threshold of +5.0%. Total allocated memory increased by +38325.2% (+512.0 KiB), exceeding threshold of +5.0%.│
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌──────────────────┬───────────┬──────────┬────────────────────────┐
│ Metric           │   Value   │ Baseline │         Delta          │
├──────────────────┼───────────┼──────────┼────────────────────────┤
│ Total Mean Time  │  3.53 µs  │ 2.79 µs  │  +746.32 ns (+26.8%)   │
│ Allocated Memory │ 513.3 KiB │ 1.3 KiB  │ +512.0 KiB (+38325.2%) │
│ Total Benchmarks │     5     │    —     │           —            │
│ Regressed        │     2     │    —     │           —            │
│ Improved         │     1     │    —     │           —            │
└──────────────────┴───────────┴──────────┴────────────────────────┘
Top Regressed Benchmarks (up to 10)
┌─────────────────────────────────────┬────────┬───────────┬──────────┬────────┬───────────┬──────────────────────────────────────────────┐
│ Benchmark                           │ Driver │ Mean Time │ Baseline │ Time Δ │  Alloc Δ  │ Highlights                                   │
├─────────────────────────────────────┼────────┼───────────┼──────────┼────────┼───────────┼──────────────────────────────────────────────┤
│ PriceMatrixSerializer.SerializeJson │ Both   │  3.34 µs  │ 2.59 µs  │ +28.9% │ +52428.8% │ 1000.0 B→513.0 KiB/op; P95 +29%; median +29% │
│ StringUtilsBenchmarks.SplitOnce     │ Time   │ 16.47 ns  │ 14.71 ns │ +11.9% │   0.0%    │ P95 +12%; median +12%                        │
└─────────────────────────────────────┴────────┴───────────┴──────────┴────────┴───────────┴──────────────────────────────────────────────┘
Top Improved Benchmarks (up to 10)
┌────────────────────────────────────────┬────────┬───────────┬──────────┬────────┬─────────┬────────────┐
│ Benchmark                              │ Driver │ Mean Time │ Baseline │ Time Δ │ Alloc Δ │ Highlights │
├────────────────────────────────────────┼────────┼───────────┼──────────┼────────┼─────────┼────────────┤
│ StringUtilsBenchmarks.FormatBytesSmall │ Time   │ 61.61 ns  │ 66.30 ns │ -7.1%  │  0.0%   │ —          │
└────────────────────────────────────────┴────────┴───────────┴──────────┴────────┴─────────┴────────────┘
Compare distributions & span trees in Codebase Pulse.

Under the hood

How the performance engine works.

Performance: checkout p95 +28% (k6); 2 benchmarks regressed; OTel INSERT N+1 (+11 DB calls).

  1. 01

    BenchmarkDotNet & Criterion item-level deltas

    Pulse parses BenchmarkDotNet JSON reports (mean, median, P95, std-dev, GC gen0/1/2, allocated bytes) and Rust Criterion estimates.json (nanosecond point estimates and slope). Every method+parameter combination is diffed against baseline.

  2. 02

    OpenTelemetry operation & N+1 detection

    Wrap integration tests with pulse-agent perf otel. Pulse captures OTLP traces and compares operations on mean duration, total duration, error rate, and external call count — highlighting when a loop added extra SQL or HTTP calls.

  3. 03

    Load-test baselines (k6, NBomber, Goose, oha)

    Import k6 summary JSON (perf k6), NBomber JSON/HTML (perf nbomber), Goose report JSON (perf goose), or oha JSON output (perf oha). Pulse rolls up suite KPIs and per-scenario latency, throughput, HTTP failures, k6 checks, and threshold outcomes — diffed against baseline like benchmarks.

  4. 04

    Runner calibration & outlier awareness

    Process durations are calibrated with the same CPU single-core baseline used in Build and Test. When a shared runner is noisy, Pulse warns you instead of polluting your benchmark or load-test history.

Engineering questions answered

What your team can answer in 10 seconds

  • “Did this refactor quietly allocate more memory on our hot path?”

    BenchmarkDotNet GC and allocation metrics are compared per operation. Guard can fail a PR even when wall time looks similar but allocated bytes spiked.

  • “Can we gate pull requests on k6 p95 or failed checks?”

    Yes. After baseline load-test runs on your branch, enable --guard on pulse-agent perf k6 (or perf nbomber). Guard fails on suite-level latency/error regressions and on threshold or check failures surfaced in the summary.

  • “How do we catch N+1 queries before they hit staging or production?”

    OpenTelemetry operations are compared run-over-run on call volume and external calls. A sudden jump in calls per operation exposes N+1-style regressions during CI tests.

  • “Can we track Rust Criterion benchmarks alongside .NET services?”

    Yes. pulse-agent perf criterion ingests Criterion estimates.json (and related sample output) into the same baseline and PR reporting model as BenchmarkDotNet.

  • “Do we need to replace Datadog, Grafana, or Application Insights?”

    No. Production APMs watch live traffic after deploy. Codebase Pulse sits in CI to stop regressions from merging in the first place.

Forensic web UI

When the CI output or PR headline is not enough.

Most runs end with CI output or a PR comment. When you need to investigate a structural bottleneck, audit a release branch, or compare historical runs, open the full Performance views in Codebase Pulse.

01 · Benchmark Diff Table

Every method and parameter set compared against baseline

Sort and filter all benchmarks in the run by time regression or allocation growth. Inspect statistical distribution metrics (mean, median, standard deviation, outliers) to be sure a change is real.

  • Side-by-side current vs baseline vs previous run values
  • GC Gen0 / Gen1 / Gen2 collections and allocated bytes per operation
  • Supports both BenchmarkDotNet (.NET) and Criterion (Rust)

Zero-code on the runner

Wrap the commands you already run.

No SDK in your application. Pulse enables and reads the artifacts your toolchain already emits and uploads metadata to the product backend.

.NET

Artifacts: BenchmarkDotNet JSON, OTel traces, k6 summary JSON, NBomber JSON/HTML, Goose & oha JSON

.NET

$pulse-agent perf bdn -f "**/BenchmarkDotNet.Artifacts/results/*-report-full*.json" -- dotnet run -c Release$pulse-agent perf otel -- dotnet test$pulse-agent perf k6 -f "./k6-summary.json" -- k6 run script.js --summary-export=./k6-summary.json$pulse-agent perf nbomber -f "./report.json" -f "./report.html" -- dotnet run --project LoadTests$pulse-agent summary --vcs-token ${GITHUB_TOKEN}

Rust

Artifacts: Criterion estimates.json, OTel traces, Goose report JSON, oha JSON

Rust

$pulse-agent perf criterion -f "**/target/criterion/**/new/estimates.json" -- cargo bench$pulse-agent perf otel -- cargo test$pulse-agent perf goose -f "./report.json" -- cargo run --release -- --report-file report.json$pulse-agent perf oha -f "./checkout-api.json" -- oha -n 10k -c 50 --output-format json -o checkout-api.json https://localhost:8080/$pulse-agent summary --vcs-token ${GITHUB_TOKEN}