Benchmark

This is an independently maintained fork of @dapplion/benchmark. This repo now maintains it's own versioning as @chainsafe/benchmark and release schedule. It was forked from the base of @dapplion/[email protected] and new functionality was added.

Ensures that new code does not introduce performance regressions with CI. Tracks:

• Do PR against the base branch include a performance regression?
• Do new commits in the main branch include a performance regression?

This tooling provides both a easy to use runner for benchmarking and easy integrations to persist past benchmark data.

Quick start

Create a benchmark test file but use itBench instead of it

import {describe, itBench, setBenchOpts} from "@chainsafe/benchmark";

describe("Sum array benchmark", () => {
  itBench("sum array with reduce", () => {
    arr.reduce((total, curr) => total + curr, 0);
  });
});

Then run the CLI.

benchmark 'test/perf/**/*.perf.ts' --local

Inspect benchmark results in the terminal

  Sum array benchmark
    ✔ sum array with reduce                                               826.0701 ops/s    1.210551 ms/op   x0.993        578 runs   1.21 s

How does it work?

This tool is a CLI tool, example usage:

benchmark 'test/perf/**/*.perf.ts' --s3

The above command will:

• Read benchmark history from the specified provider (AWS S3)
• Figure out the prev benchmark based on your option (defaults to latest commit in main branch)
• Run benchmark comparing with previous
  • Collect benchmark data in-memory while streaming results
• Add result to benchmark history and persist them to the specified provider (AWS S3)
• If in CI, post a PR or commit comment with an expandable summary of the benchmark results comparision
• If a performance regression was detected, exit 1

Track performance in CI

Below is a suggested Github action to run this tool with s3 history provider:

name: Benchmark
# Ensure a single benchmark is run at a time
concurrency: cd-benchmark-${{ github.ref }}

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  s3:
    if: always()
    runs-on: ubuntu-latest
    # Ensure both don't run at the same time
    needs:
      - local
      # - ga-cache

    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2-beta
      - run: yarn install --frozen-lockfile

      # Run benchmark with custom tooling and stores the output to a file
      - name: Run performance tests
        run: yarn benchmark --s3
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          # S3 credentials
          S3_ACCESS_KEY: ${{ secrets.S3_ACCESS_KEY }}
          S3_SECRET_KEY: ${{ secrets.S3_SECRET_KEY }}
          S3_REGION: ${{ secrets.S3_REGION }}
          S3_BUCKET: ${{ secrets.S3_BUCKET }}
          S3_ENDPOINT: ${{ secrets.S3_ENDPOINT }}
          # Key prefix to separate benchmark data from multiple repositories
          S3_KEY_PREFIX: ${{ github.repository }}/${{ runner.os }}

Track performance locally

When working on optimizing a function you may want to know if your code is actually faster than the previous implementation.

To do that you can keep a benchmark history file locally and run the benchmark first against previous code

git checkout master
benchmark test/perf/func.perf.ts --local

• Runs benchmark without comparing with previous
• Writes single benchmark data to ./benchmark_data

Then measure performance with the new code

git checkout fix1
benchmark test/perf/func.perf.ts --local

• Reads single benchmark data from ./benchmark_data
• Run benchmark comparing with prev
• Does not write benchmark data

Isolate benchmark files (Node.js)

Use --isolate to run each file sequentially in a fresh process instead of the default shared process:

benchmark 'test/perf/**/*.perf.js' --isolate --historyLocal ./benchmark_data-isolated

Each worker inherits the Node executable, runtime flags (including loaders and --expose-gc), environment, and working directory. Setup files run in each worker. The next file starts only after the previous worker exits, releasing its retained memory and native resources. Completed workers exit even if benchmark code leaves open handles.

The parent still owns history, the combined snapshot, performance comparison, and the single GitHub report. Collection failures, failed benchmarks, crashes, missing worker results, interruption, and duplicate IDs across files abort before persistence, including with --noThrow. Skipped files are allowed, but the complete run must produce at least one result.

Files must be independent: globals and hooks are not shared across files, and .only selection is local to each file. Use unique benchmark IDs across files. When switching modes, use a fresh local history path, CI cache key, or S3 prefix; shared-process and isolated measurements are not interchangeable baselines. History namespaces are not changed automatically.

Config

--isolate

Run each file sequentially in a fresh process

• type: boolean
• default: false

--defaultBranch

Provide the default branch of this repository to prevent fetching from Github

• type: string
• default:

--persistBranches

Choose what branches to persist benchmark data

• type: array
• default: default-branch

--benchmarksPerBranch

Limit number of benchmarks persisted per branch

• type: number
• default: Infinity

--threshold

Ratio of new average time per run vs previos time per run to consider a failure. Set to 'Infinity' to disable it.

• type: number
• default: 2

--compareBranch

Compare new benchmark data against the latest available benchmark in this branch

• type: string
• default: default-branch

--compareCommit

Compare new benchmark data against the benchmark data associated with a specific commit

• type: string
• default:

--prune

When persisting history, delete benchmark data associated with commits that are no longer in the current git history

• type: boolean
• default:

--persist

Force persisting benchmark data in history

• type: boolean
• default:

--noThrow

Exit cleanly even if a preformance regression was found

• type: boolean
• default:

--historyLocal, --local

Persist benchmark history locally. May specify just a boolean to use a default path, or provide a path

• type: string
• default: ./benchmark_data

--historyGaCache, --ga-cache

Persist benchmark history in Github Actions cache. Requires Github authentication. May specify just a boolean to use a default cache key or provide a custom key

• type: string
• default: benchmark_data

--historyS3, --s3

Persist benchmark history in an Amazon S3 bucket. Requires Github authentication

• type: string
• default:

Roadmap

Compare performance

bench compare --from <branch-name | commit-hash> --to <branch-name | commit-hash>

• Retrieves benchmark history
• Don't run benchmark
• Print comparision of from benchmark with to benchmark

Add more providers

• Github actions cache: Doesn't work due to Github's limitations
• Commited in the repository (gh-pages, or data branch)