corpus #

The corpus is the set of files the libraries are run on. It lives in bench/corpus/, and its manifest records what each input is and where it came from. Every input is listed at the end of this page, with its upstream commit where it has one.

Every input is labeled by its source, in its path and in every result:

The neutral set has a small, a medium, and a large file for each timed language. The other sources are kept so a reader can see whether a result holds on inputs a library chose for itself, or on hostile ones, and the results page filters by source. No summary pools two sources. A home input is always shown under the library whose samples it is, and libraries' home inputs are never summarized together: they contributed very different amounts.

The stress inputs are generated by a script in the repository, bench/corpus_stress.ts, and none is tuned to a library. A library that fails on one, or is far slower there than anywhere else, is a finding: it is shown, as an exclusion with its reason or as the slow number, never smoothed over by changing the input. They come in two kinds:

• derived — a neutral input as real code ships: the JS and the CSS minified by esbuild, and the JSON serialized with no whitespace, each one long line. It keeps its source's pins, is marked synthetic, and names how it was made, with the esbuild version, since another version may minify to other bytes
• written — a pathological file for each timed language: nesting far deeper than code is written, a line of TypeScript tens of KB long, an unterminated string and block comment near the top of a file, template literals nested in their substitutions, Markdown delimiters that never close, HTML tags that never close and attributes without quotes, and a selector list on one long line. Each says in its first comment what it stresses

Shiki is timed with its per-line time limit out of reach, as on every input, so on a stress input it highlights every line in full. With its default limit a line that slow comes back with its rest as plain text, so a consumer with default options gets that fallback there rather than the stress number.

An input's tier comes from its size in bytes, never from its name:

The upper three center on roughly 1KB, 10KB, and 100KB. The micro tier is the docs-site case: a snippet of a few lines, where the fixed cost of a call outweighs the scanning. The large tier has no upper bound, and the neutral large inputs range from about 90KB to several hundred KB, so compare throughput in MB/s across languages, not time per call.

An input is marked synthetic when it is not one real file as published but was built by joining or repeating files to reach its size, or derived from one, as the minified stress inputs are. Real files are preferred. Most of Twinkleplop's own corpus is built this way and is marked.

One neutral input is synthetic: the large Svelte file. Svelte components of around 100KB are rare, so a script builds one from several real components of other projects. It splits each component into its scripts, markup, and style, and writes one of each block holding the bodies in order, unchanged. The result is shaped like one large component and is lexically well formed, but it is not a working component: unrelated scripts share one scope, so their names collide and Svelte's parser rejects it. No source is rewritten to avoid that. The manifest lists every component it is built from.

bench/corpus/manifest.json holds one BenchInput for each input: its path, source, language, tier, size in bytes and lines, SHA-256, whether it is synthetic, and its provenance. A results file carries the same entries for the inputs it measured, described in results file.

Provenance is a list of BenchProvenance pins, each a repository, a commit, a path, a hash, and a license:

• none, for a harness snippet or a stress input written here
• one, for a verbatim copy, whose hash must equal the input's; the copy is still marked synthetic when its upstream built it by joining files
• one, for a derived stress input, whose derivation names the transformation that made it from that file
• several, for a synthetic input, naming what it was built from

The manifest also holds a hash over the whole corpus. A run records it, so two runs over different inputs are never compared as one.

Sizes, line counts, hashes, and tiers are derived from the files. The synthetic flag, the provenance, and the derivation are written by hand, or by the stress generator for its own inputs, and carried forward. The tests rebuild the manifest from disk and compare it to the committed one, so a file added, removed, or changed without regenerating fails. The tests also regenerate the stress inputs and compare them with the files.

The corpus is data: no formatter, linter, or type checker runs on it, since a reformatted file is no longer the upstream file.

Every input of the results this site renders, by source. The corpus hash of those results is e649d70d435efac06e26c753d9424bbf30122fcae8e8037396b3cc340ea1c101. The repository holds the same list as a table generated from the manifest.

Every copied file stays under its upstream license, which its provenance records. The corpus readme and the notice beside the neutral inputs give the attribution.