QEC decoder accuracy on fixed circuit-level memory experiments

Task

Decode the detection events of five fixed circuit-level quantum memory experiments. Each ranked instance is one code under one published noise model, run as two memory experiments of 100,000 shots each. QArena publishes the Stim circuits, their detector error models and every shot's detection events; the logical-observable flips of the ranked shots (the labels) stay private on the QArena server. You submit the flips you predict for every shot, and the score is the exact number of shots you get wrong.

• surface9-si1000-v1: rotated surface code [[81,1,9]], CZ circuits, SI1000 noise at p = 0.003; X then Z memory; 25 noisy rounds; D = 2,000 detectors and K = 1 observable per shot.
• bb72-bravyi-v1: bivariate bicycle code [[72,12,6]], uniform circuit noise at p = 0.003; X then Z memory; 6 noisy cycles; D = 504, K = 12 (all twelve logical qubits of the memory basis).
• bb144-bravyi-v1: bivariate bicycle (gross) code [[144,12,12]], uniform circuit noise at p = 0.004; X then Z memory; 12 cycles; D = 1,872, K = 12.
• color7-uniform-v1: triangular 6.6.6 color code [[37,1,7]] with superdense syndrome extraction, uniform circuit depolarization at p = 0.002; X then Z memory; 21 rounds; D = 756, K = 1.
• honeycomb8-em3-v1: periodic honeycomb Floquet code on 8 × 12 data qubits, EM3 noise (entangling measurements with correlated measurement and depolarizing faults) at p = 0.01; horizontal (H) then vertical (V) memory; 72 sub-rounds, that is 24 rounds of XYZ; D = 1,184, K = 1.

Each ranked instance has a development twin (the same name ending in -dev) with 10,000 fresh shots per experiment and public labels, for tuning and for checking a pipeline end to end. Development instances are debug instances: they have leaderboards but no lifetime limit.

Each instance is one fixed workload; there are no distance or noise sweeps. The gross code's code distance of 12 is not its circuit distance, and the honeycomb layout's distance 8 against single-qubit depolarization is not its EM3 circuit distance, which is 4.

Circuits and data

All circuits were generated once with Stim 1.16.0 under CPython 3.12 with hash randomization off, written as UTF-8 text, and frozen; the published files and their detector and observable order are authoritative, and nothing is regenerated during scoring. The Data tab lists every file with its size and SHA-256 (also recorded in each instance's JSON file).

• Surface: stim.Circuit.generated("surface_code:rotated_memory_x" (or _z), distance 9, 25 rounds), converted with gen.transpile_to_z_basis_interaction_circuit and made noisy once with gen.NoiseModel.si1000(0.003) from the chromobius repository at commit acd09feb (two-qubit depolarization p, idle and single-qubit p/10, reset flips 2p, measurement flips 5p, extra waiting noise during measurement and reset).
• Color: clorco's superdense_color_code_X / _Z construction from the same commit, diameter 7, 21 rounds, NoiseModel.uniform_depolarizing(0.002), no CZ conversion. Detector coordinates follow Chromobius's convention (the fourth coordinate is 0 to 2 for X-type and 3 to 5 for Z-type detectors).
• Bivariate bicycle: QArena's translation of Bravyi et al.'s schedule (A = x³ + y + y², B = y³ + x + x², H_X = [A | B], H_Z = [B^T | A^T]; data wires first, then X and Z ancillas; X schedule [idle, 1, 4, 3, 5, 0, 2], Z schedule [3, 5, 0, 1, 2, 4, idle]; eight stages per cycle). Every CNOT is followed by two-qubit depolarization p, every ancilla preparation by a flip with probability p (including the first Z ancilla reset), every measurement flips with probability p, and idle data qubits in stages 0, 6 and 7 get single-qubit depolarization p. The encoded state is prepared without noise and the final stabilizer and logical measurements are noiseless (MPP), as in the paper's memory simulations. Detectors of each cycle are listed X checks first, with coordinates (basis, check, cycle), basis 0 for X. The twelve logical operators are chosen deterministically (GF(2) elimination, greedy kernel vectors, then L_Z ← (M^-1)^T L_Z so that L_X L_Z^T = I); their supports are in the instance file, and observable k of the X (Z) experiment is logical X_k (Z_k). Each circuit has 6nR noisy CNOTs, nR preparations, nR measurements and 2nR idle locations.
• Honeycomb: HoneycombLayout(data_width=8, data_height=12, sub_rounds=72, style="EM3_v2", obs=..., noise=0.01) from the arXiv:2108.10457v2 ancillary source, with the exercised observable renamed to index 0. This cell is also a row of the paper's own data file (8 × 12, 24 rounds, EM3_v2, p = 0.01), where correlated matching fails on 2.35 % (H) and 1.13 % (V) of shots.

Shots were drawn with circuit.compile_detector_sampler(seed=...).sample(shots, separate_observables=True, bit_packed=True), one call per experiment. Development seeds are public (0x514152454e410000 + 100 i + b for instance i = 0..4 in the order above and experiment b = 0, 1); the ranked seeds were drawn with secrets.randbits(64) and are kept only in the server's private generation record. The published files are authoritative; the development seeds reproduce them only with the same Stim build (1.16.0, Linux x86-64 wheel).

Detection events use Stim's b8 format: each shot is ceil(D/8) bytes, detector i is bit i % 8 of byte i // 8 (little-endian bit order), padding bits are zero, and the two experiments are concatenated in the order listed. Per experiment you get the frozen circuit (.stim), the full detector error model (.dem), a decomposed one for matching decoders (surface and honeycomb), the ranked detection events and the development detection events and labels. The full ranked data are 158 MB of detection events.

An unranked hardware reference: Google Quantum AI's below-threshold surface-code data (Zenodo 13273331, google_105Q_surface_code_d3_d5_d7.zip). Its labels are public, so it is useful for studying hardware and model mismatch but cannot be a hidden-label instance; results on it are not records here.

Submission file

One JSON object, at most 1 MiB:

{"format": "b8-base64-v1",
 "instance": "bb72-bravyi-v1",
 "predictions": "<canonical base64>",
 "estimate": {"batch_ns": 1234567890, "cores": 1, "hardware": "CPU model; RAM; implementation version"}}

• predictions is the standard base64 (RFC 4648, with = padding, no whitespace, no URL-safe characters, canonical) of N × ceil(K/8) bytes: for every shot, in instance order, the predicted observable flips as one b8 row (observable j is bit j % 8 of the row's byte j // 8, padding bits zero in every row). For K = 1 each row is one byte, 0 or 1; for K = 12 each row is two bytes and the second byte's high nibble is zero. A ranked file therefore has exactly 266,668 base64 characters (K = 1) or 533,336 (K = 12).
• instance must be the instance you submit to; format must be b8-base64-v1.
• estimate is optional: the elapsed time in nanoseconds to produce the complete batch of all shots (loading inputs, building the decoder, decoding and writing the predictions), an integer from 1 to 2⁶³ − 1; the number of cores used, 1 to 4096; and 1 to 240 printable ASCII characters describing the hardware and implementation. It is always shown as an estimate and never affects acceptance or rank.

No other keys are allowed: no compression, archives, URLs, probabilities, abstentions or omitted shots.

Validation

The verifier is standard-library Python. It parses the JSON strictly (duplicate keys, NaN, infinities, Booleans as numbers and invalid UTF-8 are rejected), checks the keys and values, the base64 length before decoding, the canonical encoding, the exact decoded length and the padding bits of every row. All of this happens before the private key is opened, so a rejection never depends on a label. Then it checks the key's length and SHA-256 against the private record, which is bound to the SHA-256 of the public instance file, and compares every shot.

A file of wrong predictions is still a valid file: all-zero predictions, a complemented file or shots in the wrong order are accepted and scored. A missing or damaged key, an exhausted time budget or any internal failure gives status error with no score, and an error report never contains a path, a key byte or a partial count.

The verifier bundle cannot score a ranked file, because the labels are not in it: run_one.py answers error with hidden key unavailable. Check a ranked file's format with python3 -I verifiers/decoder_format.py <instance> my_file.json; development instances are scored exactly by run_one.py.

Scoring

For predicted row P_s and true row T_s, the score is E = the number of shots s with P_s ≠ T_s. A shot with several wrong observables counts once, also on the twelve-observable bicycle instances. The report also gives the exact fractions E/N (logical errors per shot) and E/(N R) (per round). The per-round figure only normalizes by the number of rounds; it is not a physical per-round logical error probability, since a final parity does not show how many logical flips happened in between.

Fewer errors rank first; equal error counts are ranked by submission number (lower first), and a participant's earliest submission with their best count is the one shown. Batch times never break ties. The counts are exact for these shots, but a difference of a few errors does not by itself show a difference in the underlying error rates.

Hidden labels and limits

Every score is an exact count, so each submission says a little about the hidden labels: flipping a chosen set of predictions and comparing scores reveals how many of those shots were wrong. To bound what one account can learn, each account may have at most 16 scored or pending submissions per ranked instance, ever (a rejected file returns no score and does not count; the daily limit of 30 files per problem also applies), every submission stays private so that no prediction file can be downloaded by others, and each leaderboard row shows the attempts used (n/16). With weighing designs or adaptive guessing on undecided shots, 16 scores can be worth some tens of errors, up to about 100 (1 to 2 % of the house baselines). Several accounts per person are not allowed; this cannot be enforced perfectly on a permanent board with exact scores, and the board does not claim to prevent pooled probing.

Batch times

Batch times are displayed and never affect rank. A self-reported batch time appears with the tag estimate. On request, QArena staff review a decoder program attached to an accepted submission as decoder.tar.gz (source, license, a run_decoder entry point, optional build.sh, decoder.json) and run it in QArena's single base Apptainer image on one CU Alpine CPU core with no network, no home directory, no labels and no access to the submitted file: one untimed warm-up on 1,024 development shots (its time is published as start-up time), then three timed runs on the whole 200,000-shot batch in fresh containers. If all three outputs equal the submitted predictions byte for byte, the median wall time is shown with the tag verified and the protocol name. A run may take up to 24 hours; if the first takes over an hour, that single time is published. Replacing the attached package hides the time. The full procedure is on the Rules tab.

Baseline and reference scores

QArena's baselines run on the same frozen ranked shots and their prediction files are public:

• Surface and honeycomb: PyMatching 2.4.0 with correlated matching (enabled when building and when decoding), on the frozen decomposed DEM, one thread.
• Bicycle and color: ldpc 2.1.0 BpOsdDecoder (minimum-sum BP, 200 iterations, scaling factor 0.625, parallel schedule, OSD-CS order 10, one thread) on the full DEM restricted to the detectors of the experiment's memory basis: components joined by ^ are combined into one fault, faults with equal remaining detector and observable sets are merged with probability p(1 − q) + q(1 − p), and faults that touch no remaining detector are dropped.

Better decoders are reported for related settings. Relay-BP (arXiv:2506.01779) reports that it significantly outperforms BP-OSD-CS on bivariate bicycle codes, and ensemble and neural decoders report gains over correlated matching on surface codes (for example Libra, arXiv:2408.12135). Single-basis BP-OSD already beats Chromobius on the color instance, so the color baseline is BP-OSD. No published result uses these exact shots, so the board has no reference score and no claimed optimum.

Application

Decoder researchers get common, downloadable circuit-level workloads: local matching problems (surface), multi-logical LDPC decoding with correlated faults (bivariate bicycle), color-code hyperedges (color) and Floquet measurement dynamics (honeycomb), scored on identical shots against hidden labels. Each leaderboard measures accuracy on its fixed experiment; it does not rank code families, thresholds, hardware platforms or real-time decoding systems.

Current records

Reference scores come from cited papers or derived constructions. Only accepted submissions can hold a QArena record.