The refit benchmark

How well do refits, and the studio's report on them, hold up by assay and mutation count? The benchmark answers on synthetic samples with known truth, built from the half of PCAWG the calibration grid did not see.

Three versions

The tables below are version 1, the benchmark the report's flags were designed on. Because the flag rule was changed after it failed there, two fresh benchmarks test it (BENCH_SEED, BENCH_CATALOG, BENCH_OFFTYPE in scripts/validate/benchmark.ts):

| | Truth | Unflagged calls right | False calls flagged | Correct calls flagged | |---|---|---|---|---| | v1 (design) | MuSiCal, COSMIC v3.2 | 0.94–0.99 | 90.8% | 7.6% | | v2a | v1's source, new seed, 30% of tumors given a signature foreign to their type | 0.93–0.99 | 89.8% | 10.3% | | v2b | as v2a, truth from SigProfilerAssignment with COSMIC v3.4 | 0.90–0.97 | 69.6% | 15.9% |

What v2 adds:

• Fitting the tumor type's signatures misses every foreign signature, and its precision falls to 0.88–0.92 (v2a) and 0.70–0.86 (v2b). The full catalog with the rule finds the foreign signature in 80–97% of samples from 1,000 mutations; the flags hold back about a quarter of those at 5,000, because the prevalence prior calls them rare.
• When the truth comes from another method and catalog, precision falls at high counts (0.90 at 5,000): differences between catalogs make confident errors the calibration never saw.
• The prior's exact values matter little, having one matters. Perturbing each signature's PCAWG prevalence by a lognormal factor (σ = 1) left 89% (v2a) of errors flagged; a uniform prevalence of 0.5 left 67%.
• Correction is a high-burden effect. On the exome it moves precision from 0.21 to 0.22 at 50 mutations and 0.46 to 0.95 at 5,000.

Samples

796 tumors from the held-out half (at most 40 per tumor type, 27 types), each seen through three assays (whole genome, the GENCODE exome, MSK-IMPACT's targets) at 50, 200, 1,000 and 5,000 mutations: 9,552 samples. A sample is the tumor's MuSiCal exposure profile projected onto the territory and drawn at the count. Truth in the territory: a signature is present at a share of 5% or more, absent at zero; in between is not scored.

Arms

| Arm | What | |---|---| | naive | NNLS with all of COSMIC v3.2, not corrected to the assay | | corrected | the same, corrected to the assay | | rule | corrected, keeping calls with share ≥ 5% whose bootstrap 5th percentile is ≥ 1% | | rule + flags | the rule, keeping only calls the confidence report leaves unflagged | | type | corrected, the tumor type's signatures only, with the rule | | MuSiCal | MuSiCal's refit (likelihood bidirectional), COSMIC v3.2 | | SigProfilerAssignment | cosmic_fit 1.1.5, COSMIC v3.2 (exome signatures for the exome) |

The in-page arms run the studio's own code (scripts/validate/benchmark.ts); the two packages run as installed by the runner.

Precision, then recall

Whole genome:

| Arm | 50 | 200 | 1,000 | 5,000 | |---|---|---|---|---| | naive / corrected | 0.19 / 0.32 | 0.34 / 0.50 | 0.66 / 0.75 | 0.96 / 0.90 | | rule | 0.75 / 0.13 | 0.69 / 0.29 | 0.77 / 0.55 | 0.96 / 0.83 | | rule + flags | 0.95 / 0.12 | 0.98 / 0.27 | 0.98 / 0.48 | 0.99 / 0.79 | | type | 0.94 / 0.41 | 0.94 / 0.69 | 0.95 / 0.88 | 0.95 / 0.93 | | MuSiCal | 0.21 / 0.33 | 0.41 / 0.54 | 0.88 / 0.81 | 0.98 / 0.88 | | SigProfilerAssignment | 0.55 / 0.49 | 0.70 / 0.64 | 0.93 / 0.76 | 0.96 / 0.83 |

Exome:

| Arm | 50 | 200 | 1,000 | 5,000 | |---|---|---|---|---| | naive | 0.21 / 0.32 | 0.29 / 0.43 | 0.40 / 0.57 | 0.46 / 0.65 | | corrected | 0.22 / 0.34 | 0.36 / 0.51 | 0.67 / 0.74 | 0.95 / 0.89 | | rule + flags | 0.94 / 0.12 | 0.96 / 0.27 | 0.98 / 0.49 | 0.99 / 0.77 | | type | 0.93 / 0.40 | 0.95 / 0.66 | 0.94 / 0.87 | 0.96 / 0.93 | | MuSiCal | 0.22 / 0.34 | 0.34 / 0.47 | 0.58 / 0.66 | 0.70 / 0.71 | | SigProfilerAssignment | 0.59 / 0.50 | 0.74 / 0.63 | 0.89 / 0.72 | 0.94 / 0.76 |

Panel (MSK-IMPACT's targets):

| Arm | 50 | 200 | 1,000 | 5,000 | |---|---|---|---|---| | naive | 0.21 / 0.32 | 0.31 / 0.45 | 0.42 / 0.59 | 0.49 / 0.67 | | corrected | 0.22 / 0.35 | 0.37 / 0.52 | 0.69 / 0.75 | 0.97 / 0.89 | | rule + flags | 0.97 / 0.13 | 0.98 / 0.27 | 0.99 / 0.48 | 0.99 / 0.77 | | type | 0.94 / 0.39 | 0.95 / 0.66 | 0.95 / 0.87 | 0.95 / 0.93 | | MuSiCal | 0.23 / 0.34 | 0.34 / 0.48 | 0.58 / 0.65 | 0.74 / 0.73 | | SigProfilerAssignment | 0.55 / 0.49 | 0.62 / 0.60 | 0.70 / 0.68 | 0.74 / 0.69 |

Mean exposure error (half the L1 distance between true and estimated shares), over assays:

| Arm | 50 | 200 | 1,000 | 5,000 | |---|---|---|---|---| | naive | 0.77 | 0.67 | 0.53 | 0.42 | | corrected | 0.75 | 0.61 | 0.39 | 0.18 | | rule + flags | 0.45 | 0.40 | 0.31 | 0.17 | | type | 0.34 | 0.21 | 0.12 | 0.08 | | MuSiCal | 0.73 | 0.54 | 0.27 | 0.19 | | SigProfilerAssignment | 0.49 | 0.33 | 0.22 | 0.18 |

What it says

• Correct for the assay. Without opportunity correction, exome and panel precision stalls below 0.5 even at 5,000 mutations; corrected, it reaches 0.95–0.97. The packages' exome and panel rows show the same: MuSiCal does not correct, and SigProfilerAssignment's exome signatures do not help a panel.
• Fit the tumor type's signatures if you know the type. It is the best arm at every count, in both precision and error.
• The report's flags hold the errors. Of the false calls the rule makes, the report flags 90.8%; of the right calls, it flags 7.6%. Unflagged calls are right 94–99% of the time at every assay and count. The price is recall: at 50 mutations the unflagged calls find 12% of what is present.

The gate

The plan's gate asked that flagged calls hold a clear majority of errors. The first version of the report flagged 50.4% of them: it trusted a low false-positive rate per absent sample, and missed that a call of a rare signature is still usually wrong. The report now flags a call whose positive predictive value, from the calibration grid's detection rate, false-positive rate and the signature's prevalence in PCAWG, is below 0.8, or that no PCAWG tumor carries at that assay and count. The prevalence is PCAWG's: a cohort with a different exposure history (a thiopurine-treated series, for one) is judged against the wrong prior, and the report says when a call is flagged on prevalence.