Release status
Every classifier cell carries a status, and the status says what a call from it is worth. For most signature cells, a gate on the classifier's own repeated cross-validation sets it; the signature cells released before that gate and the HRD and MMRD/POLE models carry the word without it, as the table says, and bundled classifiers shows which is which. This documentation reports the status word and no performance figures.
The statuses
| Status | Mark | Meaning | What a call supports |
|---|---|---|---|
| released | ● | Passed the release gate below. 39 breast, lung and skin cells were released earlier under one rule only, a 10% false-positive rate on held-out simulated negatives: their calls carry no evidence grade and no call_evidence column, and bundled classifiers marks them "released before the gate". Two prostate cells passed a single-run version of the gate. | A positive call at the stated operating point (10% or 1% false-positive rate). For a cell released before the gate, how much a positive call raises the odds of the signature was not graded. |
| released, weak evidence | ◐ | Released; the false-positive rate is controlled, but a positive call changes the odds of the signature little. call_evidence is weak in the output and the run warns. | A call at the operating point, to be followed up with other evidence before it is reported as a finding. |
| experimental (score-only) | ▲ | Withheld by the gate. run() refuses it unless force=True, and then returns scores with NA call columns. | A ranking of samples within your cohort. No calls. |
| whole-genome reference | ● on wgs | The wgs cells threshold the exposure fraction of the signature; they are not panel classifiers. | A call on a genome, which restates the exposure that labeled it. |
| legacy (SigMA v1) | listed apart | SigMA v1's SBS3 and MMRD/POLE classifiers, ported. They reproduce R's output. | Reproduction of a SigMA v1 result. No evidence grade. |
| MMRD/POLE | ● | One SigMA2 model per platform (msk, tcga_mc3, seqcap, fo, op), cross-validated; no release gate. | MMRD, POLE or MMRP: the most probable of the three classes. A per-class cutoff set at a 1% false-positive rate in cross-validation is also required, but at its values it never overrules the most probable class. |
| HRD, own or pooled | ● | Cross-validated; no release gate. A tumor type with more than ten positives in the first training step, its BRCA1/2-deficient tumors plus three neighbors of each, has its own HRD model, so an own model can rest on as few as three BRCA1/2-deficient tumors (bundled classifiers gives the count); the others share the pooled one. | HRD, HRD-low or HRP. A pooled type's call rests on other tumor types' training data. |
The status is data, one Card per cell:
from sigma2 import classifiers
for c in classifiers.cards(tumor_type="prost"):
if c.kind == "signature":
print(c.classifier, c.seq_type, c.status)
| Parameter | Description | Default |
|---|---|---|
| tumor_typestr | Keep the cards of this tumor type. | None |
The bundled classifiers page renders the status of every cell.
The gate
A classifier is trained by repeated cross-validation (Train a classifier); the gate reads the out-of-fold results. A cell is withheld (experimental) when any one of these holds:
- A 10% false-positive rate cannot be certified on its negatives: fewer than 30 distinct negatives (the rule of three).
- The operating point is unstable under resampling.
- The median realized false-positive rate over the repeats exceeds the target by more than one negative.
- The lower bound of the positive likelihood ratio is at or below 1: a call carries no information.
Otherwise the cell is released, and graded weak when the evidence is thin: fewer than ten positives, or a positive likelihood ratio below the gate's bar. The weak grade labels; it does not withhold.
The classifiers trained on PCAWG genomes, for liver, lymphoma, kidney, pancreatic, esophageal, stomach, medulloblastoma and low-grade glioma tumors, passed two more checks outside the package: no single input of the classifier beat it on held-out genomes, and, for SBS3, it matched SigMA v1 where SigMA v1 has a classifier. These are selection rules of the release scripts, not of publish().
The status in a run
Classifier: tumor_type="breast", soi="SBS3", seq_type="msk" (verified)
A released cell runs without comment. A weak one adds a warning that its calls are weak evidence, and every row carries weak in call_evidence. An experimental one refuses:
the only classifier for tumor_type="prost", soi="SBS3", seq_type="op" is EXPERIMENTAL: withheld by the
release gate (...). Pass force=True to use it anyway; its scores rank samples, but pass_mva and the
confidence labels from it are not supported.
The parenthesis names the gate rule that failed. With force=True, the warning repeats it and the call columns come back NA.
What the status does not say
- It is not a per-sample confidence. That is
confidencein the output, which says which operating point a sample's score clears. - It is not a clinical grade. A released cell is a classifier with a controlled false-positive rate on held-out simulated panels of its tumor type; what that means for a patient is a question for the study that uses it.
- It is not permanent. A release can move a cell between statuses as its classifier is retrained; the card records the training date.
Cells with a known caveat
- Breast SBS3 on
mskandtcga_mc3was trained to call strong SBS3 and is not meant to call weak SBS3; its card records this inlabel_curation. - Colorectal SBS18 calls the summed SBS18 and SBS36 process. Colorectal SBS8 on
fo,mskandopis experimental. - Medulloblastoma SBS3 follows the SBS3 exposure of its training genomes, which carry no HRD scars; its calls do not indicate HRD.
Known limits collects these with the rest.