Training
Add
The training object (Train a classifier).
sg2.Add(input, exposures, tumor_type, seq_type, soi, catalog="cosmic_v3p2_inhouse", verbose=1)
| Parameter | Description | Default |
|---|---|---|
| inputSamplesAPI | The genomes as a Samples object (VCFs or a MAF read with a reference genome). | Required |
| exposuresDataFrame | A DataFrame, samples as rows, one column per signature of the catalog; counts or fractions. | Required |
| tumor_typestr | The cohort's tumor type. | Required |
| seq_typestr or SeqType or list of str, List[SeqType] | The platforms to simulate: a name, a SeqType with a BED, or a list. | Required |
| soistr or Cosmic or list of str, List[Cosmic] | The signatures to train: a name or a list. | Required |
| catalogstr | The catalog the exposures were fit on. | 'cosmic_v3p2_inhouse' |
| verboseint | 1 prints the overview on construction and each stage's table. | 1 |
Internal parameters (1)
| loggerLogger | Internal: the logger the object reports through. | None |
Returns An Add object holding the genomes and exposures; annotate, cluster, simulate, tune, then publish or export.
Used in Train a classifier, Worked example: train on your own genomes
Add.annotate
Bin each signature's exposures into labels: 0 negative, the other bins positive bands.
a.annotate(strategy="gauss", force=False, verbose=None, min_exposure=0.0, below="negative", n_components=None)
| Parameter | Description | Default |
|---|---|---|
| strategystr, number or list of numbers | "gauss" fits a mixture to each signature's exposures and BIC picks the components; a number or list sets the bands by hand. | 'gauss' |
| forcebool | Redo an annotation that exists. | False |
| verboseint | Print the annotation table. | None |
| min_exposurefloat or dict | A relative floor below which a sample is not positive, as one number or per signature. | 0.0 |
| belowstr | "negative" labels samples under the floor as negatives; "exclude" drops them from training. | 'negative' |
Returns The object, with annotated: one bin per sample and signature; the annotation table is printed.
"gauss" fits a mixture and BIC picks two to five components unless n_components fixes it; min_exposure is a hand-set relative floor; below="exclude" drops samples under it instead of labeling them negative.
Used in Train a classifier, Curate training labels, Worked example: train on your own genomes
Add.cluster
Cluster samples within each band and take the centroids.
a.cluster(method="elbow", distance_threshold=0.3, linkage="ward", force=False, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| methodstr | "elbow" picks the cluster count per band; "distance" holds cluster tightness fixed and lets the count grow. | 'elbow' |
| distance_thresholdfloat | The Euclidean cutoff on normalized exposure fractions for method="distance". | 0.3 |
| linkagestr | The agglomerative linkage for method="distance". | 'ward' |
| forcebool | Redo clusters that exist. | False |
| verboseint | Print the cluster table. | None |
Returns The object, with clusters: the reference spectra per signature and band.
"elbow" picks the count per band; "distance" holds cluster tightness at distance_threshold and lets the count grow with the cohort.
Used in Train a classifier, Worked example: train on your own genomes
Add.simulate
Downsample every genome to each platform's capture regions, topped up by scaling_factor from addition_source (default: the genome's own spectrum).
a.simulate(scaling_factor=2.0, addition_source=None, do=None, include_wgs=True, force=False, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| scaling_factorfloat | How far the in-panel mutations are topped up from the genome's own spectrum; SigMA's breast value by default. | 2.0 |
| addition_sourceSamples | Samples to draw the top-up from instead of the genome's own spectrum. | None |
| dolist of str | Restrict the simulation to these signatures. | None |
| include_wgsbool or str | Keep the genomes themselves as a wgs cell. | True |
| forcebool | Redo simulations that exist. | False |
| verboseint | Print the simulation table. | None |
Returns The object, with simulations: one simulated panel per genome and platform; the simulation table is printed.
do restricts to some signatures; include_wgs=True keeps the genomes as a wgs cell. Needs sigma2[panel] and bedtools.
Used in Train a classifier, Curate training labels, Worked example: train on your own genomes
Add.tune
Train one classifier per signature and platform in nested cross-validation and write the result tree.
a.tune(target_fpr=0.1, mxw="max", do=None, output_dir=None, force=False, verbose=None, snv_cutoff=None,
snv_constraint=None, max_snv_cutoff=12, snv_candidates=None, estimator="gbm", monotonic=True,
stratify_by=None, stratify_unknown="drop", capture_redraws=0, cv_repeats=1, cutoff_rule="mean_score",
cross_fit=False)
| Parameter | Description | Default |
|---|---|---|
| target_fprfloat | The false-positive rate of the loose operating point. | 0.1 |
| mxwstr | The worker count; "max" uses every core. | 'max' |
| dolist of str | Restrict training to these signatures. | None |
| output_dirstr | Where the result tree is written. | None |
| forcebool | Retrain models that exist. | False |
| verboseint | Print progress and the summary table. | None |
| snv_cutoffint | Train at this SNV floor instead of the tumor type's. | None |
| snv_constraintdict or list | Search the lowest floor meeting these {fpr: sensitivity} constraints; a full training pass per candidate. | None |
| max_snv_cutoffint | The highest floor the search tries. | 12 |
| snv_candidateslist of int | Try these floors only, instead of scanning. | None |
| estimatorstr | The learner: "gbm", "hist", "monologit" (the shipped cells') or "monologit_ctx". | 'gbm' |
| monotonicbool | For "hist": constrain the sign of each feature's effect. | True |
| stratify_bytuple, str or dict | Restrict a signature's cohort to one stratum of another signature. | None |
| stratify_unknownstr | "drop" or "keep" samples the stratifier has no status for. | 'drop' |
| capture_redrawsint | The capture-fidelity diagnostic's redraw count; 0 turns it off. | 0 |
| cv_repeatsint | Repeat the outer cross-validation this many times with new splits; most shipped cells used 20. | 1 |
| cutoff_rulestr | "mean_score" or "median_repeat" (the shipped cells') for the released cutoff under repeats. | 'mean_score' |
| cross_fitbool | Fit the released model on folds rather than on everything; most shipped cells used it. | False |
Returns The object, with models; the result tree is written to output_dir and the summary table printed.
Used in Train a classifier, Curate training labels, Worked example: train on your own genomes
Add.publish
Write the models into the data directory's store behind the release gate; a withheld cell goes to experimental/.
a.publish(force=False, verbose=None, skip=None, result_dir=None, precision_bar=None, publish_experimental=True,
min_lr=5.0, min_lr_lower=1.0, min_pos=10)
| Parameter | Description | Default |
|---|---|---|
| forcebool | Overwrite models in the store. | False |
| verboseint | Print what was published. | None |
| skipdict | {soi: seq_types} to leave out. | None |
| result_dirstr | The result tree to judge from; without it everything publishes with a warning. | None |
| precision_barfloat | The older PPV partition instead of the gate; off by default. | None |
| publish_experimentalbool | Write withheld cells to experimental/ (True) or drop them (False). | True |
| min_lrfloat | The likelihood-ratio bar of the gate. | 5.0 |
| min_lr_lowerfloat | The bar for the likelihood ratio's lower bound. | 1.0 |
| min_posint | The positives a cell needs to be graded informative. | 10 |
Returns Nothing; the models that pass the gate are written into the data directory's store, the others under experimental/.
result_dir is the tree to judge from (without it, everything publishes with a warning); skip leaves {soi: seq_types} out; publish_experimental=False drops withheld cells; precision_bar is the older PPV partition, off by default.
Used in Train a classifier
Add.export
Write every model to output_dir in the store layout, with no gate.
a.export(output_dir, force=False, verbose=None, skip=None)
| Parameter | Description | Default |
|---|---|---|
| output_dirstr | The directory that becomes a store, in the <tumor_type>/<soi>/<seq_type> layout. | Required |
| forcebool | Overwrite models in the directory. | False |
| verboseint | Print what was written. | None |
| skipdict | {soi: seq_types} to leave out. | None |
Returns Nothing; every model is written to output_dir in the store layout.
The directory is then a store for sg2.model(name, root_dir=output_dir).
Used in Train a classifier, Worked example: train on your own genomes, Manage models and the store
Add.save
.pkl saves the full state at any stage; .json saves the editor export (spectra, exposures, bins, clusters, optional simulations and tuning).
a.save(path=None, force=False, verbose=None) -> Path
| Parameter | Description | Default |
|---|---|---|
| pathstr or path | A .pkl path for the full state or a .json path for the editor export; the data directory when left out. | None |
| forcebool | Overwrite the file. | False |
| verboseint | Print what was saved. | None |
Returns The path written: a .pkl of the whole state, or the .json editor export.
Used in Train a classifier, Curate training labels, Worked example: train on your own genomes
Add.load
Restore from a .pkl, a .json (clusters applied; simulate() unavailable), or an (H, W) pair of exposure and signature matrices (spectra reconstructed as W·H; simulate() and tune() on it are invalid).
sg2.Add.load(path, verbose=1, *, tumor_type=None, seq_type=None, soi=None, catalog="custom", ref_genome=None) -> Add
| Parameter | Description | Default |
|---|---|---|
| pathstr, Path or tuple | A .pkl, a .json (the editor export), or an (H, W) pair of exposure and signature matrices. | Required |
| verboseint | Print what was loaded. | 1 |
| tumor_typestr | The tumor type, for an (H, W) pair. | None |
| seq_typestr or SeqType or List | The platforms, for a JSON or an (H, W) pair. | None |
| soistr or List | The signatures, for an (H, W) pair. | None |
| catalogstr | The catalog, for an (H, W) pair. | 'custom' |
| ref_genomestr | The build, for an (H, W) pair. | None |
Returns An Add object restored from a .pkl, a .json export, or an (H, W) pair of matrices.
Add.apply
Apply a cluster-editor JSON back into the object: bins and clusters replaced, simulations and models invalidated.
a.apply(path, only=None, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| pathstr or path | The edited JSON from the cluster editor. | Required |
| onlylist of str | Take only these signatures' clusters from the JSON. | None |
| verboseint | Print what was applied. | None |
only=["APOBEC"] takes one signature's clusters.
Used in Curate training labels
Add.inject_simulations
Simulations made elsewhere: {panel}.csv files or a {panel: path} dict, sample × 96 with the object's sample IDs.
a.inject_simulations(sims, do=None, panel_aliases=None, ref_genome=None, include_wgs=True, keep=None, force=False, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| simsstr, Path or dict | {panel}.csv files in a folder, or a {panel: path or DataFrame} dict, sample × 96 with the object's sample IDs. | Required |
| dolist of str | Restrict to these signatures. | None |
| panel_aliasesdict of str, str | Map file names to platform names; mc3 to tcga_mc3 is built in. | None |
| ref_genomestr | The build the simulations were made on. | None |
| include_wgsbool or str | Keep the genomes themselves as a wgs cell. | True |
| keepstr or bool | How duplicate sample IDs are resolved. | None |
| forcebool | Replace simulations that exist. | False |
| verboseint | Print what was injected. | None |
Mismatched IDs are refused naming both sides; keep handles duplicates; mc3 maps to tcga_mc3.
Used in Curate training labels
Add.extend
Add a signature to a built object, with its own annotation.
a.extend(new_soi, strategy="gauss", verbose=None) -> Add
| Parameter | Description | Default |
|---|---|---|
| new_soistr or list of str | The signature, or list of signatures, to add. | Required |
| strategystr or float or int or list of float | The annotation strategy for the new signature. | 'gauss' |
| verboseint | Print the annotation table. | None |
Returns Add.
Used in Curate training labels
Add.split
Hold out a split; its scores land in eval_scores.tsv.
a.split(test_size=0.25, strat_on=None, rs=42, do=None, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| test_sizefloat | The share held out. | 0.25 |
| strat_onstr | A column to stratify the split on. | None |
| rsint | The random seed. | 42 |
| dolist of str | Restrict to these signatures. | None |
| verboseint | Print the split. | None |
Add.merge
Merge two objects of the same tumor type.
a.merge(other, soi=None) -> Add
| Parameter | Description | Default |
|---|---|---|
| other'Add' | Another Add object of the same tumor type. | Required |
| soilist of str | Restrict to these signatures. | None |
Returns Add.
Used in Curate training labels
Add.drop_duplicates
Removes genomes that are present twice.
a.drop_duplicates(keep=False, inplace=False, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| keep | Which of the duplicates to keep; False drops all. | False |
| inplacebool | Modify this object instead of returning a copy. | False |
| verboseint | Print what was dropped. | None |
Add.label_report
Positives, negatives, exact zeros and prevalence per signature, with a warning for negative classes under 30.
a.label_report(verbose=None) -> DataFrame
| Parameter | Description | Default |
|---|---|---|
| verboseint | Print the report as well as returning it. | None |
Returns A DataFrame, one row per signature: positives, negatives, exact zeros, prevalence; warnings for small negative classes.
Used in Curate training labels
Add.sparse_report
Simulated samples below the floor, per signature, platform and bin.
a.sparse_report(snv_cutoff=None, verbose=None) -> DataFrame
| Parameter | Description | Default |
|---|---|---|
| snv_cutoffint or dict of str or int, NoneType | The floor to count against, as one number or per signature; the tumor type's when left out. | None |
| verboseint | Print the report as well as returning it. | None |
Returns A DataFrame: the simulated samples below the floor, per signature, platform and bin.
Used in Curate training labels, Worked example: train on your own genomes
Add.filter_sparse
Drop simulated samples below a floor, optionally resimulating.
a.filter_sparse(snv_cutoff=None, resimulate=False, force=False, verbose=None) -> Add
| Parameter | Description | Default |
|---|---|---|
| snv_cutoffint or dict of str or int, NoneType | The floor below which simulated samples are dropped. | None |
| resimulatebool | Simulate again after dropping. | False |
| forcebool | Apply even when simulations would be lost. | False |
| verboseint | Print what was dropped. | None |
Returns Add.
Add.fractional_exposures
Exposures as fractions of the catalog total.
a.fractional_exposures(soi_only=False, catalog_total=True, inplace=False)
| Parameter | Description | Default |
|---|---|---|
| soi_onlybool | Convert the signatures of interest only. | False |
| catalog_totalbool | Divide by the catalog total (True) or the row total of the given columns. | True |
| inplacebool | Modify this object instead of returning a copy. | False |
Add.export_threshold_table
Re-score a finished run at several SNV floors without retraining; writes threshold_auc_table.tsv, threshold_curve.tsv, sigma2_thresholds.json and, with stratifier, stratified_auc.tsv.
a.export_threshold_table(output_dir, floors=None, objective=None, legacy_lock=("SBS3",), stratifier=None, verbose=None) -> dict
| Parameter | Description | Default |
|---|---|---|
| output_dirstr or Path | A result tree written by tune(output_dir=...). | Required |
| floorsdict | {signature: floor} to re-score at; computed when left out. | None |
| objectivestr | The floor-selection objective. | None |
| legacy_locktuple of str | Signatures pinned to the SigMA v1 floor. | ('SBS3',) |
| stratifierstr | Another signature to split every discrimination measure by. | None |
| verboseint | Print what was written. | None |
Returns A DataFrame of discrimination at each floor, written into the result tree with its curves.
Add.record_provenance
Write the provenance files of a result tree.
a.record_provenance(output_dir, do=None) -> list[str]
| Parameter | Description | Default |
|---|---|---|
| output_dir | The result tree to write provenance into. | Required |
| do | Restrict to these signatures. | None |
Returns list[str].
Add.label_definition
How each signature's labels were defined, as data.
a.label_definition(soi=None) -> dict
| Parameter | Description | Default |
|---|---|---|
| soi | The signature, or all when left out. | None |
Returns dict.
Add.supplement_from
Add donor genomes from another cohort.
a.supplement_from(panels, donor_dir, donor_tree, do, max_donors=None, composition_quantile=0.5, count_match=True,
donor_class="positive", donor_name=None, reference=None, on_ineligible="warn", random_state=42, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| panelslist of str | The platforms to supplement. | Required |
| donor_dirstr | The directory of donor genomes. | Required |
| donor_treestr | The donors' result tree. | Required |
| dolist of str | The signatures to supplement. | Required |
| max_donorsint | The most donors added per signature. | None |
| composition_quantilefloat | The composition quantile a donor must match. | 0.5 |
| count_matchbool | Match the donors' burdens to the cohort's. | True |
| donor_classstr | Which class the donors join. | 'positive' |
| donor_namestr | A name recorded for the donors. | None |
| referenceDataFrame | A reference the donors are matched against. | None |
| on_ineligiblestr | "warn" or "raise" when a donor is ineligible. | 'warn' |
| random_stateint | The random seed. | 42 |
| verboseint | Print what was added. | None |
Advanced and usually harmful (Known limits).
Add.contaminate
Mix germline calls into the simulations at a rate, for robustness checks.
a.contaminate(germline, rate, do=None, rs=42, verbose=None)
| Parameter | Description | Default |
|---|---|---|
| germlinepath or DataFrame | The germline calls to mix in. | Required |
| ratefloat | The contamination rate. | Required |
| dolist of str | Restrict to these signatures. | None |
| rsint | The random seed. | 42 |
| verboseint | Print what was done. | None |
tools.simulator.calibrate_scaling_factor
Fit the scaling factor that matches a real cohort's burden distribution on the panel: f_star, the distance curve, and whether the optimum sits on the grid's edge.
sigma2.tools.simulator.calibrate_scaling_factor(base_burdens, reference_burdens, grid=None, metric="wasserstein") -> dict
| Parameter | Description | Default |
|---|---|---|
| base_burdensarray-like | The on-panel SNV burden per simulated sample at scaling_factor=1.0 (no top-up). | Required |
| reference_burdensarray-like | The observed per-sample SNV burden of a real cohort on the same panel. | Required |
| gridarray-like | The candidate factors; 1.0 to 3.0 in steps of 0.05 when left out. | None |
| metricstr | "wasserstein" (on log counts) or "ks" selects the optimum; both are returned. | 'wasserstein' |
Returns The scaling factor that matches a real cohort's burdens on the panel.