phipml: from peptide data to model interpretation

A runnable tutorial for Random Forest, XGBoost, nested cross-validation, external validation, and publication-ready plots.

Built against phipml 4.2.0, commit 932f14f, inspected on 8 September 2026. The commands below use the implemented configuration schema and were run on the supplied synthetic data.

You will start with peptide and sample metadata CSVs, fit either estimator, compare fixed/default and tuned models, evaluate an independent synthetic cohort, and produce ROC, precision–recall, classification, SHAP, and annotated feature plots. Finished results are included so you can inspect them before rerunning anything.

Start here: follow sections 1–6 for your first model and plots. Then run the six scenarios, compare the results, and use the references when adapting your own configuration. For a workshop, start with Random Forest and add XGBoost after the workflow is familiar.

What is included

The tutorial is a standalone folder. It can also be added under tutorials/from_data_to_plots/ in the phipml repository: relative paths inside the folder continue to work. Run its commands from that folder. It does not require changes to phipml's source.

1. Install phipml

Recommended: the repository's Micromamba environment

Use Linux/macOS, or an equivalent Bash environment. Install Git and Micromamba first if needed; platform setup and Docker/Apptainer alternatives are in the repository's installation guide.

git clone https://github.com/csReynaB/phipml.git
cd phipml

# Pin the source used by this tutorial. This is a local checkout only.
git checkout 932f14ffca41ba4f24f3cc570afea27265a5893c

micromamba create --yes --name phipml --file ML_env.yml
micromamba activate phipml
python -m pip install --no-build-isolation --no-deps -e .

phipml --version
phipml -h
phipml-plot -h
phipml-heatmap -h

ML_env.yml supplies the tested scientific package pins with Python 3.10.16. --no-deps prevents pip from replacing those packages. -e links the installed package to the cloned source, so retain the checkout.

Alternative: pip in a virtual environment

In the same pinned repository checkout, use an installed Python 3.10–3.12:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .
phipml --version

Although the package declares Python >=3.10, its dependency upper bounds make 3.10–3.12 the practical choices for this tutorial; do not assume every newer Python can resolve the pinned scientific stack. Pip resolves compatible ranges and can choose versions different from ML_env.yml. The included run used Python 3.12.13, sklearn 1.5.2, XGBoost 2.1.4, and SHAP 0.46.0; exact package versions are recorded in reports/requirements-reference.txt.

The virtual environment must remain active while running the tutorial. The small examples use CPU execution; no GPU is required.

Open the tutorial folder

Extract phipml-tutorial-932f14f.zip and enter the extracted folder. For example, if your downloaded archive is in the current directory:

unzip phipml-tutorial-932f14f.zip
cd phipml-tutorial-932f14f

All commands below assume this directory. Data/config files are already included. To recreate them deterministically:

python scripts/generate_examples.py

The generator rewrites its example data/configs with the documented defaults; copy a config before customizing it. It does not delete old results. If you change --data-seed or --n-iter, keep a separate tutorial copy/output directory so old artifacts cannot be mistaken for the new experiment.

2. Understand the example data

There are two datasets, perfect and noisy. Each includes 160 training samples and 80 independent external samples, with balanced Control/Case labels. Every sample has 80 binary peptide measurements; four numeric clinical predictors are supplied through metadata.

The matching files under data/perfect/ have the same schema. Inspect the actual data with:

python - <<'PY'
import pandas as pd
X = pd.read_csv("data/noisy/peptides.csv", index_col=0)
metadata = pd.read_csv("data/noisy/metadata.csv").set_index("SampleName")
print("Peptide matrix:", X.shape)
print(X.iloc[:4, :6])
print(metadata.head(4))
print(pd.crosstab(metadata["cohort"], metadata["group_test"]))
assert X.index.is_unique and metadata.index.is_unique
assert set(X.index) == set(metadata.index)
PY

Expected counts:

group_tests: [Control, Case] encodes Control=0 and Case=1. cohort selects training/external rows and is not included in the predictor matrix. The model receives 80 peptides plus the four configured extras: 84 features before pipeline feature selection.

What “perfect” and “noisy” mean

In the perfect data, six engineered peptides exactly encode the label or its inverse. This is an intentional software demonstration. The remaining 74 peptides and clinical features are independent of class in the data-generating process. Correlated perfect peptides can share model importance unevenly.

In the noisy data, six peptides have class-dependent presence probabilities: 0.70 versus 0.30 in training, and 0.65 versus 0.35 externally, with alternating directions. The external relationship is deliberately slightly weaker. The external age distribution is also shifted by four years, independently of class. There is no promise that tuning improves the result.

The generator draws training and external samples independently. Default and tuned runs reuse exactly the same input files and model seed. Synthetic annotations identify the intended signal/noise roles; they are not real taxa or biomarkers. data/generation_manifest.json records the generation settings and SHA-256 hashes.

Matrix orientation

The tutorial uses sample rows × peptide columns with transposed: false. For a peptide-row matrix, use transposed: true. Equivalent peptides_transposed.csv files are included for illustration. The first column must contain the relevant row identifiers in either orientation.

3. Read the first model config

Open configs/models/random-forest/01_perfect.yaml. The main decisions are:

# Paths below are relative to configs/models/random-forest/.
data_input: ../../../data/perfect/peptides.csv
metadata_input: ../../../data/perfect/metadata.csv
data_input_mode: matrix
transposed: false

col_sample_name: SampleName
col_target: group_test
group_tests: [Control, Case]
peptide_prefixes: [agilent_, twist_, corona2_]
extra_features_to_include: [Sex, Smoking, Age, BMI]
fillna_value: null

lib_metadata_input: ../../../data/peptide_library.csv
lib_col_peptide_name: peptide_id
random_state: 420

classification:
  model_type: random-forest
  param_grid_name: random-forest
  seed: 420
  train_filters: {cohort: training}
  run_nested_cv: true
  only_train_model: true
  use_pretrained: false
  with_oligos: true
  with_additional_features: true
  prevalence_threshold_min: 0
  prevalence_threshold_max: 100
  outer_cv_splits: 3
  inner_cv_splits: 2
  n_iter: 1
  n_jobs_outer: 1
  n_jobs_inner: 1
  classification_threshold: 0.5
  validation_sets: []
  output_dir: ../../../results/random-forest/01_perfect
  output_name: 01_perfect

param_grid: {}

This is a complete minimal working configuration; the supplied file spells out additional options documented in the model reference.

param_grid: {} means no hyperparameter search. Constant-feature removal and elastic-net peptide selection still fit inside each training pipeline. run_nested_cv: true requests outer-CV evaluation; with an empty grid there is no inner tuning. only_train_model: true then saves a model fitted on the whole training cohort and skips external validation.

External rows are present in the input files but excluded from all training and tuning by train_filters: {cohort: training}.

4. Fit a Random Forest and locate the outputs

phipml -c configs/models/random-forest/01_perfect.yaml

The log should report samples=160, features=84. This run creates:

Filename pattern: {nested|training}_{model_type}_{output_name}_{seed}.joblib. Running the same configuration/seed again replaces the same filenames.

5. Generate the first plots

phipml-plot --plot-config configs/plots/random-forest/01_perfect.yaml

Open plots/random-forest/01_perfect/01_perfect_performance.png. It should show near-perfect discrimination because the data were constructed to be perfectly separable. The plotting recipe produces nine figure types in PDF, SVG, and PNG, plus a detailed feature-importance CSV.

The essential plotting fields are:

plotting:
  results:
    - ../../../results/random-forest/01_perfect/nested_random-forest_01_perfect_420.joblib
  config: ../../models/random-forest/01_perfect.yaml
  split: train
  plots: [all]
  formats: [pdf, svg, png]
  dpi: 180
  output_dir: ../../../plots/random-forest/01_perfect
  output_prefix: 01_perfect
  class_labels: [Control, Case]
  max_display: 10
  library_metadata: ../../../data/peptide_library.csv
  library_id_column: peptide_id

--plot-config reads the plotting recipe; its config: entry points to the model/data YAML. split: train refers to the training cohort's out-of-fold predictions. Use split: test for an external validation artifact. The separate training_...joblib file is not an evaluation result to plot.

See the plotting reference for all fields, colors, annotation controls, feature ranking, and standalone-panel commands.

6. Move to noisy data, then tune

# Identical schema, imperfect signal, default hyperparameters.
phipml -c configs/models/random-forest/02_noisy.yaml
phipml-plot --plot-config configs/plots/random-forest/02_noisy.yaml

# The same noisy samples and seed, with inner-CV hyperparameter search.
phipml -c configs/models/random-forest/03_noisy_tuned.yaml
phipml-plot --plot-config configs/plots/random-forest/03_noisy_tuned.yaml

The tuned YAML adds search spaces beneath param_grid.random-forest and sets n_iter: 12. It searches the peptide selector's regularization, forest tree count/depth, split/leaf sizes, feature subsampling, and class weighting. All candidate comparisons happen in training folds. The held-out outer samples are evaluated only after a candidate has been selected.

For example, increasing min_samples_leaf makes leaves depend on more samples and generally smooths predictions; increasing selector C weakens regularization. Those are different parts of the pipeline. The hyperparameter guide explains every search parameter, the direction of its effect, and the exact YAML syntax.

What a real noisy result looks like

This figure is generated by the command above. ROC/PR summarize outer-fold performance. The confusion matrix pools held-out predictions. The classification bars in this version use outer-fold means and variability when present; these can differ slightly from metrics recomputed on all pooled predictions.

Interpret the learned features

The highest-ranked peptides largely recover the engineered signal in this example. Noise peptides and clinical variables can still receive nonzero importance. Class-wise prevalence/means and SHAP importance answer different questions: prevalence describes the samples; SHAP explains the fitted model. The CSV also includes column types and statistic labels omitted from the compact figure. Here peptide values are percentages and Age/BMI entries are means in their original units.

7. Evaluate the external cohort

Use run_nested_cv: false, only_train_model: false, and add the validation cohort. Each model is fitted/tuned using the training rows and then evaluated on the 80 independent external samples.

classification:
  train_filters: {cohort: training}
  run_nested_cv: false
  only_train_model: false
  validation_sets:
    - name: 05_external_noisy
      filters: {cohort: external}
  classification_threshold: 0.5
  bootstrap_validation: true
  bootstrap_n_resamples: 500
  bootstrap_confidence_level: 0.95

The fragment illustrates the change; run the complete supplied files:

phipml -c configs/models/random-forest/04_external_perfect.yaml
phipml-plot --plot-config configs/plots/random-forest/04_external_perfect.yaml

phipml -c configs/models/random-forest/05_external_noisy.yaml
phipml-plot --plot-config configs/plots/random-forest/05_external_noisy.yaml

phipml -c configs/models/random-forest/06_external_noisy_tuned.yaml
phipml-plot --plot-config configs/plots/random-forest/06_external_noisy_tuned.yaml

For example, the noisy external artifact is results/random-forest/05_external_noisy/validation_random-forest_05_external_noisy_420.joblib. Its name comes from the validation set's name, not output_name.

The current CLI selects cohorts from shared input files. Separate per-validation matrix/metadata paths are not implemented in validation_sets at this commit. For new files, see external-input guidance.

8. Run the same six scenarios with XGBoost

Use the XGBoost configurations supplied alongside Random Forest:

for scenario in 01_perfect 02_noisy 03_noisy_tuned 04_external_perfect 05_external_noisy 06_external_noisy_tuned; do
  phipml -c "configs/models/xgboost/$scenario.yaml"
  phipml-plot --plot-config "configs/plots/xgboost/$scenario.yaml"
done

The input/cohort definitions stay the same. XGBoost tuning instead searches tree count, learning rate, depth, child weight, row/column subsampling, and leaf/split regularization, along with the shared peptide selector.

Each dot is one external sample. Red/blue show high/low feature values; left/right show contributions away from/toward Case. XGBoost's SHAP values here are raw-margin/log-odds contributions, not probability-point differences. Mean absolute SHAP can rank features but does not give direction, statistical significance, or a causal interpretation.

9. One command for all scenarios

After installation, from the tutorial directory:

# Six models, all plots, heatmaps, and a metric/prediction export.
bash scripts/run_all.sh random-forest

# Or use XGBoost.
bash scripts/run_all.sh xgboost

# Or run all twelve estimator/scenario combinations.
bash scripts/run_all.sh both

Core recipes use three outer folds, two inner folds, one worker, and 12 search candidates for tuned runs. This is a short tutorial budget, not a recommended final analysis design. Hardware, installation startup, tuning, and plot export affect runtime; the reference execution timings are recorded separately.

10. Compare the observed results

python scripts/summarize_results.py

phipml-heatmap --manifest manifests/random-forest.csv \
  --metric roc.auc --vmin 0 --vmax 1 --palette viridis --dpi 180 \
  --output plots/random-forest/roc_auc_overview

reports/observed_metrics.csv contains actual metrics, uncertainties, and artifact paths. reports/predictions/ contains sample IDs, encoded labels, Case probabilities, and predictions at 0.5. reports/fitted_parameters.json records the selected full-cohort pipeline settings.

Reference point estimates, seed 420:

Internal ROC-AUC/AP values are means across outer folds, not metrics calculated from pooled scores. The exported table reports both. External metrics evaluate one fixed full-cohort model. Tuning improved some internal estimates but did not improve external AP in this run. This is useful: tuning on training data cannot guarantee better generalization, especially after a distribution shift. Do not select the deployment model by repeatedly inspecting this external set.

The tutorial uses AP consistently: it is not interchangeable with trapezoidal area under a plotted precision–recall curve. The baseline AP of a random ranking is approximately the positive-class prevalence, here 0.5.

11. Repeat CV to examine stability

bash scripts/run_repeated.sh random-forest

This executes seeds 420, 421, and 422 on the same noisy dataset, stores them in results/random-forest/repeated_noisy/, and aggregates them using configs/plots/random-forest/07_repeated_noisy.yaml. It skips unnecessary full-cohort model fitting for these repeat-only runs.

Equivalent model commands, showing the relevant overrides:

for seed in 420 421 422; do
  phipml -c configs/models/random-forest/02_noisy.yaml --seed "$seed" \
    --output-dir ../../../results/random-forest/repeated_noisy \
    --no-only-train-model
done
phipml-plot --plot-config configs/plots/random-forest/07_repeated_noisy.yaml

The plot recipe keeps each run's top 10 features, then displays leading features that occur in at least 50% of runs. These three repeats demonstrate the mechanics; they are not enough to characterize feature stability precisely. More repeats reuse the same participants and must not be treated as independent studies.

Use one estimator/experiment per glob. To compare RF against XGBoost, use separate panels or a heatmap, not repeated-run aggregation.

12. Reuse the saved fitted model

After 03_noisy_tuned, validate its saved full-cohort model without another search/refit:

phipml -c configs/models/random-forest/08_reuse_noisy.yaml

The key settings are:

classification:
  run_nested_cv: false
  only_train_model: false
  use_pretrained: true
  input_dir: ../../../results/random-forest/03_noisy_tuned
  input_name: training_random-forest_03_noisy_tuned_420.joblib

The complete recipe includes the training input context and external filters, and saves validation_random-forest_08_reuse_noisy_420.joblib under results/random-forest/08_reuse_noisy/. Use the matching environment when loading serialized fitted estimators. The XGBoost equivalent is supplied too.

13. Adapt this workflow to your own data

1. Copy a supplied model YAML to another file in the same model-config folder. Replace the input paths, sample/target column names, class order, matrix orientation, and peptide prefixes. Check sample IDs and class counts.
2. Choose numeric clinical extras. Handle genuine missingness and categorical encodings deliberately; fillna_value: 0 is not a general solution.
3. Define disjoint training/validation cohort filters and choose the split design before inspecting external performance. This CLI uses stratified sample-level CV, not participant-group CV.
4. Start with param_grid: {}. Then copy the corresponding estimator's tuning space, revise it for your data size, and increase the search budget if useful.
5. Set distinct output_dir, output_name, and validation names. Fit the model.
6. Copy the matching plot YAML; point results to the new evaluation artifact, set config, select train/test, and supply annotation column names that actually exist. Plot without retraining.
7. Preserve inputs, configs, source commit, dependency versions, predictions, and evaluation outputs together.

Troubleshooting

Further implementation details are linked from the three reference documents. This tutorial's observed results are synthetic examples for learning the tool, not estimates of diagnostic performance on patient data.

Model configuration reference

This reference describes the implemented CLI at phipml 4.2.0, commit 932f14f. Start with a complete YAML in configs/models/; use this page when adapting it.

The three parts of a model YAML

Use spaces for YAML indentation, native true/false, null for no value, and lists in brackets or with hyphens. Quoted "false" is a string, not a Boolean. Class and column names must match the input files, including capitalization.

Path rule: model YAML paths are relative to the YAML's own directory. classification.output_dir and input_dir CLI overrides follow this same rule. In a plotting YAML, paths are relative to that plotting YAML; explicit plotting CLI paths are relative to your terminal's current directory. Absolute paths work in either case.

Explicit model CLI values override the classification: section, which overrides internal defaults. classification.seed overrides random_state. Avoid adding unrecognized configuration keys: nested mappings can contain keys that are not consumed by the CLI.

Inputs and target: top-level keys

The loader intersects sample IDs between metadata and the matrix. A typo can therefore remove samples without meaning that the cohort definition was right. Check sample counts before fitting and the Final training data log message.

Alternative input: one enrichment file per sample

The main examples use combined matrices. For per-sample enriched-peptide lists, replace only the data-loading part of your YAML:

data_input: ../../../my_data/enriched_samples
data_input_mode: sample-files
sample_file_patterns: ["*.csv"]
sample_file_peptide_column: ID
sample_name_regex: null

A tab-delimited manifest can list sample_name and path columns, or one path per line. Paths in a sample manifest are relative to that manifest. An explicit sample name takes precedence over the filename-derived name.

Execution: classification: keys

Model and run type

Features and preprocessing

The peptide pipeline removes constant columns, then fits an elastic-net logistic-regression selector. Clinical variables bypass that peptide selector. Feature selection and optional continuous imputation are fitted inside CV pipelines. The current CLI applies prevalence filtering to the full training cohort before outer CV, although it excludes external samples. The tutorial uses 0–100% to make this filter inactive. Do not describe a restricted prevalence filter as being fitted separately inside every CV fold in this version.

Sex/Gender values F/Female/M/Male and 0/1 are recognized. Other categorical clinical variables require numeric encoding before fitting. Define encodings using training data and apply the same schema externally. Genuine missing measurements should not automatically become biological absences.

CV, tuning, and uncertainty

The search score is average precision in this commit. It is not a YAML option. XGBoost's built-in eval_metric="auc" does not change the search scorer. Class stratification preserves class proportions; it does not group repeated samples from one participant. This CLI has no participant-group CV setting. The mock data represent independent samples.

For a larger exploratory run, a reasonable starting budget is 5 outer folds, 3 inner folds, and 30 candidates, then repeated seeds if compute permits. Choose the split design and sample sizes for the scientific study; more repeats do not create more independent participants. Never select a seed or tuning recipe by its external-cohort result.

Cohorts and optional hold-out splitting

classification:
  train_filters: {cohort: training}
  run_nested_cv: false
  only_train_model: false
  validation_sets:
    - name: independent_site
      filters: {cohort: external}

Keep participants and any repeated samples disjoint between cohorts. Labeling rows external does not make them independent if they are copied from training.

What this version supports for external files

At the inspected commit, validation_sets accepts name and filters, not a separate per-cohort data_input or metadata_input. The six examples therefore use a combined matrix and metadata with disjoint training/external IDs. The training filter excludes external rows from fitting and tuning.

If a new cohort arrives in separate files, combine compatible tables first (with unique sample IDs, a cohort column, and checked feature/metadata schemas) and then use the filters above. Or adapt the Python API explicitly. The current pretrained CLI still prepares its training input context, so these reuse recipes retain the training rows; an external-only CLI workflow is not documented here as if it were already implemented.

Useful command overrides

Run these from the tutorial directory:

# Same noisy dataset, a different CV seed, dedicated output folder.
phipml -c configs/models/random-forest/02_noisy.yaml --seed 421 \
  --output-dir ../../../results/random-forest/repeat_421

# A longer search. Output is kept separate from the reference examples.
phipml -c configs/models/xgboost/03_noisy_tuned.yaml \
  --n-iter 30 --inner-cv-splits 3 --outer-cv-splits 5 \
  --output-dir ../../../results/xgboost/noisy_longer_search

# Untuned clinical-only baseline, on the same target/cohort/CV seed.
phipml -c configs/models/random-forest/02_noisy.yaml \
  --no-with-oligos --with-additional-features \
  --output-name clinical_only --output-dir ../../../results/random-forest/clinical_only

# Untuned peptides-only comparison.
phipml -c configs/models/random-forest/02_noisy.yaml \
  --with-oligos --no-with-additional-features \
  --output-name peptides_only --output-dir ../../../results/random-forest/peptides_only

For a tuned estimator change, use the supplied estimator-specific YAML. Merely passing --model-type xgboost to a Random Forest tuning YAML leaves its selected grid pointing at Random Forest parameters unless it is also replaced.

Source: the pinned repository's data/config loader, run settings, and training CLI.

What the hyperparameters mean

phipml fits a pipeline: peptide preprocessing/selection, optional clinical imputation, then a Random Forest or XGBoost classifier. A hyperparameter is a setting chosen before fitting; training then estimates model coefficients or trees from the training samples. The tuned examples choose hyperparameters by inner cross-validation, using average precision (AP).

These are small teaching search spaces, not claimed optimal settings for PhIP-seq. The same noisy data, target, cohort filters, and seed are used for untuned/tuned pairs. Better training or inner-CV performance does not guarantee better outer-CV or external performance.

Read a parameter path

param_grid:
  random-forest:
    estimator__max_depth:
      type: integer
      low: 2
      high: 10

classification.param_grid_name: random-forest selects this section. estimator__max_depth means “the max_depth parameter of the pipeline step called estimator.” Double underscores traverse nested pipeline objects.

The longer path preprocessor__peptides__feature_selection__estimator__C changes the logistic regression used to select peptides, not the final Random Forest/XGBoost.

Do not use equal low/high to hold a value fixed. Use a one-element categorical space, e.g. {type: categorical, categories: [100]}. This still invokes the search machinery if param_grid is nonempty. The current CLI does not expose a separate estimator_params block for fixed overrides. param_grid: {} uses the built-in pipeline and estimator defaults; it does not disable feature selection.

Shared peptide selector

Both models fit elastic-net logistic regression with solver: saga, then retain peptides whose absolute coefficient meets the selection threshold.

Fixed implementation settings are VarianceThreshold(threshold=0.0), selector threshold=1e-5, and logistic-regression max_iter=10000. The selector is supervised and learns from class labels inside the fitted CV pipeline. Clinical features do not pass through this peptide selector. Inactive peptide search parameters are ignored when no peptide branch exists.

Feature selection can choose different peptides in different folds. The saved SHAP matrices retain original feature names and assign zero to unused features for the relevant fitted model/fold. A zero SHAP value is not proof of no biological association.

Random Forest

A Random Forest averages many randomized decision trees. Each tree can capture nonlinear patterns; averaging reduces the sensitivity to any one fitted tree.

Use the exact final-model prefix estimator__ for every parameter below.

Other supported Random Forest parameters can be added through a valid pipeline search path. Common optional choices:

The balanced mock classes are included to keep the demonstration easy to read; class weighting is present to teach its meaning, not because imbalance needs correction in these data.

XGBoost

XGBoost builds trees sequentially. Each boosting round adds a correction to the current model. Smaller learning steps often need more trees. The phipml factory sets objective="binary:logistic", tree_method="hist", eval_metric="auc", n_jobs=1, and the configured seed. Other untuned settings come from the installed XGBoost version. There is no early-stopping evaluation set supplied by this CLI.

Optional estimator__scale_pos_weight increases the training weight assigned to the positive class. If exploring it, use training-fold information and validate its effect. It is omitted here because the synthetic classes are balanced. Do not add class_weight to an XGBoost grid or learning_rate to a Random Forest grid.

How tuning is evaluated

For 03_noisy_tuned:

1. Split the training cohort into three outer folds.
2. For each outer fold, run 12 candidate configurations using two inner folds on the outer-training subset; refit the best candidate on that subset.
3. Predict and explain the held-out outer subset exactly once.
4. Combine the outer-fold results into the nested artifact.
5. Run a fresh inner search on all 160 training samples and save the resulting full-cohort pipeline for reuse.

This costs approximately 3 × (12 × 2 + 1) + (12 × 2 + 1) = 100 pipeline fits. It is a teaching budget for a fairly wide space. BayesSearchCV in the pinned scikit-optimize version starts with 10 initialization candidates by default; 12 candidates allow a short adaptive phase. A two-candidate smoke run exercises the search workflow but cannot demonstrate a mature Bayesian optimization.

For 06_external_noisy_tuned, there is one inner search on the full training cohort, followed by evaluation of its fixed selected model on 80 external samples. External labels do not choose parameters, features, or thresholds.

Inspect what was actually chosen:

python scripts/summarize_results.py

Open reports/fitted_parameters.json for full-cohort pipeline settings, or:

from pathlib import Path
import joblib

path = Path("results/random-forest/03_noisy_tuned/training_random-forest_03_noisy_tuned_420.joblib")
model = joblib.load(path)["best_estimator"]
print(model.named_steps["estimator"].get_params())
selector = model.named_steps["preprocessor"].named_transformers_["peptides"].named_steps["feature_selection"]
print(selector.estimator_.get_params())

Outer-fold selected settings can differ from this full-cohort model; they are available from model_list in the nested artifact. The CLI saves the chosen models, not the complete BayesSearchCV.cv_results_ optimization history.

Implementation source: pipeline, selector and search. Parameter semantics were checked against the installed scikit-learn 1.5.2 and XGBoost 2.1.4 classes used for the reference run. Further reference: RandomForestClassifier, LogisticRegression, XGBoost parameters.

Plot configuration reference

Use phipml-plot --plot-config PATH.yaml for a saved recipe. Model fitting and plotting are separate commands: change colors or displayed annotations without fitting models again. Use phipml-heatmap with a CSV/TSV manifest to compare experiments. This guide describes the current public commands, not the older plotting modules retained in the repository.

Start with the right artifact

split: train is a storage key for the training cohort's out-of-fold evaluation. It does not mean the model is evaluated on samples it just fitted. For the internal examples the file contains one held-out prediction for each of the 160 training-cohort samples.

The files embed targets, feature names, SHAP values, metrics, and data-loading context. They do not contain a portable copy of every original input file. Use plotting.config to point to the local model YAML after moving the folder; the supplied recipes already do this. The original data are still required for colored feature-value SHAP displays and prevalence/clinical summaries.

A complete plotting recipe

This example is stored at configs/plots/random-forest/02_noisy.yaml.

plotting:
  results:
    - ../../../results/random-forest/02_noisy/nested_random-forest_02_noisy_420.joblib
  config: ../../models/random-forest/02_noisy.yaml
  split: train
  plots: [all]
  formats: [pdf, svg, png]
  dpi: 180
  output_dir: ../../../plots/random-forest/02_noisy
  output_prefix: 02_noisy
  title: "random-forest | 02 noisy"
  class_labels: [Control, Case]

  max_display: 10
  feature_ranking: mean-abs-shap
  ranking_top_k: 10
  min_top_k_frequency: 0
  shap_alignment: strict
  save_standalone: true
  reconstruct_data: true

  library_metadata: ../../../data/peptide_library.csv
  library_id_column: peptide_id
  validation_bootstraps: 0
  random_state: 420
  class_colors: ["#386CB0", "#D35D4D"]
  colors:
    roc: "#176B73"
    roc_band: "#B3DAD8"
    pr: "#8A5A44"
    pr_band: "#DFC5B8"
    confusion_cmap: Blues
    classification: "#52796F"
    shap_cmap: phipml_blue_gray_red
    shap_heatmap_cmap: phipml_purple_gray_orange
    shap_importance: "#6D597A"
  feature_table:
    annotation_columns: [Description, Role]
    extra_columns: []
    title: "Synthetic features: model importance and known roles"
    header_color: "#DCE9E9"
    row_colors: ["#F5F8F8", white]
    prevalence_cmap: phipml_prevalence

phipml-plot --plot-config configs/plots/random-forest/02_noisy.yaml

The figure files and feature table appear under plots/random-forest/02_noisy/. Relative YAML paths are resolved from the plotting YAML. Explicit CLI paths are resolved from the current terminal directory, and override YAML values.

Inputs, output, and figure selection

What each plot answers

All nine figure types are generated by plots: [all]. The table CSV retains more features/annotations/audit fields than the compact rendered table.

Feature ranking and repeated runs

Only aggregate runs with the same estimator, data, cohort, feature definition, class encoding, and threshold. Do not use a broad glob that mixes Random Forest, XGBoost, noisy/perfect data, training/external cohorts, or tuned/default experiments. Use separate plots or a heatmap manifest for those comparisons.

The repeated plot recipe uses a dedicated repeated_noisy folder, and three seeds on the same dataset. Signed SHAP values are averaged by sample/feature; ranking/stability statistics also retain run-level absolute importance. Top-K frequency differs from how often the elastic-net selector retained a feature. Both statistics can be inspected in the exported CSV.

Annotations and appearance

Quote hex colors because unquoted # begins a YAML comment. Use valid Matplotlib colormaps or the supplied phipml_blue_gray_red, phipml_purple_gray_orange, and phipml_prevalence maps. Annotation values in these examples are invented and have no biological meaning.

Supplying plotting data explicitly

These optional keys are useful for older artifacts or another controlled data source. They are not needed for the included recipes.

Saved sample IDs determine alignment. The supplied model-config reconstruction route also checks reconstructed labels against saved targets. This check cannot prove all feature measurements are unchanged; preserve input data and hashes.

Common plotting commands

From the tutorial root:

# Change appearance/format without retraining.
phipml-plot --plot-config configs/plots/random-forest/02_noisy.yaml \
  --formats svg png --dpi 300 --max-display 8 \
  --output-dir plots/custom_noisy --output-prefix presentation

# Only performance panels; original feature data are not required.
phipml-plot results/random-forest/02_noisy/nested_random-forest_02_noisy_420.joblib \
  --split train --plots performance --no-reconstruct-data \
  --class-labels Control Case --output-dir plots/performance_only

# External model evaluation.
phipml-plot --plot-config configs/plots/xgboost/06_external_noisy_tuned.yaml

# Regenerate a compact table after manually editing a COPY of its exported CSV.
phipml-plot --plot-config configs/plots/random-forest/02_noisy.yaml \
  --plots feature-table \
  --feature-importance-table plots/random-forest/02_noisy/02_noisy_feature_importance.csv \
  --output-dir plots/curated_table --max-display 8

Editing a feature table changes presentation; it does not revise SHAP values or model predictions. Display a curated subset transparently.

Comparison heatmaps

phipml-heatmap takes a manifest, not a plotting YAML. Each row identifies an evaluation artifact:

training,validation,path,split
RF noisy default,Internal CV,../results/random-forest/02_noisy/nested_random-forest_02_noisy_420.joblib,train
RF noisy default,External,../results/random-forest/05_external_noisy/validation_random-forest_05_external_noisy_420.joblib,test

The path is relative to the manifest file. The training value labels a heatmap column and validation labels a row; labels can describe experiment conditions as in the supplied manifests. Multiple rows for one cell are aggregated as repeated run estimates. Use explicit files, not globs, in the manifest.

phipml-heatmap --manifest manifests/random-forest.csv \
  --metric roc.auc --palette viridis --vmin 0 --vmax 1 --dpi 180 \
  --title "Random Forest: internal and external performance" \
  --output plots/random-forest/roc_auc_overview

phipml-heatmap --manifest manifests/xgboost.csv \
  --metric pr.ap --palette viridis --vmin 0 --vmax 1 --dpi 180 \
  --output plots/xgboost/ap_overview

The rectangular overview uses only conditions that exist. An internal-CV cell and an external cell summarize different evaluation procedures; their numbers are not a paired statistical test.

Interpreting uncertainty and SHAP honestly

The perfect synthetic scenario can have a 1.0–1.0 bootstrap interval because every resampled set remains perfectly separated. This is a property of the constructed demonstration, not evidence that a clinical model has no uncertainty.

Positive SHAP contributions move the model output toward the positive class; negative values move it away. Random Forest tree SHAP is in its probability output scale here, while XGBoost's default tree SHAP explains the raw margin (log-odds for the binary logistic objective). Do not compare their numerical SHAP magnitudes directly or describe XGBoost values as probability-point changes. SHAP describes a fitted predictive relationship, not causation or a p-value. Correlated engineered peptides may share importance unevenly, and independent noise can acquire apparent importance in a finite sample.

Source: the pinned repository's plot CLI, result aggregation, and heatmap CLI.

Reference execution and observed results

Run date: 8 September 2026. Source: phipml 4.2.0, commit 932f14ffca41ba4f24f3cc570afea27265a5893c. The upstream source checkout was not modified.

Environment and budget

The run installed the pinned source into an isolated Linux Python 3.12.13 environment with the dependency ranges declared in pyproject.toml. Main versions: numpy 1.26.4, pandas 2.2.3, scipy 1.15.3, scikit-learn 1.5.2, scikit-optimize 0.10.2, XGBoost 2.1.4, SHAP 0.46.0, matplotlib 3.10.9, and seaborn 0.12.2. The full platform-specific inventory is in reports/requirements-reference.txt; it is distinct from the repository's Micromamba specification. The Micromamba, Docker, and Apptainer installation routes were inspected in upstream documentation, not separately executed here.

Data seed 20260908; model seed 420. Three outer folds, two inner folds, 12 candidates in each tuned search, one worker at each level, 500 external bootstrap draws, confidence level 0.95, classification threshold 0.5.

The complete twelve-scenario command, including per-scenario plots, ran in approximately five minutes on this environment, excluding dependency installation. This is an observation, not a runtime promise. The model-event log spans 4 minutes 36 seconds, with the final plot/heatmap/export work afterward. No outcome-dependent revisions were made to the generated data or tuning space.

What was run

python scripts/generate_examples.py
bash scripts/run_all.sh both
bash scripts/run_repeated.sh random-forest
phipml -c configs/models/random-forest/08_reuse_noisy.yaml
phipml -c configs/models/xgboost/08_reuse_noisy.yaml
python scripts/summarize_results.py

The main run covers 12 estimator/scenario combinations, each producing nine figure types in PDF/SVG/PNG plus the feature table CSV. Both estimator heatmaps were generated for ROC-AUC and AP. The optional repeated-run example was executed for Random Forest; the parallel XGBoost repeat recipe is supplied but was not separately executed. Pretrained model reuse was executed for both.

Metrics

Internal entries show mean ± sample SD across outer folds. External entries show the point estimate followed by the 95% stratified bootstrap interval. These are different uncertainty summaries and should not be read as equivalent 95% intervals. All participants here are synthetic.

Tuning did not improve external AP for either estimator in this run. The external population has a deliberately weaker signal. These results illustrate why external data must stay out of model selection; they are not evidence that tuning is generally unhelpful or that one estimator is superior for real PhIP-seq data.

The CSV additionally records pooled-score ROC-AUC/AP, accuracy, and F1. For internal evaluation, pooled-score metrics are not necessarily equal to the mean of the fold metrics. Prediction CSVs use the pooled sample-level scores.

Verification performed

• All generated CSVs match their recorded SHA-256 hashes.
• Training and external sample IDs are disjoint in both datasets.
• Every full-cohort artifact contains exactly the 160 intended training labels.
• Every nested artifact covers all 160 intended training samples, with each sample assigned to exactly one outer validation fold per run.
• Every external artifact covers exactly the 80 intended external samples.
• Saved labels agree with the input metadata, probabilities are finite/in range, and SHAP matrices contain finite values.
• Every core recipe generated all 27 requested figure files and its feature CSV.
• Representative performance, feature-table, and XGBoost SHAP figures were visually inspected for legibility and clipping.
• Reusing the saved full-cohort tuned model reproduced scenario 06 predictions and SHAP values for both estimators.
• Plotting was exercised from a relocated folder using an explicit local model YAML, with deliberately invalid original embedded paths in a temporary test artifact. Performance, SHAP beeswarm, and feature-table reconstruction passed.

Detailed checks are recorded in reports/verification_checks.json. The normal training/plotting logs are in logs/. The model/config generator, inputs, search spaces, artifacts, plot recipes, exported predictions, and observed metrics are included so that the walkthrough can be independently rerun.

Scope of interpretation

The perfect data intentionally encode their target in engineered predictors. All annotations are simulated. The noisy scenario is a controlled learning exercise, not a biological benchmark. The current phipml prevalence filter is applied before outer CV on the training cohort; it is inactive here at 0–100%. The current CLI uses sample-level stratified folds, so participant-group cross-validation would require additional support for repeated-measure studies.

These notes describe observed execution and current limitations; optional longer-search, clinical-only, and per-sample-file recipes are explanatory examples rather than additional completed benchmark runs.