Statistics and Machine Learning Toolbox: Getting started

Train your first classifier, evaluate accuracy, tune hyperparameters, switch model families, and diagnose model health.

← Statistics and Machine Learning Toolbox overview · Modeling guide · Methods guide · Examples · Reference

What you will learn

This guide walks you through using the Statistics and Machine Learning Toolbox workbook from initial baseline execution to advanced hyperparameter tuning and model diagnostics.

By the end of this guide, you will be able to:

  • Navigate the workbook architecture across control, data, config, and reporting sheets;
  • Run and interpret the baseline binary classification model (Customer Churn);
  • Switch effortlessly between classification, regularized regression, clustering, and PCA models without touching code;
  • Tune algorithmic hyperparameters (C, alpha, n_clusters, max_depth, n_components) directly on the Model Config sheet;
  • Configure feature normalization strategies (Standard, MinMax, None) and evaluate their impact on conditioning;
  • Adjust train/test holdout partitions (Test size) and verify sample-level inferences;
  • Interpret cross-validation stability across 2 to 5 folds;
  • Read and remediate warnings or failures from the 7-Check Automated Diagnostics Suite.

Prerequisites and workbook tour

Open the workbook in Excel or Excel for the Web with Python support enabled. The workbook contains six dedicated worksheets divided into Input/Configuration and Review/Inference layers.

Input and configuration sheets

  • Start — High-level model controls (Active model, Test size, Random seed, CV folds, Scale features) and the included model library catalog.
  • Data — Observation tables containing numeric features (Feature1 through Feature4) and ground-truth Target values labeled by Model ID.
  • Model Config — Algorithm-specific hyperparameters with default values, mathematical descriptions, and bounds.

Review and inference sheets

  • Results — Training summary (rows 5–16: status, sample counts, primary/secondary/tertiary metrics, cross-validation mean) and ranked feature importance/coefficients (columns D–I).
  • Predictions — Sample-level predictions, classification probabilities, regression residuals, cluster distances, train/test holdout tags, and hit/miss indicators.
  • Diagnostics — The 7-check statistical health audit flagging data quality, convergence, performance, stability, multicollinearity, balance, and outliers.

[!NOTE] Blue-shaded cells indicate authored user inputs and parameter overrides. Green-shaded cells indicate dynamic notebook publications returned by BF.OUTPUT functions.

First solve: Customer Churn Classification

The default active model is customer_churn_logistic, predicting subscriber churn from customer usage and billing behavior.

1. Verify model controls on Start

Inspect the Model controls table on Start!A5:B10:

Cell Control Value Description
Start!B6 Active model customer_churn_logistic Selects the active model from the library
Start!B7 Test size 0.25 25% of observations partitioned into holdout test set
Start!B8 Random seed 42 Pseudo-random seed for reproducible data splitting and optimization
Start!B9 CV folds 3 3-fold cross-validation for generalization estimation
Start!B10 Scale features Standard Z-score standardization (\(z = (x - \mu) / \sigma\))

2. Inspect observation data on Data

Navigate to the Data sheet. The table contains 16 historical subscriber records for customer_churn_logistic:

  • SampleID: Unique customer record identifier (1 to 16).
  • Feature1 (Usage): Monthly talk/data minutes (e.g., 80 to 450 minutes).
  • Feature2 (Tenure): Account tenure in months (e.g., 2 to 60 months).
  • Feature3 (Complaints): Customer support complaints filed (0 to 4).
  • Feature4 (MonthlySpend): Monthly bill amount in USD ($40 to $115).
  • Target (Churn): Binary ground-truth indicator (0 = Retained, 1 = Churned).

3. Review model summary and coefficients on Results

Open the Results sheet. The Marimo engine computes the logistic regression model and populates the summary:

Summary block (Results!A5:B16)

Metric Value Meaning
Active model Customer Churn — Logistic Regression Model description label
Task Classification Supervised binary classification
Status OPTIMAL Solver converged successfully
Algorithm LogisticRegression Scikit-learn L2-penalized estimator
Total samples 16 Number of active observations
Feature count 4 Number of predictor columns
Primary metric 1.000000 Overall training accuracy (100.0%)
Secondary metric 1.000000 F1 score (harmonic mean of precision & recall)
Tertiary metric 1.000000 Area Under the ROC Curve (ROC-AUC)
CV score mean 0.944444 3-fold stratified cross-validation accuracy
Scaling method Standard Z-score normalization applied
Runtime (s) < 0.05 Computation execution time

Coefficients block (Results!D5:I10)

The notebook ranks features by absolute importance:

Feature Weight (\(w_j\)) OddsRatio (\(\exp(w_j)\)) AbsImportance (\(|w_j|\)) Rank Significance
Feature3 1.135471 3.1127 1.135471 1 High
Feature4 0.874974 2.3988 0.874974 2 High
Feature2 -0.801402 0.4487 0.801402 3 High
Feature1 -0.174595 0.8398 0.174595 4 Moderate

[!TIP] Interpreting Odds Ratios: An odds ratio of 3.1127 on Feature3 (Complaints) means that for each 1-standard-deviation increase in complaints, the odds of customer churn increase by approximately 211% (\(3.1127 - 1\)). Conversely, an odds ratio of 0.4487 on Feature2 (Tenure) indicates that longer customer tenure cuts the odds of churning by more than half.

4. Inspect sample inferences on Predictions

Navigate to Predictions!A5:F21. Rows display individual customer scores:

SampleID Actual Predicted Probability Split Correct
1 0 0 0.1176 Train YES
2 1 1 0.9632 Train YES
3 0 0 0.0815 Train YES
4 1 1 0.9941 Train YES
… … … … … …
  • Probability: Predicted probability of the positive class (\(P(\text{Churn} = 1)\)).
  • Split: Indicates whether the observation was used for model training (Train) or held out for unbiased verification (Test).
  • Correct: Displays YES if prediction matches actual label, NO otherwise.

5. Validate the 7-Check Diagnostics Suite

Switch to the Diagnostics sheet. All 7 statistical health checks report PASS:

Check Severity Detail
Data validation PASS Sample adequacy confirmed: n=16 > p=4 features, 0 missing values.
Convergence & configuration PASS Algorithm LogisticRegression converged; hyperparameters within valid bounds.
Model performance PASS Primary metric (Accuracy) = 1.0000 meets threshold (>= 0.80).
Generalization & stability PASS Generalization gap |train - CV| = 0.0556 (<= 0.15 indicates stability).
Multicollinearity & conditioning PASS Condition number kappa = 3.64 (< 30 indicates well-conditioned features).
Target & class balance PASS Minority class proportion is 37.5% (>= 25% balanced).
Outlier detection PASS 0 observation(s) flagged as potential outliers (|z| > 2.5 sigma).

Switching across model families

You can switch the entire statistical engine to another analytical task by changing a single cell: Start!B6. The dropdown provides five options:

graph TD
    Start["Start!B6 Dropdown"] --> M1["customer_churn_logistic\n(Logistic Regression)"]
    Start --> M2["housing_regression_ridge\n(Ridge Regression)"]
    Start --> M3["iris_clustering_kmeans\n(K-Means Clustering)"]
    Start --> M4["credit_default_tree\n(Decision Tree Classifier)"]
    Start --> M5["feature_pca\n(Principal Component Analysis)"]
  1. housing_regression_ridge (Property Valuation)
    • Predicts continuous residential prices from square footage, bedrooms, age, and distance.
    • Sets primary metric to \(R^2 = 0.986518\), secondary metric to RMSE = \(14.398038\), and tertiary metric to MAE = \(10.988185\).
    • Predictions!Probability column displays the regression residual (\(y_i - \hat{y}_i\)).
  2. iris_clustering_kmeans (Botanical Segmentation)
    • Unsupervised clustering of flower morphological measurements into 3 clusters.
    • Primary metric evaluates cluster coherence via Silhouette Score = \(0.540379\); secondary metric reports Inertia = \(11.238210\).
    • Predictions!Probability column displays the Euclidean distance to the assigned cluster centroid (\(1.0623\) for Sample 1).
  3. credit_default_tree (Credit Risk Classification)
    • Fits an interpretable decision tree on applicant financial profiles.
    • Primary metric achieves Accuracy = \(1.000000\); CV Mean = \(0.722222\).
    • Top feature is Feature4 (CreditScore) with Gini importance = \(1.000000\).
  4. feature_pca (Dimensionality Reduction)
    • Decomposes 4 correlated signal features into orthogonal principal axes.
    • Primary metric indicates Total Explained Variance = \(1.000000\) (\(99.96\%\) in PC1).
    • Inferences display projected coordinates on principal component axes.

Tuning hyperparameters on Model Config

Hyperparameters control model complexity, regularization penalties, and tree depths. Adjust them on the Model Config worksheet:

Model ID Hyperparameter Default Valid Range Role & Effect
customer_churn_logistic C 1.0 \([10^{-4}, 10^4]\) Inverse regularization strength. Smaller \(C\) increases penalty, shrinking weights.
customer_churn_logistic max_iter 200 \([10, 5000]\) Solver iteration limit.
housing_regression_ridge alpha 1.0 \([10^{-5}, 10^5]\) L2 Tikhonov regularization penalty. Larger \(\alpha\) shrinks coefficients toward zero.
iris_clustering_kmeans n_clusters 3 \([2, \min(n-1, 10)]\) Number of cluster centroids to partition.
credit_default_tree max_depth 3 \([1, 20]\) Maximum tree depth. Lower depth prevents overfitting.
feature_pca n_components 2 \([1, \min(n, p)]\) Number of principal axes to extract.

Try tuning Ridge \(\alpha\)

Change Model Config!C7 from 1.0 to 10.0 for housing_regression_ridge: - \(R^2\) changes from 0.986518 to 0.935190. - Top feature weight Feature1 shrinks from 60.173745 down to 29.944689. - The increased penalty suppresses coefficient variance and reduces overfitting risk.

Configuring feature scaling and holdouts

The toolbox provides two critical modeling switches on Start:

1. Feature scaling (Start!B10)

Select between three standard normalization modes: - Standard (Recommended): Centers columns to \(\mu = 0\) and scales to unit variance \(\sigma = 1\). Essential for distance-based models (Ridge, K-Means, Logistic Regression, PCA). - MinMax: Scales features into \([0, 1]\). Preserves exact zeros and bounds. - None: Uses raw unscaled features. > [!WARNING] > When running customer_churn_logistic with None, feature scales diverge, causing the condition number check to flag FAIL (\(\kappa > 100\)) and CV accuracy to drop from 0.944444 to 0.755556.

2. Train / test holdout ratio (Start!B7)

Set Test size between 0.0 and 0.49: - At 0.25, 25% of samples are assigned to the holdout test set (Split = Test). - For classification models, the split is stratified, guaranteeing proportional representation of classes in both train and test partitions. - At 0.0, all observations are used for model fitting (Split = Train).

Evaluating cross-validation folds

Set CV folds on Start!B9 (allowed range: 2 to 5 folds):

  • Cross-validation partitions the dataset into \(K\) disjoint folds, training on \(K-1\) folds and evaluating out-of-fold performance on the held-out fold.
  • The CV score mean on Results!B15 reflects true out-of-sample generalization.
  • Setting CV folds = 4 on the baseline churn model yields 0.937500 mean accuracy with stable convergence.

Interpreting the 7-Check Diagnostics Suite

The diagnostics engine runs seven automated verification checks on every solve:

flowchart TD
    D1["1. Data Validation\n(n > p, 0 missing)"] --> D2["2. Convergence & Config\n(OPTIMAL & in-bounds)"]
    D2 --> D3["3. Model Performance\n(Metric >= threshold)"]
    D3 --> D4["4. Generalization Gap\n(|Train - CV| <= 0.15)"]
    D4 --> D5["5. Multicollinearity\n(Condition kappa < 30)"]
    D5 --> D6["6. Class Balance / Normality\n(Minority >= 25% / Shapiro p >= 0.05)"]
    D6 --> D7["7. Outlier Detection\n(|z| <= 2.5 sigma)"]

Remediation guidance

  1. Data validation WARN / FAIL: Impute missing values or provide at least \(p + 1\) samples.
  2. Convergence FAIL: Increase max_iter on Model Config or verify hyperparameter bounds.
  3. Model performance WARN / FAIL: Re-examine feature relevance or reduce excessive regularization penalties.
  4. Generalization FAIL: A large gap between training metric and CV score indicates overfitting. Prune tree depth, increase regularization (\(\alpha\) or \(1/C\)), or gather additional training records.
  5. Multicollinearity FAIL: High condition number (\(\kappa \ge 100\)). Enable Standard scaling on Start!B10, remove redundant correlated features, or use PCA.
  6. Class balance WARN / FAIL: Severe class imbalance. Collect additional minority class examples or adjust decision thresholds.
  7. Outliers WARN / FAIL: Inspect rows on Predictions flagged with large residuals or probability errors; investigate measurement errors or data anomalies.