Curve Fitting Toolbox: Reference

Complete reference specification covering workbook controls, input and output schemas, diagnostic check thresholds, numerical tolerances, and troubleshooting.

← Curve Fitting Toolbox overview · Getting started · Modeling guide · Methods guide · Examples

Control panel reference (Start!A5:B10)

The Start worksheet hosts the top-level execution parameters controlling the nonlinear optimization process:

Cell	Control label	Data type	Allowed values	Default	Description & Solver behavior
B6	Active model	Dropdown	Valid Model ID in Start!D5:D20	michaelis_menten	Sets the model equation to optimize. Filters the observation rows in Data and parameter entries in Parameters.
B7	Loss function	Dropdown	Linear, Huber, Soft L1, Cauchy	Linear	Selects the residual penalty function \(\rho(z)\). Linear minimizes squared residuals; Huber, Soft L1, and Cauchy progressively down-weight outliers.
B8	Confidence level	Dropdown	0.90, 0.95, 0.99	0.95	Sets the confidence coefficient \((1 - \alpha)\) for parameter confidence intervals and pointwise curve bands (\(t_{\alpha/2, \nu}\)).
B9	Eval grid points	Integer	\(10 \le N \le 50\)	50	Number of equidistant evaluation points generated between \(X_{\min}\) and \(X_{\max}\) on the Fitted Curve sheet.
B10	Robust outliers	Dropdown	Auto, Disabled, Strict	Auto	Selects standardized residual threshold for Check 7: Auto (\(|z_i| > 2.5\)), Strict (\(|z_i| > 2.0\)), or Disabled (skips test).

---

Input ranges and capacity limits

The workbook layout reserves dedicated table regions designed for decision-scale scientific and engineering datasets:

Worksheet	Cell range	Header row	Maximum capacity	Purpose
Start	A5:B10	Row 5	5 configuration controls	High-level fitting controls and solver configuration
Start	D5:H20	Row 5	15 model definitions	Pre-registered model library (Model, Label, Family, Formula, Description)
Data	A5:E200	Row 5	195 observation rows	Raw experimental coordinates (\(X, Y\)) with individual \(Weight\) and \(Sigma\) values
Parameters	A5:G50	Row 5	45 parameter rows	Parameter initial guesses, lower bounds, upper bounds, and parameter locks

---

Published output schemas

The Python for Excel notebook exports four structured data blocks into Excel via bf.publish(). Each block corresponds to a styled spill range anchored in the workbook.

1. summary (Results!A5:B17)

Summary metrics providing overall fit quality and convergence status:

Row	Cell	Metric name	Typical format	Description
5	A5:B5	Table Header	Metric, Value	Column headers
6	A6:B6	Active model	Text	Long label of the fitted model
7	A7:B7	Fit status	Text	OPTIMAL or FAILED
8	A8:B8	Family	Text	Model family classification (e.g. Saturation, Decay, Sigmoidal)
9	A9:B9	Formula	Text	Mathematical equation evaluated by the solver
10	A10:B10	SSE (Sum of Squared Errors)	Number (0.00000000)	Sum of squared residuals: \(\sum (y_i - \hat{y}_i)^2\)
11	A11:B11	RMSE	Number (0.00000000)	Root mean square error: \(\sqrt{\text{SSE} / n}\)
12	A12:B12	R-squared	Number (0.0000000)	Coefficient of determination: \(1 - \text{SSE} / \text{SST}\)
13	A13:B13	Adjusted R-squared	Number (0.0000000)	Degree-of-freedom penalised \(R^2\): \(1 - (1 - R^2)\frac{n-1}{\nu}\)
14	A14:B14	AIC	Number (0.0000)	Akaike Information Criterion: \(n \ln(\text{SSE}/n) + 2p_{\text{free}}\)
15	A15:B15	BIC	Number (0.0000)	Bayesian Information Criterion: \(n \ln(\text{SSE}/n) + p_{\text{free}} \ln(n)\)
16	A16:B16	Data points	Integer	Number of valid numeric \((X, Y)\) observation pairs (\(n\))
17	A17:B17	Residual DoF	Integer	Residual degrees of freedom: \(\nu = n - p_{\text{free}}\)

2. parameters (Results!D5:J16)

Parameter estimation table supporting up to 11 parameter rows:

Column	Header	Type	Description
D	Parameter	Text	Name of the parameter (e.g. Vmax, Km, top, ec50).
E	Estimate	Numeric	Point estimate \(\hat{p}_j\) found by nonlinear least squares.
F	StdError	Numeric	Asymptotic standard error \(\text{SE}(p_j) = \sqrt{C_{jj}}\). (\(0.0\) if locked).
G	tStat	Numeric	Student-t test statistic \(t = \hat{p}_j / \text{SE}(p_j)\). (\(0.0\) if locked).
H	pValue	Numeric	Two-tailed p-value for \(H_0: p_j = 0\) given \(\nu\) degrees of freedom.
I	CI_Lower	Numeric	Lower \((1 - \alpha)\) asymptotic confidence bound: \(\hat{p}_j - t_{\alpha/2, \nu} \cdot \text{SE}(p_j)\).
J	CI_Upper	Numeric	Upper \((1 - \alpha)\) asymptotic confidence bound: \(\hat{p}_j + t_{\alpha/2, \nu} \cdot \text{SE}(p_j)\).

3. curve (Fitted Curve!A5:E55)

High-resolution curve evaluation table supporting up to 50 grid points:

Column	Header	Type	Description
A	X	Numeric	Equidistant grid evaluation coordinate across \([X_{\min}, X_{\max}]\).
B	Y_Fitted	Numeric	Predicted model response \(\hat{y}(x) = f(x, \hat{\mathbf{p}})\).
C	CI_Lower	Numeric	Pointwise \((1 - \alpha)\) lower confidence limit: \(\hat{y}(x) - t_{\alpha/2, \nu} \cdot \text{SE}(\hat{y}(x))\).
D	CI_Upper	Numeric	Pointwise \((1 - \alpha)\) upper confidence limit: \(\hat{y}(x) + t_{\alpha/2, \nu} \cdot \text{SE}(\hat{y}(x))\).
E	Derivative	Numeric	First derivative \(f'(x) = \frac{df}{dx}\) computed via finite differences.

4. diagnostics (Diagnostics!A5:C25)

Validation suite reporting statistical integrity and potential numerical pitfalls:

Column	Header	Type	Description
A	Check	Text	Diagnostic check title.
B	Severity	Text	Status rating badge: PASS, WARN, or FAIL.
C	Detail	Text	Quantitative metrics, test statistics, and actionable advice.

---

Diagnostic checks specification table

The 7-check Diagnostics Suite operates under strict mathematical criteria:

#	Check name	Pass criteria (PASS)	Warning criteria (WARN)	Failure criteria (FAIL)	Test formula & Definition	Practical interpretation
1	Model validation	\(n \ge p_{\text{free}} + 1\)	—	\(n \le p_{\text{free}}\)	\(\nu = n - p_{\text{free}} \ge 1\)	Ensures there are sufficient data points to estimate free parameters with positive residual degrees of freedom.
2	Convergence	Solver converged; all \(|p_j - \text{bound}| > 10^{-4}\)	Parameter touches bound (\(|p_j - \text{bound}| \le 10^{-4}\))	Solver error or max iterations reached	\(\min(\|p_j - l_j\|, \|p_j - u_j\|) \le 10^{-4}\)	Detects boundary hits where optimal parameters may lie outside the user-configured box bounds.
3	Goodness of fit	\(R^2 \ge 0.95\)	\(0.80 \le R^2 < 0.95\)	\(R^2 < 0.80\)	\(R^2 = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}\)	Measures fraction of observed variance explained by the model equation.
4	Jacobian conditioning	\(\kappa < 1000\)	\(1000 \le \kappa < 10000\)	\(\kappa \ge 10000\)	\(\kappa = \text{cond}(\mathbf{J} \mathbf{D}^{-1})\) where \(D_{jj} = \|\mathbf{J}_{*, j}\|_2\)	Detects parameter collinearity, overparameterization, or insufficient dynamic range in \(X\).
5	Residual normality	\(p \ge 0.05\)	\(p < 0.05\)	Solver failed or \(n < 3\)	Shapiro-Wilk \(W\) test on \(r_i = y_i - \hat{y}_i\)	Verifies Gaussian assumption of residuals. Rejection implies missing structural terms or outliers.
6	Residual autocorrelation	\(1.5 \le d \le 2.5\)	\(1.0 \le d < 1.5\) or \(2.5 < d \le 3.0\)	\(d < 1.0\) or \(d > 3.0\)	Durbin-Watson \(d = \frac{\sum (r_i - r_{i-1})^2}{\sum r_i^2}\)	Detects systematic runs of positive or negative residuals indicating curve shape mismatch.
7	Outlier detection	All \(|z_i| \le \text{cutoff}\)	Any \(|z_i| > \text{cutoff}\)	Solver failed	Standardized residual \(z_i = \frac{\|y_i - \hat{y}_i\|}{\text{RMSE}}\)	Identifies individual measurements distorting the fit (\(\text{cutoff} = 2.5\) for Auto, \(2.0\) for Strict).

---

Numerical tolerances and solver limits

The underlying Trust Region Reflective algorithm and statistical modules operate with the following default numerical tolerances:

Parameter	Setting	Impact
Maximum function evaluations (maxfev)	5,000 evaluations	Prevents infinite loops in severely non-linear or poorly conditioned landscapes.
Finite difference step size (\(h\))	\(1.0 \times 10^{-6}\)	Balances truncation error with floating-point roundoff in numerical Jacobian evaluation.
Minimum observation count	\(n \ge 3\) coordinate pairs	Required to evaluate residual variance, standard errors, and Shapiro-Wilk normality.
Bound proximity threshold	\(1.0 \times 10^{-4}\)	Parameter within \(0.0001\) of LowerBound or UpperBound triggers boundary warning.
Weight clamping floor	\(w_i \ge 1.0 \times 10^{-6}\)	Replaces zero or negative user weights to avoid division by zero in \(\sigma_{\text{eff}} = \sigma / \sqrt{w}\).
Sigma clamping floor	\(\sigma_i \ge 1.0 \times 10^{-6}\)	Replaces zero or negative user sigmas to avoid infinite weights.
Minimum evaluation grid	10 to 50 points	Clamps Eval grid points within \([10, 50]\) to fit the pre-styled Fitted Curve spill table.

---

Troubleshooting guide

1. Fit status reports FAILED

• Symptom: Results displays Fit status: FAILED with an error message in Diagnostics.
• Likely causes:
  • InitialValue estimates are too far from the physical optimum, causing TRF to encounter numeric overflow or non-finite evaluations.
  • Box bounds are inverted (LowerBound > UpperBound).
  • Coordinate pairs contain non-positive values for equations requiring \(X > 0\) (e.g. logarithmic terms in 4PL or fractional exponents in Power Law).
• Resolution:
  • Verify initial parameter estimates on the Parameters sheet using the heuristics in the Modeling guide.
  • Check that lower bounds for strictly positive parameters are set to small positive numbers (e.g. 0.01).

2. Convergence warning: “Parameter is at its bound”

• Symptom: Diagnostic check 2 flags WARN: Parameter [name] is at its bound.
• Likely causes:
  • The parameter’s true value lies outside the user-configured [LowerBound, UpperBound] window.
  • The model is overparameterized, causing the optimizer to push an unidentifiable parameter toward infinity or zero.
• Resolution:
  • Widen the parameter’s box bounds on Parameters if physically plausible.
  • If the parameter represents a known constant, set Fixed = TRUE to lock it.

3. Ill-conditioned Jacobian: \(\kappa \ge 10000\)

• Symptom: Diagnostic check 4 flags FAIL: Scaled Jacobian condition number κ = ....
• Likely causes:
  • Two or more parameters are mathematically collinear across the observed domain (e.g., trying to fit \(V_{\max}\) and \(K_m\) when all measured substrate concentrations \([S] \ll K_m\)).
  • Insufficient dynamic range in \(X\).
• Resolution:
  • Add observations across higher or lower \(X\) regions to capture both initial slopes and saturation plateaus.
  • Lock one of the correlated parameters using Fixed = TRUE.

4. Residual autocorrelation: Durbin-Watson \(d < 1.0\)

• Symptom: Diagnostic check 6 flags FAIL: Durbin-Watson statistic d = ....
• Likely causes:
  • The chosen model equation does not capture the true curvature of the physical system (e.g., fitting a linear or mono-exponential model to biphasic data).
• Resolution:
  • Switch to a higher-order or sigmoidal model family (e.g. logistic_4p or polynomial_3).

5. Outlier detected: \(|z_i| > 2.5\)

• Symptom: Diagnostic check 7 flags WARN: 1 observation(s) exceed 2.5 std residuals.
• Likely causes:
  • A transient measurement spike, pipetting discrepancy, or bubble artifact in the dataset.
• Resolution:
  • Switch Loss function on Start!B7 to Huber or Cauchy to automatically suppress the outlier’s influence.
  • Review the suspect observation on the Data sheet to confirm whether it was an erroneous recording.