Testing and Verification

Testing, verification protocol, error inspection, and convergence checks.

This section defines how AI assistants and developers verify changes to Boardflare notebooks and ensure calculations and published outputs remain robust.

1. What status: "ok" Means (and Doesn’t Mean)

When modifying a notebook via the code sheet (_BOARDFLARE), incrementing the request counter in H1 causes the add-in to reload and evaluate the notebook.

  • What ok means: All Marimo cells were parsed, compiled, and executed without raising uncaught Python exceptions.
  • What ok does NOT mean:
    • It does not mean the analytical logic or business calculations are correct.
    • It does not mean worksheet outputs are populated (e.g. if bf.publish() was not called or omitted an output name).
    • It does not mean downstream worksheet formulas (BF.OUTPUT, BF.FUNCTION) evaluated without errors.

Always inspect both the H2 response status and the actual worksheet cells after applying an edit.

2. Inspecting Worksheet Formula Errors

When checking cells in Excel containing =BF.OUTPUT(...) or =BF.FUNCTION(...), look for the following indicator values:

Cell Display Cause Resolution
#BUSY! The shared runtime is booting or recalculating. Normal during cold start. Interactive task pane startup times out after 90 seconds; when worksheet BF.OUTPUT or BF.FUNCTION formulas wait on the runtime, the cold-start deadline is five minutes.
#NAME? The formula name is unrecognized. Ensure the Boardflare add-in is running and formulas are spelled =BF.OUTPUT or =BF.FUNCTION.
#VALUE! An invalid argument was passed to a function, a timezone-aware datetime was returned, or an empty sequence [] was published. An unknown notebook output or function name also gives #VALUE! (see troubleshooting item 20). Check argument types and ensure returned datetimes/times are timezone-naive.
#NUM! The calculation produced NaN, inf, or pd.NaT. Replace missing or invalid numeric values with valid numbers or "".
#N/A The calculation returned pd.NA or a complex number, ragged 2D arrays were padded, a published function exceeded the 60-second execution deadline, or the notebook is stopped. Verify array rectangularity and DataFrame values. For a timeout, optimize function logic and move expensive computations into reactive notebook cells. For a stopped notebook, restart it.
#SPILL! The dynamic array output cannot expand because adjacent cells contain data. Clear all non-empty cells in the spill range below and to the right of the anchor cell.
#CALC! Excel encountered a calculation engine error with dynamic arrays. Ensure the published data format is supported (scalars, 1D/2D arrays, Series, DataFrames).
0 (or "None") The notebook published a Python None value. In Excel, None displays as "None". Use empty strings "" for intentional blank cells.

3. Verifying Formula Consumers

To confirm that worksheet formulas are successfully connected to notebook outputs: - Inspect publication.consumers["outputs"] and publication.consumers["functions"]. - The consumers map lists active worksheet cells and ranges consuming each published name. - If a name is missing from consumers, no worksheet formula has requested it yet, or the formula contains a typo.

4. Scenario Testing (Deterministic Verification)

The gold standard for validating a Boardflare workbook is scenario testing: 1. Assert Baseline: Confirm that when the workbook opens with its default inputs, output cells evaluate to expected baseline values. 2. Apply Input Edits: Modify input cells in the worksheet (e.g. changing an interest rate or driver assumption). 3. Verify Convergence: Confirm that the dependent output cells recalculate and match the expected new values. 4. Assert Changed State: At least one output value must differ from the baseline state to prove that the reactive notebook executed.

Example testing pattern: - Baseline: Set Assumptions!B2 to 0.08; assert Dashboard!D4 evaluates to 125,000. - Stress Case: Change Assumptions!B2 to 0.05; assert Dashboard!D4 converges to 95,000.

5. AI Agent Verification Protocol

When an AI assistant updates a notebook on behalf of a user:

  1. Step 1: Diagnostic Read
    • Check H1 (request counter) and H2 (response JSON).
    • If status === "errors", read the traceback to diagnose the existing failure.
  2. Step 2: Apply Targeted Edit
    • Modify only the required cell rows in column A of _BOARDFLARE (or the header text in cell H9, to change notebook width or app settings).
    • Increment H1 by 1.
  3. Step 3: Await Settlement
    • Poll H2 until request === H1 and status !== "running".
  4. Step 4: Check Response & Cell Values
    • If status === "errors", evaluate the traceback.
    • If status === "ok", inspect the destination cells in the visible worksheet to verify numbers, dates, and tables populated cleanly.
  5. Step 5: Two Fix Attempts Rule
    • If an error occurs, attempt at most two automated fixes.
    • If the issue persists after two attempts, stop and ask the user for clarification, presenting the error message and current status.