A research job exits successfully but writes no observations. Is the system healthy? Only if successful execution was the entire contract. If downstream research needs fresh observations, a green process indicator answers the wrong question.
This research note describes a proposed testing discipline. It does not certify the present state of Monolith infrastructure or report an operational incident.
Separate three outcomes
A research verdict should distinguish PASS, FAIL and ERROR. PASS means admissible evidence met the declared criterion. FAIL means admissible evidence did not. ERROR means the evidence or computation was insufficient to reach either verdict. Treating an empty dataset as zero loss confuses ERROR with PASS; treating it as proof of no edge confuses ERROR with FAIL.
Write the input contract before the verdict: required fields, valid units, expected observations, permitted missingness, freshness and version identity. These are design recommendations, not universal constants. A scheduled market holiday may legitimately produce no observations; the contract must distinguish that case from a broken feed rather than demanding rows unconditionally.
The same separation belongs in monitoring. Google’s SRE guidance distinguishes white-box monitoring of internal metrics from black-box checks of externally visible behaviour.[9] A process heartbeat is one internal signal. A downstream reader successfully consuming the expected fresh artifact checks something closer to the research service being delivered.
Conceptual example
Suppose a validator expects a return series and a cost specification. A symbolic fixture supplies both and is designed to satisfy the criterion. A second fixture removes the cost field. A third supplies valid inputs designed to fail the criterion.
The expected outputs are PASS, ERROR and FAIL respectively. This is not a miniature market study: the fixtures establish control flow. If the missing field is replaced with zero, a schema error has silently become a favourable economic assumption. If every fixture returns ERROR, the system blocks unsafe promotion but is still broken: it cannot recognise admissible evidence.
Cheap check
Run the production entry point against the fixtures, then against an empty series, a stale artifact, a wrong-unit field and a duplicated observation. Include a legitimate no-observation case under a separately declared calendar rule. Assert the exact outcome and reason, not merely that execution finishes.
Pass: valid positive and negative controls retain their intended verdicts; each malformed input produces a specific ERROR; expected calendar emptiness is represented without masquerading as a performance observation. Verify that the output artifact and downstream reader preserve those distinctions.
Fail: any malformed input obtains a permissive default, a genuine negative control is labelled an infrastructure fault, or a wrapper catches an exception and exports a stale PASS. Test exception behaviour explicitly; Python’s unittest provides assertRaises for that purpose.[8]
Then deliberately disable one safeguard in a disposable test copy. The corresponding test must fail. A test that still passes has not demonstrated that it protects the boundary it names.
Limitations
Fault injection only covers faults actually planted. Tests can share the implementation’s mistaken assumption, including assumptions that make results look worse. Keep simple independent fixtures for both admission and rejection, and revisit every consumer when a new input format makes a formerly impossible state possible.
A hard error is useful only if it reaches a responsible reader with enough context to diagnose it. Preserve the run identity, source version and reason without logging credentials. Never turn error volume into permission to weaken the evidence rule silently.
Return to execution costs for a concrete missing-input boundary, or restart with the estimator contract. See research infrastructure for the wider system.
Sources
[8] https://docs.python.org/3/library/unittest.html — Python — unittest reference [9] https://sre.google/sre-book/monitoring-distributed-systems — Google SRE — Monitoring Distributed Systems