Pass, fail, and unknown
The CLI compares a saved contract with a candidate and returns one of three results. It does not invoke tools.
Example: a baseline requires id as a string. The candidate requires id as a number. ContractCanary exits 1 and the CI job fails. Adding an optional field exits 0. A missing schema exits 2, which is also unsuccessful.
- pass, exit 0. The candidate still accepts what the baseline accepted and still returns what the baseline guaranteed. The committed additive fixture adds an optional field and passes.
- fail, exit 1. A previously supported shape breaks. The committed breaking fixture narrows the contract and fails.
- unknown, exit 2. The input is malformed, a schema is missing, or a keyword is outside the supported subset. Unknown is not a pass. The committed malformed, missing, and adversarial fixtures return unknown.
Supported checks are explicit types, properties, required, boolean additionalProperties, items, scalar enums, and numeric or length bounds. Refs, unions, regex, formats, and composition are unsupported.
Adjacent tools cover a different job. Promptfoo evaluates prompts. MCPJam Inspector talks to a live MCP server. Braintrust stores evals. ContractCanary only fails the process when a saved schema breaks.