Design with Temper
This tutorial designs a real project: Temper itself. Temper is built with Temper. Every snippet below comes from its repository, and the matrix at the end is its own.
You design in this order. Decide what counts as proof. Say what the system does. Say what it must never do. Then make each test state what it proves.
1 Decide what counts as proof
A level is one kind of proof. Temper has two. Unit tests check that the logic is right. Integration tests run the built binary against real projects. Name each level for what it proves, not for the tool that runs it.
temper.yml
temper: "0.1"
project: temper
levels:
- name: unit
description: go test ./...; the logic is right
- name: integration
description: the built binary against fixture projects and examples/inventory
2 List what the system does
A feature is one thing the system does, written so a person can check it. requires lists the levels that must prove it. Write features before the code. A list generated from the code describes only what exists. It cannot show what is missing.
temper/features.yml (two of them)
features:
- id: cli.check
description: Validates declarations without running any test and names each problem's file, id and rule.
requires: [unit, integration]
- id: cli.matrix
description: Joins declarations with evidence and writes the committed statuses and MATRIX.md.
requires: [unit, integration]
3 Write what must never happen
An invariant is a promise the system never breaks. This step matters most. Ask what would make the system a failure, even if it technically works. For Temper, the worst failure is a green matrix that hides a gap. So its invariants are about honesty. applies_to names the kind of subject the promise is about.
temper/invariants.yml (two of them)
invariants:
- id: matrix.skip_never_covers
claim: A skipped test never makes a cell covered, at any level.
applies_to: cli
class: safety
family: honesty
requires: [unit, integration]
- id: matrix.failure_beats_pass
claim: >
When several records claim one cell at one level, a failed or errored
record makes the cell failing, whatever passed beside it. A pass never
hides a failure.
applies_to: cli
class: safety
family: honesty
requires: [unit]
4 List every case
Some promises cover a complete list of values. A lattice is that list. Every test outcome is one, so Temper declares all four.
temper/lattices/outcomes.yml
id: outcomes
description: >
Every outcome an evidence record can carry, from the spec's record format.
An adapter answers for each one it can produce.
applies_to: adapter
requires: [unit]
members:
- key: pass
- key: fail
- key: skip
- key: error
A subject is one concrete thing the promises apply to. Each subject must say, for every value, whether it supports it. The Go adapter supports all four. A new adapter that leaves out error fails temper check. Nothing is guessed.
temper/subjects/adapter.go.yml
id: adapter.go
kind: adapter
feature: adapter.go
answers:
outcomes:
pass: supported
fail: supported
skip: supported
error: supported
5 Make each test state what it proves
A test calls temper.Covers once for each claim it proves. This test proves the invariant from step 3. It records two skipped tests and checks that the cell stays missing. Temper never infers a claim from a test's name.
internal/matrix/matrix_test.go
func TestBuild_SkipNeverCovers(t *testing.T) {
temper.Covers(t, "cli.matrix")
temper.Covers(t, "matrix.skip_never_covers", temper.Subject("cli"))
r := Build(project(), []evidence.Located{
rec("f.a", "", "TestA", "unit", "skip"),
rec("f.a", "", "TestB", "unit", "skip"),
}, status.NewSet(), both)
if c := cell(t, r, "feature", "f.a", "", "", "unit"); c.Status != status.Missing || len(c.Evidence) != 2 {
t.Fatalf("all-skip cell = %+v, want missing with both records listed", c)
}
if !inGaps(r, "f.a", "", "unit") {
t.Fatal("an all-skip cell must be a gap")
}
}
6 Run it, and read the result
Run each level through temper run, then build the matrix:
temper clean
temper run --level unit -- go test ./...
temper run --level integration -- go test -tags integration -run '^TestIntegration' ./...
temper matrix --all-levels
Each cell has one status. Here is what to do about each.
| Status | Meaning | What to do |
|---|---|---|
| ✅ covered | A passing test claimed it, and no test that claimed it failed. | Nothing. |
| ⬜ missing | No passing evidence. Every skip lands here. | Write the test that proves the claim. Do not weaken the declaration. |
| ❌ failing | A test that claimed it failed or errored. | Fix the code or the test. Never skip it. |
| ☑️ exempt | The subject is excused, and a passing test proves it. | Keep the proving test. |
| ⏳ unenforced | A known gap, tracked in an issue. | Close the issue, then remove tracked_by. |
7 Hand it to an LLM
People own the declarations. An LLM writes the code and the tests. Give it the declarations and the guide, which temper guide prints. CI runs every level and fails the build on a gap.
The LLM cannot pass by cutting corners. A skipped test never covers a cell. A test cannot choose its own level. A change to an invariant shows up in review, because people own that file.
The result: Temper's matrix
This is Temper's own matrix, from MATRIX.md.
temper matrix
Every row in the tables below is part of Temper's specification. Temper declares each one in its own temper/ directory. A cell is covered when a test that ran at that level passed and none failed. A skip proves nothing, and a pass never hides a failure. Exempt cells are excused by a passing proof. Regenerate with temper matrix.
✅ covered · ☑️ exempt · ❌ failing · ⬜ missing · ⏳ unenforced · — not required
Summary
| covered | exempt | failing | missing | unenforced |
|---|---|---|---|---|
| 46 | 0 | 0 | 0 | 0 |
Features
| Feature | What it does | unit | integration |
|---|---|---|---|
adapter.go | Lets a Go test state what it proves and records how it ended. | ✅ | — |
declarations.load | Reads every declaration into one model, rejecting unknown fields. | ✅ | — |
declarations.extension_fields | Ignores x- fields, so a project keeps its own metadata beside Temper's. | ✅ | — |
cli.check | Validates declarations without running any test and names each problem's file, id and rule. | ✅ | ✅ |
cli.matrix | Joins declarations with evidence and writes the committed statuses and MATRIX.md. | ✅ | ✅ |
cli.matrix_json | Prints the full matrix, errors and summary as JSON for CI and agents. | — | ✅ |
cli.matrix_check | Writes nothing and exits 3 when committed statuses or MATRIX.md are out of date. | ✅ | ✅ |
cli.render | Rewrites MATRIX.md from the committed statuses alone. | ✅ | — |
cli.run | Runs a command with the level and evidence directory set, so no test chooses its own level. | ✅ | ✅ |
cli.clean | Deletes all evidence, the start of a full run. | ✅ | — |
cli.guide | Prints the guide an LLM or a person needs to work in a Temper project, matching the installed version. | ✅ | ✅ |
cli.version | Prints the name, Temper by Turbolytics, and the version the binary was built as, whether stamped at release or installed with go install. | ✅ | ✅ |
site.findings | Holds findings.csv, the public record of what Temper helped find, to its rules, and builds the Temper site from it. | ✅ | — |
Invariants
adapter
| Invariant | Subject | unit | integration |
|---|---|---|---|
adapter.failure_never_passes | adapter.go | ✅ | — |
adapter.level_from_runner | adapter.go | ✅ | — |
adapter.never_truncates | adapter.go | ✅ | — |
adapter.silent_without_runner | adapter.go | ✅ | — |
adapter.skip_records_skip | adapter.go | ✅ | — |
adapter.failure_never_passes— A test that fails, errors or panics never records pass.adapter.level_from_runner— The level a record carries comes only from TEMPER_LEVEL; no test can set it.adapter.never_truncates— The adapter only appends, to a file unique to its process; it never truncates or removes evidence.adapter.silent_without_runner— Without TEMPER_LEVEL and TEMPER_EVIDENCE the adapter records nothing and changes nothing.adapter.skip_records_skip— A skipped test records skip.
honesty
| Invariant | Subject | unit | integration |
|---|---|---|---|
declarations.ambiguous_answer_rejected | cli | ✅ | — |
matrix.exact_reference_only | cli | ✅ | — |
matrix.exemption_needs_passing_proof | cli | ✅ | ✅ |
matrix.failure_beats_pass | cli | ✅ | — |
matrix.no_evidence_is_an_error | cli | ✅ | ✅ |
matrix.requires_any_one_level_proves_it | cli | ✅ | — |
matrix.scenario_needs_declaration | cli | ✅ | — |
matrix.skip_never_covers | cli | ✅ | ✅ |
matrix.test_only_credits_nothing | cli | ✅ | — |
matrix.unknown_reference_fails | cli | ✅ | ✅ |
declarations.ambiguous_answer_rejected— A lattice answer map that does not say exactly one of supported: true and unsupported: true is an error; it is never read as supported.matrix.exact_reference_only— Evidence credits only the item its reference names exactly; never by test name, id prefix, file path or level.matrix.exemption_needs_passing_proof— A cell is exempt only while a passing exempts record from the declared proven_by test exists; a covers record, another test, a skip or a failure leaves it missing or failing.matrix.failure_beats_pass— When several records claim one cell at one level, a failed or errored record makes the cell failing, whatever passed beside it. A pass never hides a failure.matrix.no_evidence_is_an_error— With no evidence at any level and without --all-levels, temper matrix exits 2 and writes nothing. It never reports the committed statuses as the result of a run.matrix.requires_any_one_level_proves_it— An invariant with requires_any is proven by a pass, or a proven exemption, at any one listed level; it is a gap once, per subject, only when no level is proven; and a failure at any level is still a gap.matrix.scenario_needs_declaration— A member of a scenario-covered lattice is credited only by evidence from a declared scenario that lists it, and a member no scenario lists is a gap before any test runs.matrix.skip_never_covers— A skipped test never makes a cell covered, at any level.matrix.test_only_credits_nothing— Evidence for a test-only subject credits no cell.matrix.unknown_reference_fails— Evidence or a declaration that names an id Temper does not know is an error that exits 2; it is never ignored and never creates a row.
output
| Invariant | Subject | unit | integration |
|---|---|---|---|
matrix.not_run_never_judged | cli | ✅ | — |
matrix.output_deterministic | cli | ✅ | — |
matrix.stale_output_detected | cli | ✅ | ✅ |
matrix.not_run_never_judged— When another level ran, a level with no evidence keeps its committed statuses and never affects the exit code, unless --all-levels judges it.matrix.output_deterministic— The same declarations and evidence always produce the same bytes.matrix.stale_output_detected— matrix --check exits 3 whenever the committed statuses or MATRIX.md differ from what the evidence produces.
Lattices
outcomes
Every outcome an evidence record can carry, from the spec's record format. An adapter answers for each one it can produce.
| Member | adapter.go |
|---|---|
pass | ✅ |
fail | ✅ |
skip | ✅ |
error | ✅ |
Work with us
Contact us to get onboarded to Temper, or to train your team to build better software with LLMs.