Temper by Turbolytics

Design with Temper

This tutorial designs a real project: Temper itself. Temper is built with Temper. Every snippet below comes from its repository, and the matrix at the end is its own.

You design in this order. Decide what counts as proof. Say what the system does. Say what it must never do. Then make each test state what it proves.

1 Decide what counts as proof

A level is one kind of proof. Temper has two. Unit tests check that the logic is right. Integration tests run the built binary against real projects. Name each level for what it proves, not for the tool that runs it.

temper.yml

temper: "0.1"
project: temper

levels:
  - name: unit
    description: go test ./...; the logic is right
  - name: integration
    description: the built binary against fixture projects and examples/inventory

2 List what the system does

A feature is one thing the system does, written so a person can check it. requires lists the levels that must prove it. Write features before the code. A list generated from the code describes only what exists. It cannot show what is missing.

temper/features.yml (two of them)

features:
  - id: cli.check
    description: Validates declarations without running any test and names each problem's file, id and rule.
    requires: [unit, integration]
  - id: cli.matrix
    description: Joins declarations with evidence and writes the committed statuses and MATRIX.md.
    requires: [unit, integration]

3 Write what must never happen

An invariant is a promise the system never breaks. This step matters most. Ask what would make the system a failure, even if it technically works. For Temper, the worst failure is a green matrix that hides a gap. So its invariants are about honesty. applies_to names the kind of subject the promise is about.

temper/invariants.yml (two of them)

invariants:
  - id: matrix.skip_never_covers
    claim: A skipped test never makes a cell covered, at any level.
    applies_to: cli
    class: safety
    family: honesty
    requires: [unit, integration]

  - id: matrix.failure_beats_pass
    claim: >
      When several records claim one cell at one level, a failed or errored
      record makes the cell failing, whatever passed beside it. A pass never
      hides a failure.
    applies_to: cli
    class: safety
    family: honesty
    requires: [unit]

4 List every case

Some promises cover a complete list of values. A lattice is that list. Every test outcome is one, so Temper declares all four.

temper/lattices/outcomes.yml

id: outcomes
description: >
  Every outcome an evidence record can carry, from the spec's record format.
  An adapter answers for each one it can produce.
applies_to: adapter
requires: [unit]
members:
  - key: pass
  - key: fail
  - key: skip
  - key: error

A subject is one concrete thing the promises apply to. Each subject must say, for every value, whether it supports it. The Go adapter supports all four. A new adapter that leaves out error fails temper check. Nothing is guessed.

temper/subjects/adapter.go.yml

id: adapter.go
kind: adapter
feature: adapter.go
answers:
  outcomes:
    pass: supported
    fail: supported
    skip: supported
    error: supported

5 Make each test state what it proves

A test calls temper.Covers once for each claim it proves. This test proves the invariant from step 3. It records two skipped tests and checks that the cell stays missing. Temper never infers a claim from a test's name.

internal/matrix/matrix_test.go

func TestBuild_SkipNeverCovers(t *testing.T) {
	temper.Covers(t, "cli.matrix")
	temper.Covers(t, "matrix.skip_never_covers", temper.Subject("cli"))
	r := Build(project(), []evidence.Located{
		rec("f.a", "", "TestA", "unit", "skip"),
		rec("f.a", "", "TestB", "unit", "skip"),
	}, status.NewSet(), both)
	if c := cell(t, r, "feature", "f.a", "", "", "unit"); c.Status != status.Missing || len(c.Evidence) != 2 {
		t.Fatalf("all-skip cell = %+v, want missing with both records listed", c)
	}
	if !inGaps(r, "f.a", "", "unit") {
		t.Fatal("an all-skip cell must be a gap")
	}
}

6 Run it, and read the result

Run each level through temper run, then build the matrix:

temper clean
temper run --level unit -- go test ./...
temper run --level integration -- go test -tags integration -run '^TestIntegration' ./...
temper matrix --all-levels

Each cell has one status. Here is what to do about each.

StatusMeaningWhat to do
✅ coveredA passing test claimed it, and no test that claimed it failed.Nothing.
⬜ missingNo passing evidence. Every skip lands here.Write the test that proves the claim. Do not weaken the declaration.
❌ failingA test that claimed it failed or errored.Fix the code or the test. Never skip it.
☑️ exemptThe subject is excused, and a passing test proves it.Keep the proving test.
⏳ unenforcedA known gap, tracked in an issue.Close the issue, then remove tracked_by.

7 Hand it to an LLM

People own the declarations. An LLM writes the code and the tests. Give it the declarations and the guide, which temper guide prints. CI runs every level and fails the build on a gap.

The LLM cannot pass by cutting corners. A skipped test never covers a cell. A test cannot choose its own level. A change to an invariant shows up in review, because people own that file.

The result: Temper's matrix

This is Temper's own matrix, from MATRIX.md.

temper matrix

Every row in the tables below is part of Temper's specification. Temper declares each one in its own temper/ directory. A cell is covered when a test that ran at that level passed and none failed. A skip proves nothing, and a pass never hides a failure. Exempt cells are excused by a passing proof. Regenerate with temper matrix.

✅ covered · ☑️ exempt · ❌ failing · ⬜ missing · ⏳ unenforced · — not required

Summary

coveredexemptfailingmissingunenforced
460000

Features

FeatureWhat it doesunitintegration
adapter.goLets a Go test state what it proves and records how it ended.✅—
declarations.loadReads every declaration into one model, rejecting unknown fields.✅—
declarations.extension_fieldsIgnores x- fields, so a project keeps its own metadata beside Temper's.✅—
cli.checkValidates declarations without running any test and names each problem's file, id and rule.✅✅
cli.matrixJoins declarations with evidence and writes the committed statuses and MATRIX.md.✅✅
cli.matrix_jsonPrints the full matrix, errors and summary as JSON for CI and agents.—✅
cli.matrix_checkWrites nothing and exits 3 when committed statuses or MATRIX.md are out of date.✅✅
cli.renderRewrites MATRIX.md from the committed statuses alone.✅—
cli.runRuns a command with the level and evidence directory set, so no test chooses its own level.✅✅
cli.cleanDeletes all evidence, the start of a full run.✅—
cli.guidePrints the guide an LLM or a person needs to work in a Temper project, matching the installed version.✅✅
cli.versionPrints the name, Temper by Turbolytics, and the version the binary was built as, whether stamped at release or installed with go install.✅✅
site.findingsHolds findings.csv, the public record of what Temper helped find, to its rules, and builds the Temper site from it.✅—

Invariants

adapter

InvariantSubjectunitintegration
adapter.failure_never_passesadapter.go✅—
adapter.level_from_runneradapter.go✅—
adapter.never_truncatesadapter.go✅—
adapter.silent_without_runneradapter.go✅—
adapter.skip_records_skipadapter.go✅—

honesty

InvariantSubjectunitintegration
declarations.ambiguous_answer_rejectedcli✅—
matrix.exact_reference_onlycli✅—
matrix.exemption_needs_passing_proofcli✅✅
matrix.failure_beats_passcli✅—
matrix.no_evidence_is_an_errorcli✅✅
matrix.requires_any_one_level_proves_itcli✅—
matrix.scenario_needs_declarationcli✅—
matrix.skip_never_coverscli✅✅
matrix.test_only_credits_nothingcli✅—
matrix.unknown_reference_failscli✅✅

output

InvariantSubjectunitintegration
matrix.not_run_never_judgedcli✅—
matrix.output_deterministiccli✅—
matrix.stale_output_detectedcli✅✅

Lattices

outcomes

Every outcome an evidence record can carry, from the spec's record format. An adapter answers for each one it can produce.

Memberadapter.go
pass✅
fail✅
skip✅
error✅

Work with us

Contact us to get onboarded to Temper, or to train your team to build better software with LLMs.

Contact us