Playhead

Verification

Anyone can render a video. The reason Playhead can be trusted in a pipeline is that it reviews its own output afterwards and refuses to call a bad one publishable.

What is being checked, exactly#

Not the plan — the delivered pixels. Verification decodes frames out of the finished MP4, projects the geometry the compositor recorded in compose-manifest.json, and compares the two. A compositor bug that draws the wrong frame or the wrong crop fails here even though every internal calculation was self-consistent.

The manifest is the contract between the two halves: the renderer records what it actually drew — positions, times, overflow facts, camera fallbacks, clock gaps — and the verifier holds it to that.

The ten checks#

CheckAsserts
encodeThe file is a real, decodable H.264 video at the planned resolution, frame rate and duration.
target-in-frameEvery action target was fully inside the frame and rendered large enough to read — at least 28px tall for a close-up, unless a wide shot waived it.
cursor-on-targetThe rendered cursor was on the element at the moment of each click, confirmed against the cursor sprite on film.
action-effectEach action visibly changed the UI, compared frame to frame before and after the beat.
output-pixelsFor sampled steps, the region of the MP4 where the subject should be correlates with the same region of the source bundle frame.
frozen-blankNo frozen stretches spanning an action beat, and no blank content regions.
captionsEvery caption fit its card at full size — nothing shrunk, nothing truncated.
pacingNo step is too fast to follow (at least 1.2 seconds each).
masksMasked regions are opaque in the delivered output and absent from the event log.
audioEach spoken line is audible inside its own step window, at the target loudness.

Reading a verdict#

This is the real verify/verdict.json for the video on the examples page, abridged:

{
  "schema": "playhead/verdict@3",
  "verdict": "publishable",
  "video":     { "sha256": "0b4ed39080fc7658770aac9a913b7c9b…" },
  "manifestHash": "4a700ec618ad55872ee71d446a786beb…",
  "coverage": {
    "actionsVerified": 13,
    "actionsTotal": 13,
    "outputSamplesVerified": 3,
    "checksSkipped": 1,
    "checksWarned": 1
  },
  "provenance": {
    "playheadVersion": "0.4.0",
    "specHash": "445f690cea1f2f91aa445064…",
    "appUrl": "https://www.saucedemo.com/",
    "checkSuiteVersion": "playhead/checks@4",
    "host": "win32-x64"
  },
  "checks": [ … ]
}

Coverage is as important as the verdict. A run where every check passed but only two of seven actions could be pixel-verified is much weaker evidence than one with full coverage — and the verdict says so out loud instead of hiding it behind a green tick.

Pass, fail, skip, warn#

StatusMeaning
passAsserted and satisfied.
failAsserted and violated. The verdict becomes not-publishable and render exits 2.
skipNothing to assert — masks on a spec with no masking rules, audio on a silent video. Skips do not block publication; they are counted in coverage.checksSkipped.
warnThe check could not measure what it wanted to. It reports honestly rather than passing by default, and the count lands in coverage.
A check that cannot measure will not pass. If capture left no pre-action frame for a step, action-effect cannot compare before and after, so it warns and records the gap. That is the difference between a suite that proves something and a suite that decorates a build log.

A real failure, and what it taught#

The first cut of this site’s hero video clicked the 20-pixel “toggle subsection” chevrons in Wikipedia’s contents rail. Every geometry check passed. output-pixels did not:

    FAIL  output-pixels: 2/6 sampled regions diverge from the plan
          contents/0  @19.4s — output region does not match the planned
                               source region (correlation 0.49)
          mechanical/0 @57.2s — correlation 0.49
  ✗ verdict: NOT PUBLISHABLE

The threshold is 0.5, so this was a near miss rather than a gross error — and it was correct. A subject that small, mutating the instant it is clicked, cannot be confirmed against its own source frame. The fix was an authoring fix: give the camera substantial subjects. Re-authored with real content links instead of chevrons, the same flow verified clean.

That is the intended experience. The check does not tell you your renderer is broken; it tells you the video is not good enough to publish, and points at the two seconds to look at.

Running verification on its own#

playhead verify out/capture out/out.mp4 --json
playhead verify out/capture out/out.mp4 --junit report.xml

JUnit output emits one testcase per check, so a CI failure names the craft rule it broke rather than saying “video job failed”. CI setup →

The contact sheet#

Every run writes verify/contact-sheet.png: key moments as an annotated grid, so a reviewer can sanity-check a two-minute video in ten seconds. Its hash is recorded in the verdict, which means the sheet cannot be swapped after the fact.

Optional: a second opinion#

playhead render specs/tour.yaml --vision

Sends the contact sheet to a local Ollama model for a qualitative review and records the result under vision in the verdict. It is advisory and never overrides a deterministic check. Frames do not leave your machine, and the flag is off by default.

Comparing runs#

playhead compare baseline/ current/

Verdict drift, newly failing checks, coverage change and per-step timing deltas — and exit 2 when the walkthrough regressed. This is how you notice that a step got three seconds slower before anyone complains about the video feeling sluggish.

PreviousCommands