Case study: DataFawn

Overview#

DataFawn is an SEO product for freelancers and small agencies, built by the team behind MLola. Its promise is “Know what to fix. Prove it worked.” On 9 November 2026 the second half of that promise did not work in production, although every test passed. This page is the record of three change requests that fixed it, each written as a contract and proven with mlola browser prove against the live site.

What production showed#

A client had marked three changes as applied in early September. Two months later:

• The Overview still read “Measuring until 30 Sep”, five weeks after that date.
• Results showed “Baseline not captured” for clicks, impressions, CTR and position.
• Every evaluation window of all three experiments had failed, and none was retried.

The code that computes a baseline was correct and tested. Nothing called it. The tests covered the schedule and the arithmetic as pure functions, and not the chain from “applied” to a measured result, so the gap shipped.

Three contracts#

The product owner approved each change request before any code was written. Each contract mixes layers: tests for the rules, the browser for what a signed-in user sees on the live site, and one manual criterion for what only a person can judge.

The browser criteria ran against production with a saved signed-in session (--storage-state). They only open pages and click navigation; they submit nothing.

{
  "id": "AC-6",
  "text": "On the live site, every change past its last window has its final result: none is still being calculated",
  "proof": {
    "layer": "browser",
    "url": "/clients",
    "steps": [{ "click": "<client name>" }, { "click": "Results" }],
    "expect": {
      "kind": "all",
      "of": [
        { "kind": "text_visible", "text": "measured changes" },
        { "kind": "not", "of": { "kind": "text_visible", "text": "Calculating the result due" } }
      ]
    },
    "timeoutSeconds": 20
  }
}

What the live criteria caught#

After each deploy, the proof was made again against production. Three times it stayed not proven while the test suite was green, and each time the cause was a real defect.

• A result that read as a win and was not. A page with 84 impressions before its change and none after showed position “0” with a green “↓100% vs baseline”. The screenshot kept with the proof showed it. CR-DF-3 makes that read “No data after the change”.
• A worker that could not finish. The final windows were retried and failed with NoReferencedTableError: the worker process never imported the module that defines a table the write refers to. Every test imports the whole application, so no test could see it. The fix adds two tests that start a fresh interpreter, as the worker does.
• A model call that never succeeded. Every written explanation failed in production. The request asked for bare JSON, and the model answered with an object where a string was expected. Constraining the answer to a schema fixed it, and the same fault was then found and fixed in a second call.

A first guess about the failed final windows was wrong: that they had simply not been retried yet. The contract's first criterion required the API to say how often a window was tried and why it failed. Once that was deployed, the API named the real error.

Before and after#

CR-DF-3, as mlola browser receipt show prints the two proofs. The first was made before the change, the second after the last fix was deployed and the worker had retried.

contract   CR-DF-3 · Every result reads true, and a result that failed says why
verdict    failed; 0/7 acceptance criteria proven
  ? AC-1 test "test_an_evaluation_names_its_attempts_and_its_error" is not in the report
  …
  ✖ AC-5 text "No data after the change" not found on the page
  ✖ AC-6 text "Calculating the result due" is present, and must not be
  👤 AC-7 a person checks it: the product owner compares each before and after with Search Console
  ✔ INV-1 5 tests passed
  ✔ INV-2 text "proves whether it worked" is present

contract   CR-DF-3 · Every result reads true, and a result that failed says why (approved)
verdict    verified; 6/7 acceptance criteria proven
auto-merge no
  ✔ AC-1 1 test passed
  ✔ AC-2 1 test passed
  ✔ AC-3 1 test passed
  ✔ AC-4 1 test passed
  ✔ AC-5 text "measured changes" is present
  ✔ AC-6 text "measured changes" is present
  👤 AC-7 a person checks it: the product owner compares each before and after with Search Console
  ✔ INV-1 5 tests passed
  ✔ INV-2 text "proves whether it worked" is present

Auto-merge stays “no” on all three contracts, by design: each has a manual criterion, so a person still compares the results with Search Console and runs a five-second test of the Overview.

What it changed in MLola#

Two faults in prove appeared only on a real application, and version 0.15.1 fixes both.

• A step now waits for its control. The live pages load their data after the first paint, so a step that clicked a label failed when the label arrived a moment later.
• A negated check that fails now says why: “text … is present, and must not be”.

What this does not show#

• The proofs here are unsigned. They were made on a developer machine, not in CI with a signing key.
• The live criteria check production, so they turn green only after a deploy. A preview environment with test data would move that before the merge.
• Two of the three live defects were found by reading the proof's screenshots and the API, and then written into the next contract. A contract proves what it states, and the first one did not state enough.
• The manual criteria are still open: the proofs say so, and do not count them as proven.