Why My MVP Readiness Review Failed — And Why That Was the Correct Result

A green test suite is one of the easiest things in software engineering to misunderstand.

Lootwright reached a point where a large amount of the system looked healthy. The parser had defensive tests. Architecture boundaries were being checked. Browser flows existed. Policy and evaluation machinery existed. A substantial automated test surface was green.

And the MVP readiness review still ended with:

FAIL.

Nothing had crashed.

The build wasn't red.

The application wasn't missing because of a syntax error.

The failure was more important than that: I did not yet have enough evidence to trust one of the core claims the product wanted to make.

Tests can prove the wrong thing perfectly

Lootwright analyzes Path of Exile builds.

A system like that can be internally consistent and still be wrong.

If I implement an incorrect rule and then write a hundred tests asserting that incorrect rule, the tests can all pass. They prove that the code behaves according to my implementation.

They do not prove that the implementation represents the authoritative domain rules.

That gave me three different states I needed to stop conflating:

Implemented — the code exists.

Verified — the code behaves reproducibly according to its specification.

Authoritative — the specification or ruleset itself is trustworthy enough to make the claim in production.

A feature can reach the first two states and still fail the third.

That was the problem.

The missing part wasn't another controller test

The release blocker was not a missing HTTP test or another validation rule.

The real missing artifact was a production-grade deterministic analysis source I was willing to treat as authoritative.

That matters because Lootwright deliberately puts deterministic reasoning underneath generative AI.

The system should first establish things it can actually compute:

input
  ↓
bounded parsing
  ↓
normalized representation
  ↓
deterministic rules
  ↓
structured findings
  ↓
optional generative explanation

The model is downstream.

If I cannot establish the deterministic finding, I don't want an LLM improvising one because it sounds plausible.

Sometimes "Unavailable" is the correct implementation

This is an uncomfortable design choice because unavailable features look unfinished.

They are unfinished.

But unfinished and explicit is much better than complete-looking and dishonest.

The production binding in Lootwright is straightforward:

$this->app->bind(
    DeterministicAnalysisEngine::class,
    UnavailableDeterministicAnalysisEngine::class
);

When the authoritative analyzer is unavailable, the safe state is not "best effort" or "probably true" or "ask the LLM."

It is:

throw new TerminalWorkflowFailure(
    'deterministic_analyzer_unavailable',
    'No approved deterministic analyzer is active.'
);

That gives the application a very useful property:

It knows when it does not know.

For analysis software, that is much more important than producing an answer every time.

So what did all the green tests prove?

Quite a lot.

The August 2026 readiness review recorded 555 PHPUnit tests with 7,809 assertions, 330 architecture checks with 6,471 assertions, 39 parser-security tests with 223 assertions, 71 Policy Gate tests with 367 assertions, 31 fast eval cases, 35 extended eval cases, 15 Vitest tests, and 7 Chromium Playwright tests.

They gave confidence in surrounding mechanisms:

• malformed input could be rejected;
• parsing was bounded;
• architectural boundaries were enforceable;
• policy behavior was reproducible;
• application workflows could be regression-tested;
• browser-level flows could be exercised;
• deterministic components behaved consistently where authoritative inputs existed.

What they could not prove was that a missing authoritative source had somehow become authoritative because everything around it worked.

That distinction is exactly what a release review is supposed to expose.

The test that matters most

One test stands out from the readiness review:

public function test_production_analysis_binding_fails_closed_without_an_approved_ruleset_or_analyzer(): void
{
    $engine = $this->app->make(DeterministicAnalysisEngine::class);
    self::assertSame(
        UnavailableDeterministicAnalysisEngine::class,
        $engine::class,
    );

    $this->expectException(TerminalWorkflowFailure::class);
    $this->expectExceptionMessage('No approved immutable ruleset is active');
    $engine->resolve(...);
}

This test explicitly verifies that the production container will not quietly substitute a fake for a missing authoritative implementation.

It proves the application refuses to lie.

Fixture-backed tests have their place

The test suite includes a comprehensive end-to-end acceptance test that exercises the real parser, policy gates, persistence, queues, and deletion:

public function test_release_harness_runs_real_poe1_import_for_anonymous_user_with_ai_off_and_complete_deletion(): void

But notice what it does:

$this->app->instance(
    DeterministicAnalysisEngine::class,
    new FakeDeterministicAnalysisEngine(true)
);

The test explicitly replaces the production binding with a fake that returns fixture data.

This is valuable conformance testing.

It proves the workflow orchestration is correct.

It proves persistence works.

It proves deletion cascades properly.

It proves the encrypted recipe storage contains the expected trace codes and numeric ranges.

What it cannot prove is that those fixture findings represent real Path of Exile rules.

The readiness document is explicit about this:

Fixture-backed success tests are valuable conformance evidence but are not game-accuracy or production evidence.

Readiness checks need veto power

A release checklist that cannot stop a release is paperwork.

If everybody has already decided to ship, the checklist is just a ceremony performed before deployment.

I wanted a readiness review capable of returning exactly two meaningful states:

PASS

or

FAIL

Not "mostly ready."

Not "8.5/10."

Not "good enough except for the central correctness problem."

A system either has enough evidence to support a critical product claim or it does not.

That forces better questions:

• Where did this conclusion come from?
• Can I reproduce it?
• Which exact rule produced it?
• What happens when the rule source is absent?
• Does the system distinguish false from unknown?
• Can the model alter deterministic truth?
• Can a ruleset version invalidate historical conclusions?

Those questions are far more useful than asking whether CI happens to be green.

Production readiness is partly an evidence problem

I used to think about readiness mostly operationally.

Does it build?

Does it deploy?

Do migrations work?

Can I roll it back?

Are secrets configured?

Are tests green?

Those remain necessary.

But analytical systems add another question:

What evidence allows the software to make the claims it makes?

A system can be operationally healthy and still return the wrong deterministic conclusion.

The architectural boundaries held

One of the more satisfying outcomes of the readiness review is that the architectural decisions documented in ADR 0002 (Deterministic Core) and ADR 0004 (Provider-neutral Optional AI) held up under scrutiny.

The deterministic analysis port is clean:

interface DeterministicAnalysisEngine
{
    public function resolve(
        AnalysisRecord $analysis,
        ArtifactRecord $artifact
    ): ResolvedAnalysisContext;

    public function run(
        AnalysisRecord $analysis,
        ArtifactRecord $artifact,
        ResolvedAnalysisContext $context,
    ): DeterministicAnalysisSnapshot;
}

The AI gateway is explicitly optional:

interface AiGateway
{
    public function extractIntent(
        NaturalLanguageIntentRequest $request
    ): AiGatewayOutcome;

    public function explain(
        GatewayExplanationRequest $request
    ): AiGatewayOutcome;
}

The Policy Gate prevents unauthorized execution:

interface AnalysisPolicyGate
{
    public function authorize(
        ResolvedAnalysisContext $context
    ): void;
}

Those boundaries mean the system can fail in the right place.

The missing ruleset doesn't cause the parser to fail.

The missing ruleset doesn't cause the AI gateway to improvise.

The missing ruleset causes the exact component responsible for deterministic analysis to refuse execution.

The architecture is designed so generative AI explains deterministic findings rather than owning them. The current AI port is intentionally downstream of deterministic analysis and does not expose an API for producing deterministic findings, recommendations, or game facts.

What I would repeat

I would make the same release decision again.

The part I would change is timing.

I would define these three categories before implementation begins:

Implemented
Verified
Authoritative

Then each important capability has to state which level it has reached.

That prevents "code exists" from quietly turning into "feature complete."

The readiness document now makes this explicit:

Critical blockers:

1. POE1-RULES-001 is a disabled candidate with no approved source, version, permission scope, checksum, transformation/redistribution analysis, or activation implementation.
2. The production deterministic analyzer and real PoE1 finding/ranking formulas do not exist.
3. Production findings/upgrades/recipes pages are fixture-backed rather than bound to authoritative application results.

Lootwright failed the MVP readiness review.

That wasn't a failure of the review.

It was proof that the review worked.