Most example applications are designed to demonstrate that something works.

Create a record. Dispatch a job. Cache a result. Call an API. Load a relationship.

Everything goes right, the demo ends, and the interesting part never happens.

Production systems are usually less polite.

The database waits on a lock. A queue retries after some work has already happened. Multiple workers miss the same cache entry. An upstream API takes several seconds longer than expected. An innocent-looking relationship turns one query into dozens.

I wanted a project where those were not accidental failures.

I wanted them to be the product.

That became RescueBench, a public Laravel failure-reproduction laboratory built around a simple loop:

REPRODUCE
    ↓
OBSERVE
    ↓
DIAGNOSE
    ↓
FIX
    ↓
VERIFY

The important word is not fix.

It is reproduce.

A bug you cannot reproduce is mostly a story

A production incident often arrives as a sentence:

"Sometimes this request takes forever."

Or:

"We found two payments."

Or:

"The database suddenly gets hammered."

Those statements are useful clues, but they are not yet engineering evidence.

Before I want a clever fix, I want a controlled way to make the failure happen again.

That changes the questions.

Instead of:

What code should I change?

I can ask:

  • What exact state triggers the failure?
  • What can I measure while it happens?
  • Can I make it happen repeatedly?
  • Does the proposed fix remove the failure mechanism?
  • Can a regression test prove that?

RescueBench is built around those questions.

The scenario contract is deliberately boring

Each failure mode implements the same small interface:

interface Scenario
{
    public function getId(): string;

    public function getName(): string;

    public function getDescription(): string;

    public function runBroken(): void;

    public function runFixed(): void;

    public function reset(): void;
}

There is no elaborate plugin framework.

There does not need to be one.

A scenario has to be identifiable, runnable in a deliberately broken state, runnable in its corrected state, and resettable.

A registry makes those scenarios discoverable by the CLI.

That is enough structure to add failure experiments without turning the laboratory itself into the thing I am debugging.

Five different ways to ruin an otherwise normal backend

The current repository contains five scenarios.

RB-001 — Database lock contention

Two pieces of work compete for database access.

The interesting part is not simply producing a lock.

It is seeing how transaction boundaries, database behavior and concurrency assumptions affect the result.

The development setup can use SQLite, but that has an important limitation: SQLite's locking behavior is not PostgreSQL row-level locking.

So PostgreSQL is the more useful environment when the purpose is to study this failure properly.

That distinction matters.

A reproduction laboratory should not pretend two different concurrency models are equivalent just because both can eventually produce the word "lock."

RB-002 — Queue retries and duplicate side effects

Queue systems normally provide at-least-once processing semantics.

That means retry behavior has to be treated as part of the application's correctness model.

The broken version performs work and then deliberately fails:

Payout::create([
    'recipient' => $this->recipient,
    'amount' => $this->amount,
    'status' => 'processing',
]);

if ($this->simulateFailure && $this->attempts() === 1) {
    throw new RuntimeException('Payment gateway timeout (simulated)');
}

The job gets another attempt.

Without protection, repeating the operation can repeat the side effect.

The corrected laboratory path introduces an idempotency key and database constraints around repeated processing.

The broader lesson is not:

Add an idempotency column and payment systems are solved.

They are not.

The current scenario models the retry problem locally. A real external payment operation introduces another consistency boundary because a database transaction cannot roll back something that already happened at a remote provider.

That is exactly the kind of distinction I want a failure lab to make visible.

A demo should teach the boundary of the solution, not just show the green path.

RB-003 — Cache stampede

Caching is usually introduced with one diagram:

request
   ↓
cache hit? ── yes ──> return value
   │
   no
   ↓
compute
   ↓
cache

Real systems add another question:

What happens when several workers reach "no" at the same time?

If every worker performs the expensive work before any of them populates the cache, the cache can briefly amplify the load it was supposed to reduce.

RescueBench makes that concurrency visible rather than discussing "thundering herds" as an abstract pattern.

The useful result is not merely that locking can fix it.

It is seeing the number of computations and cache accesses change between the broken and corrected paths.

RB-004 — Slow upstream API

External APIs eventually become slow.

Not necessarily unavailable.

Slow.

That distinction is unpleasant because a dependency that takes several seconds to fail can consume more capacity than one that fails immediately.

The scenario focuses on observable behavior around:

  • request duration;
  • timeout handling;
  • upstream call count;
  • fallback behavior.

This is the kind of failure I prefer to reproduce deliberately because otherwise timeout values tend to become arbitrary configuration copied between projects.

RB-005 — ORM N+1 queries

N+1 is almost too familiar to be interesting.

That is probably why it keeps surviving code review.

The failure is simple:

load a collection, lazily touch related data for every item, and turn one logical operation into a growing number of SQL queries.

The laboratory records actual queries rather than printing a fictional benchmark.

That difference matters.

I do not need a fake claim that one implementation is "37 times faster."

I need to show that one path produces repeated relationship queries and the other does not.

The query log is enough evidence.

Broken and fixed belong next to each other

One design rule became more important as I built the scenarios:

The broken implementation is part of the documentation.

A typical tutorial shows the final pattern:

Cache::lock(...);

or:

->with('posts');

or:

idempotency_key

That teaches syntax.

It does not necessarily teach the failure.

In RescueBench, I want both states available:

php artisan rescuebench:run rb-003 --broken
php artisan rescuebench:run rb-003 --fixed

The broken version provides a baseline.

The fixed version provides a comparison.

Without the first result, the second is just a recommendation.

Observable beats convincing

I deliberately do not want the scenario implementation to print:

Performance improved dramatically.

That sentence contains almost no information.

A useful scenario should expose evidence appropriate to the failure:

database contention  → waits / transaction behavior
queue retry          → resulting records / attempts
cache stampede       → computation count
slow upstream        → duration / timeout behavior
N+1                   → query count and query pattern

The metrics do not need to be sophisticated.

They need to be real enough to answer the question the scenario is asking.

That is also why RescueBench is not a benchmarking project.

Small local experiments are excellent for explaining a mechanism.

They are terrible evidence for claims such as:

This implementation scales to 50,000 requests per second.

That requires a different test.

Isolation is more useful than realism everywhere

Each scenario owns its own namespace, tables and reset behavior.

The database tables are separated with scenario-specific prefixes.

One experiment does not need to know about another.

That keeps a cache experiment from turning into a database-lock debugging session because some shared fixture was left behind.

It also makes destructive experimentation cheap.

I can reset a scenario and run it again.

That is one of the advantages a laboratory has over production: I can intentionally build the bad state instead of waiting for it.

CLI first was the right choice

There is no dashboard required to understand RescueBench.

That is deliberate.

Failure reproduction benefits from a boring interface.

A command can be:

  • copied;
  • scripted;
  • run repeatedly;
  • used in CI;
  • compared before and after a change.

A web UI might be useful later, but it would not make the experiments more correct.

For this project, the terminal is the better first interface.

Regression tests are where the fix becomes useful

A successful manual reproduction tells me I understand something.

A regression test makes that understanding harder to lose.

Each scenario is structured so tests can verify things such as:

  • the scenario is registered;
  • reset behavior actually clears its state;
  • the broken implementation exhibits the intended failure;
  • the fixed implementation removes the specific condition being tested;
  • the required schema and relationships exist.

That is different from saying the lab perfectly models every production environment.

It does not.

The purpose of the tests is narrower:

given this controlled reproduction, does the failure continue to behave the way the experiment expects?

That is enough to stop the demonstration itself from silently rotting.

"Production-relevant" does not mean "production"

This distinction is important.

RescueBench models production failure patterns.

It is not production software.

The repository explicitly supports simpler local fallbacks, including SQLite and database-backed cache/queue behavior, while PostgreSQL and Redis provide more representative behavior for some scenarios.

Those environments are not interchangeable.

A cache-lock experiment backed by a local mechanism is useful for understanding the shape of a stampede.

It is not proof of Redis behavior across a real cluster.

A SQLite contention experiment is useful.

It is not PostgreSQL row-lock behavior.

A locally simulated payout is useful for demonstrating retries.

It is not an end-to-end guarantee against duplicate charges at an external processor.

This is a feature of the project, not something I want to hide.

A laboratory becomes less useful the moment it starts pretending its model is the thing being modeled.

What I would keep

If I started RescueBench again, I would keep the core very small.

I would keep:

  • one minimal scenario contract;
  • explicit broken and fixed modes;
  • isolated scenario state;
  • CLI-first execution;
  • failure-specific evidence;
  • regression tests around the observed behavior.

I would also keep the refusal to turn it into a generic Laravel showcase.

There is already plenty of sample software demonstrating controllers, forms and CRUD.

I wanted something that begins where those examples normally stop.

What I would improve next

The next useful work is not adding animation to the output.

It is improving failure fidelity.

The most interesting future scenarios are the ones where local transactions stop being enough:

  • external side effects;
  • worker crashes between state transitions;
  • lost acknowledgements;
  • competing consumers;
  • distributed locks;
  • partial network failures;
  • stale reads;
  • retries across service boundaries.

Those failures force the system to answer harder questions about ownership, idempotency and recovery.

They also make excellent experiments because the naive implementation often looks completely reasonable until the failure is reproduced.

The point of RescueBench

The project is not about collecting "best practices."

That phrase often removes the context that made the practice useful in the first place.

I would rather keep the failure attached.

Show me the bad state.

Show me the evidence.

Show me why it happened.

Then show me the fix.

Then make the test fail if somebody reintroduces it.

That is RescueBench.

I built a Laravel project whose job is to fail on purpose, because failure is much easier to understand when I can ask it to happen again.