A retry is not free. See when your recovery policy becomes the outage.

Retry Storm Lab is a deterministic Python simulator for partial dependency failures. It models fresh requests and retries competing for the same capacity, then compares immediate, fixed, exponential, and full-jitter backoff policies.

Retries improve reliability when failures are brief and spare capacity exists. During a sustained partial outage, synchronized clients can instead amplify demand, starve fresh requests, and keep a recovering service overloaded.

The simulator makes that feedback loop visible:

• Generate a steady stream of fresh requests.
• Reduce successful capacity during a configurable failure window.
• Reschedule failed and over-capacity attempts using the selected policy.
• Measure amplification, peak demand, abandoned requests, and recovery time.
• Compare all four policies against the exact same scenario and random seed.

This is an intentionally small queueing model, not a production capacity planner. It omits network latency distributions, client timeouts, autoscaling, circuit breakers, and correlated downstream failures. Its job is to make retry dynamics easy to inspect and discuss.

Python 3.11 or newer is required.

python -m venv .venv
source .venv/bin/activate
pip install -e .
streamlit run app.py

Open http://localhost:8501, choose a preset, and adjust the workload from the sidebar.

retry-storm-lab --preset "Rate-limit spiral" --compare
retry-storm-lab --preset "Slow recovery" --policy full_jitter --json

Example comparison:

immediate    amplification=1.11x peak=2720 rps success=97.2% recovery=1s
fixed        amplification=1.08x peak=1600 rps success=99.9% recovery=3s
exponential  amplification=1.07x peak=1600 rps success=100.0% recovery=3s
full_jitter  amplification=1.10x peak=1651 rps success=99.9% recovery=3s

Exact values depend on the selected preset and seed.

from retry_storm_lab import BackoffPolicy, SimulationConfig, simulate

result = simulate(
    SimulationConfig(
        baseline_rps=800,
        server_capacity_rps=1_000,
        outage_failure_percent=80,
        policy=BackoffPolicy.FULL_JITTER,
    )
)

print(result.amplification_factor)
print(result.peak_rps)
print(result.recovery_seconds)

The engine uses aggregate 100 ms buckets. Full jitter distributes retries individually with a seeded pseudo-random generator, so the same configuration always produces the same result.

The lab works entirely offline. To record aggregate usage bands with Telemetry, install the optional dependency and provide a project API key:

pip install -e '.[telemetry]'
export TELEMETRY_API_KEY=your_project_api_key
streamlit run app.py

Only the selected policy, broad capacity/amplification bands, and whether the run recovered are sent. Raw workload values and timeline data are not logged.

python -m unittest discover -s tests -v
python -m compileall app.py retry_storm_lab

MIT