Retry without circuit breaker or load shedding

errorretry-without-adaptive-shedding

Retries into a service that can neither shed load nor have its circuit opened produce a metastable failure: the retries keep the dependency saturated long after the original fault is gone. Either break the circuit at the caller or put an adaptive limiter in front of the callee.

Severity: error
Selects: every edge
Components in scope: 2,366

What it checks

It looks at every edge, and fires when:

— all of these hold:
— the edge's `retryPolicy` is one of "fixed", "exponential" or "exponential-jitter"
— this does not hold:
— the edge's `circuitBreaker` is true
— the edge's target node:
— this does not hold:
— the node has at least one incoming edge from a rate-limiter

Message

Edge '{{edge.from}}' -> '{{edge.to}}' retries without a circuit breaker, and '{{edge.to}}' has no rate limiter in front of it. Retry storms will cascade.

{{node.id}} style paths are filled in from the node, the edge or the design the finding is about.

Applies to · 2,366 components

This rule is about an edge, so it belongs to a pair of nodes rather than one component.

This rule names no kind: it reads the design as a whole.

Tripped by · 30

Reference designs whose broken twin names this rule in its header. The gate reads the same header, and fails when a rule no design trips.

Source

1# yaml-language-server: $schema=../schema/rule.schema.json
2# Retries into a service that cannot shed load or open a circuit are the recipe for a
3# metastable failure: the retries keep the dependency saturated after the original
4# fault is gone. Either break the circuit on the caller or put an adaptive limiter in
5# front of the callee.
6# Tripped by: designs/doordash-delivery.broken.yaml
7schemaVersion: "1"
8type: "rule"
9id: "retry-without-adaptive-shedding"
10severity: "error"
11select:
12  edge: {}
13when:
14  all:
15    - field: "retryPolicy"
16      in: ["fixed", "exponential", "exponential-jitter"]
17    - not:
18        field: "circuitBreaker"
19        equals: true
20    - to:
21        not:
22          hasEdge:
23            direction: "in"
24            peerKind: "rate-limiter"
25message: "Edge '{{edge.from}}' -> '{{edge.to}}' retries without a circuit breaker, and '{{edge.to}}' has no rate limiter in front of it. Retry storms will cascade."

The same page as markdown, for an agent: /catalogue/rules/retry-without-adaptive-shedding.md