Advanced Reading #design-rfc #trade-offs #engineering #decisions

📜 Reading Internal Design RFCs

5 exercises — read one realistic internal engineering RFC and follow its structure: motivation and non-goals, proposed design, alternatives considered, trade-offs and open questions.

Anatomy of a design RFC

Motivation → the problem; watch for explicit non-goals that bound scope
Proposed design → the mechanism and the specific benefit it claims
Alternatives considered → each option paired with why it was rejected
Trade-offs → "we accept X because Y" = a deliberate, justified downside
Open questions → known unknowns, owners, "TBD", "needs sign-off"

0 / 5 completed

1 / 5

📜 RFC-021 — Motivation

RFC-021: Tiered Rate Limiting for the Public API
Status: Draft   Author: platform-team   Reviewers: 3

## Motivation

Today every API token is subject to a single global limit of 600
requests/minute. This was adequate at launch but is now causing two
problems. First, a handful of high-volume integration partners
regularly exhaust the limit and get throttled, generating support
tickets. Second, abusive free-tier traffic can consume capacity that
should be reserved for paying customers, because all tokens share one
pool.

We are NOT trying to solve per-endpoint cost weighting in this RFC;
that is tracked separately in RFC-024. The goal here is narrow: let
limits vary by plan tier.

The Motivation section says: "We are NOT trying to solve per-endpoint cost weighting in this RFC." Why do engineering RFCs typically include a statement like this?

2 / 5

📜 RFC-021 — Proposed Design

RFC-021: Tiered Rate Limiting (continued)

## Proposed Design

Introduce a 'rate_tier' column on the api_token table with values
{free, pro, partner}. The gateway reads the tier on each request and
applies the corresponding limit from a config map:

    free    -> 60 req/min
    pro     -> 600 req/min
    partner -> 6000 req/min

Limits are enforced with a token-bucket algorithm in Redis, keyed by
(token_id). Buckets refill continuously rather than resetting on a
fixed window, which avoids the "thundering herd" spike seen at the top
of each minute under the current fixed-window counter.

The design chooses a continuously-refilling token bucket over the current fixed-window counter. What specific problem does the author say this avoids?

3 / 5

📜 RFC-021 — Alternatives Considered

RFC-021: Tiered Rate Limiting (continued)

## Alternatives Considered

1. Per-customer custom limits stored in the billing system.
   Rejected: couples the gateway to the billing service on the hot
   path, adding latency and a new failure mode on every request.

2. Sliding-window log instead of token bucket.
   Rejected: accurate, but stores a timestamp per request, which is
   far more memory in Redis at our request volume. The token bucket is
   "good enough" and O(1) in memory per key.

3. Do nothing / raise the single global limit.
   Rejected: does not address the free-vs-paying isolation problem,
   which is the main driver for this work.

On what grounds was the sliding-window log alternative rejected?

4 / 5

📜 RFC-021 — Trade-offs

RFC-021: Tiered Rate Limiting (continued)

## Trade-offs

The token-bucket approach trades a small amount of burst tolerance for
much lower memory use. A client can briefly exceed its steady-state
rate by consuming a full bucket, then is limited to the refill rate.
For our use case this burst behaviour is desirable, not a defect.

Keying buckets by token_id (not by IP) means a single leaked token
cannot be mitigated by blocking an IP alone; revocation must go
through token rotation. We accept this because IP-based limiting is
ineffective behind shared NATs and corporate proxies.

The Trade-offs section notes that keying buckets by token_id (not by IP) has a consequence. What is it, and how does the author frame it?

5 / 5

📜 RFC-021 — Open Questions

RFC-021: Tiered Rate Limiting (continued)

## Open Questions

- Should the 'partner' tier be a hard ceiling, or soft (allow
  overage with alerting)? Leaning hard for v1; revisit after launch.

- What status code and headers do we return on throttle? Proposal:
  429 with Retry-After and X-RateLimit-Remaining. Needs sign-off from
  the API-docs owner before we commit to the header names publicly.

- Migration: existing tokens have no tier. Backfill plan TBD —
  default everyone to 'pro' on rollout, or infer from current plan?
  Owner: @maria, due before implementation starts.

In the Open Questions section, what does the line "Needs sign-off from the API-docs owner before we commit to the header names publicly" tell a reviewer about the status of the response-header decision?