API Gateway Patterns Vocabulary

The API gateway pattern solves the cross-cutting concerns problem in microservice architectures.

Without an API gateway — each microservice must handle:
• JWT validation: every service imports and runs token verification
• Rate limiting: every service has its own rate limiting logic and state
• CORS headers: every service configures allowed origins
• TLS termination: every service manages certificates
• Request logging: every service emits its own access logs
• API versioning: every service manages its own URL versioning

With an API gateway — centralised handling:

Client
  ↓
[API Gateway]
├── Auth check (JWT validation)
├── Rate limiting
├── Request transformation
├── TLS termination
└── Route to:
    ├── /orders/* → Orders service (HTTP)
    ├── /users/*  → Users service (HTTP)
    └── /products/* → Products service (gRPC, translated to HTTP)

Key API gateway products:
• Kong, AWS API Gateway, Azure API Management, Google Cloud Apigee
• NGINX, Traefik, Envoy (infrastructure-level proxies with gateway features)
• Apollo Router (GraphQL-specific gateway)

Key vocabulary:
• API gateway — the single entry point that routes requests and handles cross-cutting concerns
• Cross-cutting concerns — functionality needed by all services (auth, logging, rate limiting)
• TLS termination — ending the encrypted HTTPS connection at the gateway level; backend services use plain HTTP
• Reverse proxy — a proxy that forwards client requests to backend servers (the core of a gateway)

Rate limiting protects backend services from abuse, scrapers, and accidental traffic spikes. The algorithm choice affects burst tolerance.

Fixed window algorithm:
• Limit: 100 requests per minute
• Counter resets at :00 of every minute
• Problem — boundary burst: a client sends 100 requests at :59, then 100 more at :01 — 200 requests in 2 seconds, both "within limits"

Sliding window algorithm:
• The window always covers the most recent 60 seconds from the current moment
• No boundary — the rate is smooth
• More accurate but more expensive to compute (requires storing per-request timestamps or using a Redis sorted set)

Token bucket algorithm (common in API gateways):
• Each client has a "bucket" with a capacity of N tokens
• Tokens refill at a fixed rate (e.g., 10 tokens/second)
• Each request costs 1 token; if no tokens, request is rejected (HTTP 429)
• Naturally handles bursts: a client with a full bucket can burst, but sustained rate is limited by refill speed

Rate limiting response codes:
• HTTP 429 Too Many Requests — the standard code for rate limit exceeded
• Retry-After header — tells the client how many seconds to wait before retrying

Key vocabulary:
• Rate limiting — restricting request frequency per client/IP/API key per time period
• HTTP 429 — standard status code for rate limit exceeded
• Token bucket — a rate limiting algorithm that allows controlled bursting
• Throttling — another word for rate limiting; sometimes used to mean "slow down" rather than "reject"

The circuit breaker pattern prevents cascading failures by failing fast when a dependency is clearly unhealthy.

Circuit breaker states:

[CLOSED] → normal operation; requests pass through
   ↓ failure threshold exceeded (e.g., 5 failures in 10s)
[OPEN] → all requests fail immediately without calling the service
   ↓ timeout expires (e.g., after 30 seconds)
[HALF-OPEN] → allow one probe request through
   ↓ probe succeeds → [CLOSED]
   ↓ probe fails   → [OPEN] (reset timeout)

Why fast-failing is better than waiting for timeouts:
• Without circuit breaker: a 30-second timeout × 50 concurrent requests = 50 threads blocked for 30 seconds each → thread pool exhaustion → cascade failure
• With circuit breaker (open): all 50 requests fail in <1ms → threads are freed immediately → the rest of the system stays healthy

Fallback strategies for an open circuit:
• Return stale cached data (slightly outdated inventory count)
• Return a degraded response (show "check availability in store" instead of live count)
• Return HTTP 503 with a Retry-After header

Key vocabulary:
• Circuit breaker — a pattern that stops requests to a failing service to prevent cascade failures
• Closed state — normal operation (circuit allows current to flow)
• Open state — service considered unhealthy; requests fail immediately without being forwarded
• Half-open state — testing if the service has recovered by allowing a probe request
• Fast fail — immediately returning an error without waiting for a timeout

The BFF pattern solves the "one API for all clients" problem — different clients have genuinely different data needs.

The problem with a single gateway for all clients:
• Mobile app needs compact responses (limited bandwidth, small screens)
• Web app needs rich, detailed data for a dashboard
• A partner API needs a stable, versioned interface
• The unified gateway must serve all three → compromise responses that are too big for mobile and too small for the web dashboard

BFF architecture:

[Mobile App]   →  [Mobile BFF]   → microservices
[Web App]      →  [Web BFF]      → microservices
[Partner API]  →  [Partner BFF]  → microservices

What each BFF does for its client:
• Data aggregation: calls multiple services and composes the response the client needs
• Response shaping: strips unnecessary fields, formats dates in the client's preferred timezone/format
• Authentication translation: the mobile BFF handles OAuth tokens; the partner BFF handles API keys
• Protocol translation: the web BFF may call a gRPC backend and return JSON/REST to the browser

Tradeoff: BFF adds operational complexity (more services to deploy and monitor). It's worth it when different clients' needs diverge significantly.

Key vocabulary:
• BFF (Backend for Frontend) — a dedicated backend service optimised for a specific client type
• Data aggregation — combining data from multiple services into a single composite response
• Response shaping — transforming the response structure/content to match the client's needs
• Protocol translation — converting between protocols at the gateway (gRPC → REST, REST → GraphQL)

The north-south vs east-west traffic distinction is fundamental to API gateway and service mesh architecture.

North-south traffic (use API gateway):
• External clients (browsers, mobile apps, partner systems) making requests into the cluster
• Requires internet-facing auth, rate limiting, DDoS protection, TLS termination
• The API gateway is the right tool here

East-west traffic (don't use API gateway):
• Service A in the cluster calling Service B in the same cluster
• Routing via the gateway adds: an extra network hop, gateway processing time, and the gateway becomes a bottleneck for ALL internal traffic
• The gateway also becomes a single point of failure — if it goes down, ALL inter-service communication fails

Better tools for east-west traffic:
• Kubernetes Service DNS: http://orders-service.default.svc.cluster.local — direct, no extra hop
• Service Mesh (Istio, Linkerd): adds mTLS, observability, and traffic management (retries, circuit breaking) for internal calls without routing through a central gateway
• Service discovery (Consul, Eureka): services find each other directly via a registry

Key vocabulary:
• North-south traffic — network traffic flowing between external clients and the cluster (the gateway's primary domain)
• East-west traffic — network traffic flowing between services within the cluster
• Service mesh — infrastructure layer for managing internal service-to-service communication (mTLS, observability, circuit breaking)
• Single point of failure (SPOF) — a component whose failure brings down an entire system or traffic path