Vocabulary for Load Balancing and Traffic Routing

Load balancing conversations get imprecise fast — “it’s routing traffic weirdly” doesn’t tell a teammate whether the problem is a health check, a routing algorithm, or session affinity. This vocabulary shows up constantly in incident calls, capacity planning discussions, and architecture reviews, so being specific about which mechanism is involved matters for getting to a fix quickly.

Foundational Concepts

1. Load balancer

A component that distributes incoming requests across multiple backend servers, aiming to spread load evenly and avoid overwhelming any single instance.

Usage: “The load balancer is sending disproportionate traffic to one instance — that’s usually a sign its health check or routing algorithm needs a second look.”

2. Health check

A periodic probe the load balancer sends to each backend to determine whether it’s healthy enough to receive traffic, removing unhealthy instances from rotation automatically.

Usage: “This instance kept receiving traffic during its slow restart because the health check endpoint returned success before the app had actually finished initializing.”

3. Backend (upstream)

An individual server or service instance that a load balancer forwards requests to, sometimes called an upstream in configuration contexts like Nginx.

Usage: “We added a new backend to the pool, but forgot to update the health check path, so the load balancer never marked it as ready.”

4. Round robin

A load balancing algorithm that distributes requests sequentially and evenly across all available backends, regardless of each backend’s current load.

Usage: “Round robin works fine here since all our instances are identical in capacity, but it wouldn’t handle a mixed fleet with different instance sizes well.”

5. Least connections

A load balancing algorithm that routes each new request to the backend currently handling the fewest active connections, useful when request processing time varies significantly.

Usage: “We switched to least connections because some of our requests take much longer than others, and round robin was sending new requests to already-busy instances.”

Routing and Distribution

6. Session affinity (sticky sessions)

A configuration where a load balancer routes all requests from the same client to the same backend instance, typically to preserve in-memory session state.

Usage: “We rely on sticky sessions here since session data lives in each instance’s memory — without affinity, a user could get bounced to an instance that doesn’t have their session.”

7. Failover

The automatic redirection of traffic away from a failed or unhealthy component to a working one, ideally with minimal or no disruption to the end user.

Usage: “Failover kicked in within a few seconds of the primary instance failing its health check, and traffic shifted cleanly to the standby without dropping requests.”

8. Circuit breaker

A pattern that stops sending requests to a failing dependency after a threshold of failures, allowing it time to recover instead of continuing to overwhelm it with traffic.

Usage: “The circuit breaker tripped after this dependency’s error rate crossed our threshold, which actually protected the rest of the system from cascading failures.”

9. Rate limiting

The practice of capping the number of requests a client can make in a given time window, protecting a service from being overwhelmed by a single source.

Usage: “We’re rate limiting this endpoint per API key now, after one misbehaving client’s retry loop nearly saturated the whole cluster.”

10. Traffic shaping

The deliberate control of how much and what kind of traffic flows to a system, such as gradually ramping up load during a rollout rather than sending it all at once.

Usage: “We’re using traffic shaping to send only 5% of requests to the new version initially, ramping up gradually as we confirm it’s behaving correctly.”

Deployment Patterns

11. Canary deployment

A rollout strategy where a new version receives a small percentage of production traffic first, allowing issues to be caught before a full rollout.

Usage: “We caught this regression during the canary deployment stage, when it was only affecting five percent of traffic, well before it reached everyone.”

12. Blue-green deployment

A deployment strategy where two identical environments (blue and green) exist, with traffic switched entirely from one to the other, allowing an instant rollback if needed.

Usage: “With blue-green deployment, rolling back this bad release was just a matter of switching the load balancer back to the previous environment — no redeploy required.”

13. Weighted routing

A traffic distribution strategy where different backends or versions receive different proportions of traffic based on assigned weights, rather than an even split.

Usage: “We’re using weighted routing to send 90% of traffic to the stable version and 10% to the experimental one while we validate it.”

14. Draining (connection draining)

The process of allowing existing connections to a backend to finish naturally before removing it from rotation, rather than terminating in-flight requests abruptly.

Usage: “We enabled connection draining before taking this instance out of rotation, so in-flight requests complete instead of getting dropped mid-response.”

15. Latency-based routing

A routing strategy that directs traffic to the backend or region expected to serve the request with the lowest latency, often used across geographically distributed deployments.

Usage: “Latency-based routing sends European users to our Frankfurt region automatically, without needing any client-side logic to pick the closest endpoint.”

Key Takeaways

• Name the specific mechanism (health check, routing algorithm, session affinity) rather than describing traffic behavior only vaguely as “weird” or “broken.”
• Distinguish round robin from least connections deliberately based on whether request processing time varies significantly across traffic.
• Use canary and blue-green deployment vocabulary precisely when proposing a rollout strategy, since they represent genuinely different risk and rollback tradeoffs.
• Mention connection draining explicitly when discussing how instances are removed from rotation, to avoid abruptly dropping in-flight requests.
• Reach for circuit breakers and rate limiting as distinct, complementary protective patterns — one protects a failing dependency, the other protects against an overwhelming client.

In Practice: Navigating Nuance in Team Communication

Understanding the terminology around load balancing and traffic routing is only half the battle. The real challenge for non-native English speakers often lies in how you communicate those concepts within a technical team. It’s not just about knowing that “session affinity” means directing users to the same server, but also being able to explain why it’s important, or suggest an alternative approach clearly and confidently. Let’s consider a few scenarios.

During a code review of a new feature implementing a round-robin load balancing algorithm, your senior engineer might leave a comment: “This implementation lacks detailed health check integration; relying solely on CPU utilization is a brittle strategy. We need to ensure proactive detection of server failures.” The key here isn’t just understanding the words “brittle strategy,” but grasping the implication – that the current approach is prone to failure and requires more robust monitoring. It’s about responding with something like, “I understand your concern regarding the health checks. I’ve added a script to periodically ping each server and log any unresponsive instances. This provides a more immediate feedback loop than just CPU utilization.” Notice the phrasing: acknowledging the original comment (“I understand…”), explaining what you’ve done (“I’ve added…”), and stating the benefit (“This provides…”).

Another situation might arise in a Pull Request description for a change to a traffic routing policy. Let’s say you’re introducing weighted round-robin. A less detailed description could read, “Updated load balancing algorithm to use weights.” This is too vague! A more effective approach would be: “Implemented weighted round-robin traffic distribution across the three application servers. The weights are configured based on server capacity and current request volume as measured by our monitoring system. This allows us to dynamically adapt to fluctuating demand, ensuring optimal performance during peak periods.” Again, it’s about conveying why the change was made – to handle dynamic workloads effectively—and providing context for the technical details.

Finally, think about Slack conversations. You might receive a message: “Server 2 is overloaded! What’s going on?” A simple response like “Check CPU usage” isn’t sufficient. A more polished reply would be: “Let me investigate. I’m running kubectl top pods -n my-app to see which pods are consuming the most resources and will update you with my findings in a few minutes.” This demonstrates proactive investigation and provides a clear timeline for resolution, using familiar command-line tools in a professional context.

Here’s an example of how you might use kubectl top pods to diagnose resource issues:

kubectl top pods -n my-app

This command retrieves the CPU and memory usage statistics for all pods within the my-app namespace, giving you immediate data to inform your response. Focusing on clear, contextualized explanations – demonstrating understanding of the underlying principles – will significantly improve your ability to collaborate effectively with a technical team.