Understanding the Problem
A rate limiter is a control layer that restricts how frequently a client can access an API during a defined period of time. You can think of the rate limiter as a gate in front of the application. It decides which requests are allowed through and which should be slowed down or rejected.
Rate limiting serves several purposes: it prevents abusive or accidental traffic from overwhelming backend services, protects the system from sudden request spikes, and distributes limited capacity more fairly so that one client cannot consume resources at the expense of everyone else.
Functional Requirements
- The system should recognize callers using identifiers such as a user ID, IP address, or API key so that different clients can be rate limited independently.
- The platform should enforce configurable policies for different request types and identities. For example, an account might be allowed to perform up to 60 search requests within a one-minute window.
- When a caller exceeds the configured allowance, the request should be rejected with HTTP
429 Too Many Requests. The response should also expose useful metadata, such as how much quota remains and when the client can begin sending requests again.
Non-Functional Requirements
Before choosing the architecture, we should establish the expected traffic scale with the interviewer. A rate limiter protecting a small internal API has very different needs from one sitting in front of a globally distributed consumer platform.
- Each rate-limit decision should add very little overhead to the request path, ideally less than 10 milliseconds.
- The rate limiter should remain highly available. Strong consistency across every node is not required; small temporary differences in counters are acceptable if they help keep the system responsive during failures or replication delays.
- The architecture should support approximately 1 million request checks per second while serving a user base of around 100 million daily active users.
Data Model
-
Rule: Represents one rate-limiting policy. A rule defines how much traffic is allowed, the length of the enforcement window, which callers it applies to, and which API operations it protects. For example, one rule might allow authenticated users 2,000 requests per hour, while another limits password-reset attempts to 5 requests every 10 minutes per IP address.
-
Client: Represents the identity whose usage is being measured. Depending on the API, this could be a user ID, IP address, API key, tenant ID, or a combination of several identifiers. The system tracks each client’s consumption against the rules that apply to it.
-
Request: Represents an incoming API call that must be evaluated before reaching the backend. It carries information such as the caller identity, endpoint, HTTP method, and request time, which the limiter uses to determine the relevant policy and current usage.
These entities come together on every request. The system identifies the client, determines which rules apply to the requested endpoint, checks the client’s current usage against those limits, and then returns an allow or deny decision. That interaction between Request, Client, and Rule forms the basic data model for the rate-limiting system.
The rate limiter exposes a small internal interface that application services can call before processing a request. Its job is to return a fast allow-or-reject decision together with the client’s remaining quota information. A simple interface could look like this:
checkRateLimit(clientId, ruleId) -> { allowed: boolean, remaining: number, resetAt: timestamp }
The caller provides the identity being limited—such as a user ID, IP address, or API key—along with the rule that should be evaluated.
The response indicates whether the request can proceed, how much quota is still available, and when the current limit resets. The API gateway or application service can then use those values to populate headers such as X-RateLimit-Remaining and X-RateLimit-Reset.
High-level Design
The System Should Identify Clients by User ID, IP Address, or API Key
Before enforcing any rate limit, we need to answer two architectural questions. First, where should the rate limiter sit in the request path? Its location determines which traffic it can block and what request information it can inspect.
Second, how do we define the identity being limited? Depending on the product, that identity might be a logged-in user, an IP address, an API key, or some combination of them.
These decisions are related. The placement of the limiter determines what identity information is readily available, while the type of identity we want to enforce may influence where the limiter should run. There are several possible places to enforce limits, but for a general-purpose API, one of the most useful locations is at the system boundary.
Approach: Rate Limiting at the API Gateway
We can integrate the rate limiter directly into the API Gateway or edge proxy. Every incoming request reaches this layer before touching the application services. The gateway extracts relevant request information, evaluates the applicable rate-limit rules, and makes an immediate decision.
If the request is allowed, it continues to the appropriate backend service. If the caller has exhausted its quota, the gateway terminates the request immediately and returns: HTTP 429 Too Many Requests.
This placement is attractive because rejected traffic never consumes application-server resources. Imagine a client accidentally sends 50,000 requests per second to an endpoint that normally receives only a few hundred. If the limiter lives inside the application service, those requests still consume load-balancer connections, application threads, CPU, and memory before being rejected.
At the gateway, unwanted traffic is stopped before it reaches those downstream components. This makes the rate limiter behave like an admission-control layer: traffic that violates policy is rejected at the entrance rather than after it has already consumed expensive backend capacity.
Challenges
The trade-off is that the gateway has limited business context. It can easily inspect information already contained in the HTTP request, including:
- URL and HTTP method
- Source IP address
- Authorization headers
- API keys
- JWT claims
- Query parameters
- Other request headers
But it should avoid making expensive calls to downstream databases or services for every request. Doing so would add latency to a component that sits on the critical path of all traffic.
For example, suppose premium accounts are allowed ten times more traffic than free accounts. The gateway can enforce this efficiently if the subscription tier is included in a trusted JWT claim.
If determining the tier requires a database query on every request, rate limiting itself may become a performance bottleneck. This usually means that information required for enforcement should either be embedded in authenticated request metadata or cached close to the gateway.
The limiter also needs somewhere to maintain counters and expiration state. A distributed in-memory system such as Redis is a natural option because these lookups need to happen quickly.
That introduces another dependency, however. We eventually need to decide what happens when the counter store is slow, partitioned, or unavailable. We will address those failure cases later.
For this design, we will place the limiter at the API Gateway. It gives us centralized enforcement and prevents rejected requests from reaching application services. The next question is how the gateway identifies who should be limited.
How Do We Identify a Client?
Because the limiter operates at the gateway, we want to derive identity primarily from information already available in the request. There are three common identifiers.
User ID
For authenticated consumer APIs, the user account is often the most natural unit of enforcement. After the gateway validates the authentication token, it can extract a user identifier from the trusted JWT claims: userId = 482901. Each authenticated account then receives its own quota.
For example:
user:482901 limit: 2,000 requests/hour
This avoids penalizing one user because another user happens to share the same network.
IP Address
For anonymous traffic, we may not have a user identity at all. In that case, the source IP can act as the rate-limit key:
ip:203.0.113.42 limit: 60 requests/minute
IP-based limiting is useful for unauthenticated endpoints such as login, signup, password recovery, or public content APIs.
However, IP addresses are an imperfect identity. Thousands of users inside the same company, university, hotel, or mobile carrier may appear behind one public IP because of NAT. Applying an overly strict IP limit could accidentally throttle many legitimate users together.
The gateway also needs to determine the real client IP carefully when requests pass through trusted proxies or CDNs. Headers such as X-Forwarded-For should only be trusted when they were inserted or sanitized by infrastructure we control.
API Key
Developer and partner APIs commonly identify clients using an API key. For example: X-API-Key: abc123... The gateway can map that key to its own quota:
api_key:abc123 limit: 50,000 requests/day
This allows different customers, integrations, or applications to receive independent limits even when they originate from the same network.