Building a Distributed AI Rate Limiting Server with RPM and TPM Support Using Envoy Ratelimit
For conventional HTTP APIs, limiting the number of requests per minute is usually enough. For large language model APIs, however, limiting only the request count is far from sufficient: one request may consume only a few dozen tokens, while another may consume tens of thousands. Therefore, a practical AI gateway usually needs to support all of the following: RPM (Requests Per Minute): the number of requests per minute. TPM (Tokens Per Minute): the number of tokens per minute....