Skip to content

Rate Limiting

SlidingWindowStrategy

chatbot_plugin_sdk.rate_limit.SlidingWindowStrategy 

SlidingWindowStrategy(
    rpm: int = 0, tpm: int = 0, rpd: int = 0
)

Sliding-window rate limiter that tracks RPM, TPM, and RPD.

Thread-safe (threading.Lock guards mutable state) and async-safe (asyncio.sleep yields the event loop during waits).

A single instance can be shared across threads — each thread's asyncio.run() call uses asyncio.sleep in its own event loop, but state is protected by the threading lock.

Parameters:

Name Type Description Default
rpm int

Max requests per minute. 0 disables this limit.

 0
tpm int

Max tokens per minute (estimate: 4 chars ≈ 1 token). 0 disables.

 0
rpd int

Max requests per day. When reached, :exc:RateLimitExhausted is raised. 0 disables this hard cap.

 0

Usage::

strategy = SlidingWindowStrategy(rpm=10, tpm=40_000, rpd=1_500)
provider = EndpointProvider(
    url="https://generativelanguage.googleapis.com/...",
    dimension=768,
    api_key="AIza...",
    rate_limit=strategy,
)

acquire async 

acquire(estimated_tokens: int = 0) -> None

Async-friendly wait loop. Uses asyncio.sleep to yield the event loop.

record_usage 

record_usage(actual_tokens: int) -> None

Replace the last TPM estimate with the actual token count.

RateLimitStrategy Protocol

chatbot_plugin_sdk.rate_limit.RateLimitStrategy

Bases: Protocol

Protocol for injectable rate-limiting strategies.

Pass an instance to EndpointProvider(rate_limit=...). Omit the argument (or pass None) for internal services with no external rate limits.

acquire async 

acquire(estimated_tokens: int = 0) -> None

Await until a request slot is available. May raise :exc:RateLimitExhausted.

record_usage 

record_usage(actual_tokens: int) -> None

Correct the token estimate after a successful call.

Optional — useful when the API response includes the exact token count. For embeddings the estimate is usually close enough; omitting the call is fine.

RateLimitExhausted

chatbot_plugin_sdk.rate_limit.RateLimitExhausted

Bases: Exception

Raised when the daily request cap (rpd) is reached.

Callers may catch this to fall back to a different provider or abort.