A practical guide for platform teams managing shared AI deployments

Fee Limiting vs. Quota Reservations: when to make use of every

You could have a single gpt-oss-20b deployment. Six groups wish to use it. Advertising is working batch summarization jobs at 3am. The fraud crew wants sub-second responses 24/7. An intern’s Jupyter pocket book is unintentionally hammering the endpoint in a good loop. And your GPU invoice is already eye-watering.

Sound acquainted? DataRobot offers you two instruments to resolve this: Fee Limiting and Quota Reservations. This put up explains when to succeed in for every, backed by an actual load check instance on a staging deployment.

Fee Limits and Quota Reservations, in plain English

Fee Limits – Obtainable in DataRobot v11.4

Fee limits units per-consumer caps throughout a number of dimensions: requests per minute, token rely per hour, concurrent requests, and enter sequence size. A default coverage applies to all customers, with per-entity exceptions out there for particular overrides.

What it protects in opposition to: Any single client overconsuming — whether or not via excessive request quantity, massive inputs, or extreme concurrency.

Quota Reservations – out there in DataRobot v11.9

Quota reservations outline the deployment’s whole potential throughput (worth per minute) and a utilization threshold that triggers enforcement. Inside that finances, particular entities could be allotted a reserved share — guaranteeing them a minimal slice of capability that different customers can’t take away.

What it protects in opposition to: Precedence hunger. With out reservations, a loud neighbor can devour your entire capability finances, leaving your crucial workloads with nothing.

How Fee Limits and Quota Reservations work collectively (and aside)

Used alone, every instrument solves a particular downside:

• Fee limiting alone caps whole throughput. Underneath saturation, all customers compete equally — first come, first served.
• Quota reservations alone assure minimal throughput for particular customers, no matter what others are doing.

Collectively, they offer you each management surfaces: a ceiling that protects the mannequin and assured flooring for the customers that matter most.

Load testing a multi-tenant deployment

To judge these options underneath stress, we load-tested a gpt-oss-20b deployment in our staging setting. The setup simulates an actual multi-tenant situation: 4 customers sharing one mannequin, every with totally different precedence ranges.

Instance configuration

This left a 20% unreserved pool (200 RPM) for the dev consumer and any overflow.

Instance load profile

We ran six escalating situations over 17 minutes to watch behaviour at totally different saturation ranges:

When to make use of Fee Limiting alone

Fee limiting by itself is the best selection when:

• All customers are equally necessary. If no crew’s site visitors is extra crucial than one other’s, there’s no want for reservations. Equal competitors underneath saturation is truthful sufficient.
• You simply want to guard the GPU. Your major concern is {that a} spike in site visitors doesn’t degrade mannequin latency or trigger OOM errors. You desire a security valve, not a site visitors coverage.
• You could have a single client. If there’s just one utility hitting the deployment, reservations are meaningless — there’s nobody to order in opposition to.

What the instance confirmed

In the course of the regular site visitors situation (~600 RPM mixed, effectively under the 800 RPM utilization threshold), the speed limiter was invisible and all 4 customers achieved 100% success charges with zero rejected requests.

Dimension your reservations based mostly on absolutely the minimal throughput every client requires throughout peak rivalry. That is by design, so that you’re not penalizing regular site visitors.

And it protects the mannequin even underneath excessive abuse. In the course of the excessive overload situation (20,000+ RPM in opposition to 1,000 RPM capability, which is a a 20× overload), the speed limiter rejected 95% of requests. However the mannequin itself stayed completely wholesome:

The speed limiter acted as a firewall between chaotic shopper demand and secure mannequin inference. With out it, these 20,000 requests per minute would have queued up contained in the NIM, latency would have ballooned, and the mannequin would have successfully turn out to be unusable for everybody.

Takeaway: In case your solely objective is “don’t let site visitors spikes kill the mannequin,” fee limiting alone is ample and zero-config past setting the capability quantity.

When so as to add Quota Reservations

Quota reservations turn out to be important when:

• Some customers are extra necessary than others. Your fraud detection system can’t afford to be starved out by a batch analytics job. Your manufacturing agent wants assured throughput {that a} developer’s check harness can’t steal.
• You could have a multi-tenant deployment. A number of groups, purposes, or downstream deployments share the identical mannequin. With out reservations, the loudest client wins.
• You need predictable SLAs. For those who’ve promised a crew “your utility will get at the least 300 RPM,” reservations are the way you implement that promise on the infrastructure degree.
• You could have a mixture of interactive and batch workloads. Batch jobs are bursty and can fortunately devour all out there capability. Reservations guarantee interactive workloads nonetheless get their share throughout batch spikes.

Easy methods to measurement reservations

Dimension your reservations based mostly on absolutely the minimal throughput every client requires throughout peak rivalry.

Guidelines of thumb:

• Don’t reserve 100%. Go away an unreserved pool (10–20%) for ad-hoc site visitors, new customers, and overflow. For those who reserve all the things, any new utility will get zero throughput till you reconfigure.
• Dimension reservations to minimal wants, not peak wants. Reservations assure a flooring, not a ceiling. An entity with 30% reserved can nonetheless use greater than 30% when capability is accessible.
• Match reservation measurement to enterprise criticality, not crew measurement. Your fraud detection system might need fewer requests than your analytics pipeline, however it wants assured entry extra.

In our instance, three manufacturing brokers acquired 30%/20%/30% reservations, leaving a 20% unreserved pool for the dev consumer. This meant the dev consumer might nonetheless use the deployment — they simply wouldn’t get assured entry throughout rivalry.

Do reservations work underneath actual load?

At slight overload (1.2× capability): The system degrades gracefully

In the course of the slight overload situation (~1,200 RPM in opposition to 1,000 RPM capability), all 4 customers achieved 100% success — the token bucket’s burst capability absorbed the slight overage. That is the “swish degradation” zone the place reservations aren’t but wanted, however the system is proving it may well deal with bursts.

At heavy-to-extreme overload (7–12× capability): reservations keep a assured flooring

When all 4 customers fired as quick as potential (7,000–12,000 RPM in opposition to a 1,000 RPM capability), the system was overwhelmed. Right here’s what every client skilled throughout the complete check:

Why the success charges look related: At 12× overload, even a 300 RPM reservation is simply ~2.5% of what every client is trying to ship (~3,000 RPM per client vs. a 300 RPM assure). The reservation works by guaranteeing every client receives its assured 200–300 RPM. Nonetheless, as a result of 97% of whole site visitors is rejected throughout excessive overloads, the relative share variations compress.

The extra revealing metric is absolute throughput. Reserved customers accomplished 4,172–4,332 profitable requests. The unreserved dev consumer accomplished 2,828 — about 34% fewer. Even accounting for the dev consumer’s shorter energetic time, reserved customers constantly acquired extra requests via throughout shared situations.

At saturation with a late joiner: reservations shield incumbents

Within the late joiner situation, the three manufacturing brokers have been already flooding the system when the dev consumer joined 60 seconds later. With all reserved capability spoken for, the dev consumer was confined to the 20% unreserved pool (~200 RPM). The manufacturing brokers continued drawing from their assured buckets, unaffected by the brand new arrival.

That is the situation that issues most in manufacturing. A batch job kicks off, or a brand new utility goes dwell, and immediately there’s extra demand than provide. With out reservations, the brand new load pushes everybody’s throughput down equally. With reservations, your crucial customers are shielded.

Reserved customers compete pretty amongst themselves

Within the reserved-only situation, the dev consumer went silent and solely the three manufacturing brokers competed. Their success charges have been almost an identical (28.9%–30.2%) — the system divided throughput proportionally throughout their reservations.

What the server sees: OTEL metrics inform the story

Shopper-side metrics (success charges, 429 counts) let you know what your customers skilled. Server-side OTEL metrics let you know what the platform skilled. Right here’s what our instance deployment regarded like from the within.

The speed limiter protects mannequin well being

Throughout peak load (20,596 requests/minute hitting the endpoint), the NIM was serving solely the ~1,000 RPM that the speed limiter let via:

With out fee limiting, these 20,000 RPM would have queued contained in the NIM. The GPU wouldn’t have gotten extra productive — it’s already at 91–95% — however latency would have spiraled as requests stacked up. As a substitute, the speed limiter rejected extra requests instantly (at 429-response speeds, not inference speeds), holding the mannequin responsive for the site visitors it did settle for.

Token throughput follows profitable requests

Peak token throughput was ~199,350 tokens/min (whole), with ~115,939 enter and ~83,411 output. These numbers monitor straight with the speed limiter’s allowed throughput — not with the tried request quantity. One other means of seeing that the speed limiter is accurately shaping site visitors.

Deciding between Fee Limits and Quota Reservations

Use this flowchart to determine what to configure:

Step 1: Do you may have a shared deployment with a number of customers?

• No → Fee limiting alone is ample. Set capability to guard the GPU and transfer on.
• Sure → Proceed to Step 2.

Step 2: Are all customers equally necessary?

• Sure → Fee limiting alone could also be sufficient. Underneath saturation, all customers compete equally — first come, first served. If that’s acceptable, cease right here.
• No → Proceed to Step 3.

Step 3: Do any customers want assured minimal throughput?

• Sure → Add quota reservations. Dimension them to the minimal RPM every crucial client wants throughout peak rivalry.
• No, however some customers have to be deprioritized → Use per-entity exceptions as an alternative of reservations. Cap the noisy neighbors relatively than guaranteeing the crucial ones.

Step 4: Configure the unreserved pool.

• Don’t reserve 100% of capability. Go away 10–20% unreserved for ad-hoc site visitors, overflow, and new purposes that haven’t been assigned reservations but.

Sensible configuration suggestions

Begin with fee limiting solely. Monitor your deployment’s site visitors patterns for per week. Take a look at peak RPM, who’s sending what, and whether or not anybody is constantly overconsuming. Then add reservations the place the info tells you they’re wanted.

Set utilization threshold at 70–80%. This provides the token bucket burst room to soak up brief spikes with out triggering fee limiting on each minor fluctuation. In our instance, we used 80% and the system dealt with 1.2× capability gracefully earlier than enforcement kicked in.

Monitor with OTEL metrics. After configuring fee limiting, test these server-side metrics to verify issues are working:

• deployment.requests vs deployment.requests.rate_limited — are you rejecting the correct amount?
• nvidia_gpu_utilization — is the mannequin nonetheless saturated or did fee limiting create headroom?
• nvidia_vllm:e2e_request_latency_seconds — is latency secure underneath load?
• deployment.concurrent_requests — are requests queuing up or flowing easily?

Reservation sizing formulation:

Reserved RPM = Capability × Reserved %

Instance: 1000 RPM × 30% = 300 RPM assured

Don’t confuse this with a fee restrict. A 30% reservation means “you’ll at all times get at the least 300 RPM, even when the system is saturated.” The entity can nonetheless use extra when capability is accessible.

Abstract

When contemplating Fee Limiting vs. Quota Reservations: use every instrument the place it suits. Layer them the place the issue calls for it.