Carrier CPS limits: the approaches, animated

Plivo and Vobiz reject (they don't queue) SIP INVITEs that go over the account's calls-per-second limit. Plivo returns hangup 5180 cps_limit_reached and Vobiz returns 429 / SIP 503. Each tab below runs the same campaign burst through our pipeline under a different strategy: scheduler → orch-dispatch → worker → CreateSIPParticipant → carrier. Set up the campaign in Providers: add or remove carriers (up to 5) and set each one's calls and CPS. Each carrier enforces its own account limit, independently of the others. The default is 10 calls on carrier A (2 CPS) and 10 on carrier B (4 CPS). With a single provider, tab 4 shows the per-tenant pitfall instead of the global-key one. In Workers & LiveKit, set the worker pods and their limits: LiveKit assigns each job to a worker, and each job then needs a pre-started process (or waits for a cold start) before it can dial.

Side by side, with the current settings

The same burst, the same random warm-up times, run through every approach. Change the sliders above and this table updates.

How the gate decides: "defer to the next free slot"

-- KEYS[1] = {voicebot:cps}:<carrier_account>
-- ARGV[1] = interval_ms (1000/CPS × 1.15)
-- ARGV[2] = max_wait_ms
local t   = redis.call('TIME')  -- Redis clock, not pod clocks
local now = t[1]*1000 + math.floor(t[2]/1000)
local tat = tonumber(redis.call('GET', KEYS[1]) or now)
if tat < now then tat = now end
local wait = tat - now
if wait > tonumber(ARGV[2]) then return -1 end
local iv = tonumber(ARGV[1])
redis.call('SET', KEYS[1], tat + iv, 'PX', wait + iv + 1000)
return wait

async def wait_for_dial_slot(redis, key, cps, max_wait_s=30):
    interval_ms = int(1000 / cps * 1.15)
    wait_ms = await redis.evalsha(GCRA_SHA, 1, key,
                                  interval_ms, int(max_wait_s * 1000))
    if wait_ms < 0:
        raise CpsBackpressure(key)   # → DEFERRED(CPS_LIMIT), no attempt used
    await asyncio.sleep(wait_ms / 1000)

# sip_dispatch.create_sip_participant, just before the RPC
await wait_for_dial_slot(
    redis, f"{{voicebot:cps}}:{telephony.rate_limit_key}",
    telephony.cps_limit)             # new fields on the telephony blob
info = await lkapi.sip.create_sip_participant(request)

// DispatchCall: plan a dial slot per provider before CreateAgentDispatch
slot := pacer.Reserve(ctx, provider.RateLimitKey, provider.CPS) // same GCRA
if due := slot.Add(-warmLead); due.After(now) {
    // per-message delay (x-delay plugin or TTL+DLX), not the 30 s DEFERRED path
    return publisher.PublishCallQueuedAfter(ctx, call, due.Sub(now))
}
// otherwise dispatch now; the worker gate (A) still has the final say

except SipDispatchError as e:
    if is_cps_reject(e) and attempt < 3:   # 429 / 503 / 5180, confirm from logs
        await asyncio.sleep(random.uniform(0.6, 1.6))
        continue                            # re-reserve a slot, re-dial
    raise

Model assumptions (illustrative): dispatch 0.25–0.45 s per message, 10 dispatch consumers. The governor admits a call only while calls in progress < pods × calls per worker; otherwise it defers the call (back in 30 s, outside this window). LiveKit takes 50–250 ms to assign a job and picks the worker with the lowest reported load below its cap (ties broken randomly). Workers report load every 0.5 s, so a burst can push a worker over its cap. Each worker keeps N processes pre-started. A job takes one if available; otherwise it waits for the next process to finish starting (cold start). Every process taken is replaced by a new one, which takes the process start time. Our code defaults are 3 calls per worker, 2 pre-started processes and ~8–9 s process start (core/constants.py). Job setup (room join, GetCallExecutionConfig, CheckDNDWindow, session build) 0.6–1.2 s. LiveKit→carrier latency 40–120 ms. The carrier meter is modelled as a sliding 1 s window over accepted INVITEs; neither carrier publishes its exact algorithm. Untracked traffic ≈ 0.6 INVITE/s. Random draws are seeded, so every approach sees the same warm-up times. With several providers, their calls are interleaved in the backlog, and untracked traffic hits the first provider's account. Total calls are capped at 40.