Speculative Cascades: Latency-Optimal Inference via Coupled Draft-Verify Routing

1 Introduction

The cost of running a large language model is dominated by decoding, and decoding is slow for a reason that has nothing to do with how much arithmetic a token requires. Generating one token from an autoregressive transformer ^[22] at the small batch sizes typical of interactive serving requires streaming the model's entire weight matrix out of high-bandwidth memory and into the compute units, and it is that streaming, not the multiply-adds, that sets the wall-clock time ^[11]. The arithmetic intensity of batch-one decoding in sixteen-bit precision is on the order of half a floating-point operation per byte moved, which places it far below the ridge point of every modern accelerator's roofline ^[12]. The compute units sit mostly idle, waiting on memory. This single fact is the reason the two most successful inference-acceleration techniques of the last three years both work, and it is the reason we can unify them.

The first technique is speculative decoding. A small draft model proposes a block of several tokens, the large target model scores the whole block in one forward pass, and a verification test keeps the longest prefix of draft tokens that pass. Leviathan, Kalman, and Matias^[1] and, concurrently, Chen and colleagues at DeepMind ^[2] showed that a modified rejection-sampling test makes the output distribution identical to the target model's, so the speedup is free of any quality change. They report two to three times faster decoding on T5-XXL and Chinchilla-scale models respectively, with no loss. The mechanism exploits exactly the idle compute: because the target pass is memory bound, verifying a block of guesses costs about the same wall-clock time as generating one token alone, so every accepted draft token is a token the target did not have to generate sequentially.

The second technique is the model cascade. Run the cheap model first, measure a confidence or a score, and defer to the expensive model only when the cheap answer is not good enough. FrugalGPT^[3] reports matching GPT-4-level quality at up to a ninety-eight percent reduction in cost by cascading models of increasing size behind a learned scorer and threshold. The theory of when such deferral is optimal has been worked out for classification cascades by Jitkrittum and colleagues ^[6], and extended to the token level for generation by Gupta and colleagues ^[10]. Cascades trade a controlled amount of quality for a large amount of cost, which is precisely what speculative decoding refuses to do.

It is easy to state the two techniques as if the choice between them were obvious, but in practice teams agonize over it. Speculative decoding is safe, because it changes nothing about the output, but its speedup is bounded and it offers no way to chase a cost target the draft cannot reach on its own. Cascades are cheap, because they offload most work to a small model, but they are risky, because a badly set threshold silently degrades quality, and they are slow on the hard queries they are supposed to help with, because escalation happens after the cheap model has already run. Neither is dominant, and a team that needs both a latency guarantee and a cost ceiling finds itself owning two separate inference paths with two separate tuning stories. The unification we develop collapses that into one path with one dial, which is the practical reason to care about the theory beyond its elegance.

These two techniques are duals. Speculative decoding interleaves a draft and a target and is lossless but cannot spend quality; cascades interleave a cheap and an expensive model and can spend quality but route at the sequence level and pay a sequential latency penalty on every deferral. Read side by side, they are two settings of the same underlying machine: a draft proposes, a verifier decides, and control escalates to the target when the draft is not trusted. The question this paper answers is what the best setting of that machine is. Narasimhan and colleagues took the decisive first step, showing that a cascade's deferral rule can be implemented through speculative execution and characterizing the optimal token-level rule for the resulting speculative cascade ^[4], with a public account in the Google Research blog ^[5]. We build a self-contained frontier analysis on top of that idea: a latency-cost- quality model grounded in the roofline, the coupling view of verification, a closed-form optimal block length, a Lagrangian optimal deferral threshold, and an honest account of where the whole construction stops working.

The gap we address is that the two paradigms are still taught, tuned, and deployed as if they were rivals. A practitioner picks speculative decoding or a cascade; the surveys treat them in separate chapters^[17]; and there is no compact statement of the joint frontier that tells an infrastructure team which knob to turn and how far. We give that statement.

The stakes are practical. Interactive inference is where users feel latency directly, in the pause before a chatbot responds or the stutter in a streaming completion, and it is also where the memory-bound regime that makes draft-verify work actually holds. As applications move from single completions to long reasoning traces and multi-step agent loops, the number of sequential decode steps per user action multiplies, and so does the value of shaving each step. At the same time, cost pressure pushes teams toward cascades that offload easy work to small models. A team that must satisfy both a latency target and a cost target currently reaches for two different techniques and has no principled way to combine them; the practical contribution of this paper is a single dial, with a derived optimal setting, that satisfies both at once when the regime permits, and an honest statement of when it does not. The cost-governance dimension connects to our work on agentic FinOps, and the latency dimension to the reasoning-cost analysis in the cost of reasoning.

The rest of the paper is organized as follows. Section 2 reviews the roofline reason decoding is slow, the speculative-decoding machinery and its speedup law, and cascade deferral theory. Section 3 builds the unified model and the coupling view. Section 4 states and proves the three main results. Section 5 reads the published record through the frontier. Section 6 discusses why the interior of the dial wins and situates the coupling among other efficiency methods. Section 7 is the honest limitations section, Section 8 the practitioner's decision rule, and Section 9 the open problems. A glossary and full references close the paper. The taxonomy in Figure 1 fixes vocabulary for everything that follows.

graph TD
    A["Autoregressive decode<br/>one target pass per token<br/>memory-bandwidth bound"] --> B["Speculative decoding<br/>draft block + parallel verify<br/>LOSSLESS, latency-oriented"]
    A --> C["Model cascade<br/>cheap model + sequence-level defer<br/>LOSSY, cost-oriented"]
    B --> D["Speculative cascade<br/>tunable token-level deferral<br/>inside the parallel verify"]
    C --> D
    D --> E["lambda = 0<br/>recovers lossless SD"]
    D --> F["lambda = 1<br/>recovers token cascade"]
    D --> G["0 < lambda* < 1<br/>latency-optimal interior"]

2 Background and Related Work

2.1 Why decoding is slow: the roofline

The roofline model of Williams, Waterman, and Patterson^[12] plots achievable throughput against arithmetic intensity, the ratio of floating-point operations performed to bytes moved from memory. A kernel is memory bound when its intensity falls below the ridge point, where the memory-bandwidth slope meets the peak-compute ceiling, and compute bound above it. For a decode step, the bytes moved are dominated by reading the model weights, which for a model with $P$ parameters at $b$ bytes each is $bP$ bytes, essentially independent of batch size at small batch. The arithmetic is roughly $2P$ operations per token per sequence in the batch. So the intensity is

where $B$ is the batch size. At $B=1$ and $b=2$ bytes (sixteen-bit weights), the intensity is about one operation per byte, and after accounting for activation and key-value traffic the effective figure reported in the roofline survey of Yuan and colleagues is closer to half a FLOP per byte ^[11]. Every mainstream accelerator has a ridge point at tens of FLOPs per byte, so batch-one decode sits an order of magnitude or two into the memory-bound region (Figure 2). The time to produce a token is therefore floored by $bP/\text{BW}$, the weight bytes divided by the memory bandwidth, and adding compute does not help. Pope and colleagues make the same decomposition in their analysis of transformer inference scaling^[24], and it is the premise of paged-attention serving systems that batch aggressively to climb toward the ridge^[23].

The consequence that matters for this paper is the flat left arm of the roofline. Because the compute units are idle while weights stream in, doing more arithmetic on data that is already resident is nearly free. A single target forward pass that scores $g+1$ candidate tokens costs almost the same wall-clock time as one that produces a single token, because both load the weights exactly once. That is the entire economic engine of draft-verify decoding, and it is why the same engine can power a cascade.

It is worth making the floor concrete, because the numbers explain why the gains are large rather than marginal. Consider a target model with fifty billion parameters stored in sixteen-bit precision, so $bP$ is about one hundred gigabytes of weights that must cross the memory bus for each decode step. On an accelerator delivering a few terabytes per second of memory bandwidth, that single weight sweep takes tens of milliseconds, and no amount of added arithmetic reduces it, because the arithmetic was never the bottleneck^[11][24]. The time to first token, by contrast, is set by the prefill phase, which processes the whole prompt in parallel and is compute bound, sitting on the flat top of the roofline rather than its diagonal arm. The two phases live on opposite sides of the ridge point, which is why serving systems report and optimize them separately and why the decode phase is the one that draft-verify schemes attack^[23]. The gap between the memory-bound decode floor and the idle compute ceiling is the free lunch: a block of $g$ draft tokens verified in one pass consumes a sliver of that idle compute and returns up to $g$ tokens for the price of one weight sweep.

This also tells us where the free lunch ends, a point we return to formally in Section 7. The moment batching lifts the workload over the ridge, by processing enough sequences that each weight sweep is amortized across many tokens, decode becomes compute bound and the idle compute that verification was consuming for free is gone. The roofline is therefore not just the reason speculative cascades work; it is also the precise statement of their domain of validity. Everything in this paper is a claim about the memory-bound, small-batch, latency-sensitive corner of the serving space, which is exactly the corner interactive applications and agent loops occupy.

2.2 Speculative decoding and its speedup law

Blockwise parallel decoding ^[8] introduced the draft-then-verify pattern: predict several future tokens in parallel, then keep the longest prefix a scoring model validates. That early work established the shape of the idea, generate speculatively and validate cheaply, but it did not guarantee that the validated output matched what the model would have produced on its own, so it traded a little quality for speed the way a cascade does. The decisive step, taken independently by two groups in early 2023, was to make the validation a sampling test rather than a match test, so that the speculative process became a new way to draw exact samples from the target rather than an approximation of it. Speculative decoding^[1][2] made the verification exact. Let $p$ be the target distribution over the next token given the context and $q$ the draft distribution. The draft samples a token $x\sim q$; the target accepts it with probability $\min (1,p(x)/q(x))$, and on rejection resamples from the normalized residual $(p-q{)}_{+}/\Vert (p-q{)}_{+}{\Vert }_1$. This modified rejection-sampling step has a clean invariant: the accepted token is distributed exactly as $p$. Losslessness is not an empirical observation about these schemes; it is an algebraic identity of the accept rule.

The identity is worth seeing, because everything downstream generalizes it. The probability that the scheme emits a particular token $x$ is the probability it drafts $x$ and accepts it, plus the probability it rejects whatever it drafted and then resamples $x$ from the residual. The first term is $q(x)\min (1,p(x)/q(x))=\min (q(x),p(x))$. The total rejection probability is $1-{\sum }_y\min (q(y),p(y))$, and the residual distribution places mass $(p(x)-q(x){)}_{+}/{\sum }_y(p(y)-q(y){)}_{+}$ on $x$. Since ${\sum }_y(p(y)-q(y){)}_{+}=1-{\sum }_y\min (p(y),q(y))$, the rejection weight and the residual normalizer cancel, and the second term is exactly $(p(x)-q(x){)}_{+}$. Adding the two, $\min (q(x),p(x))+(p(x)-q(x){)}_{+}=p(x)$, so the emitted token is distributed as $p$ for every context, with no approximation. This cancellation is the seed of the coupling view in Section 3.2: the two terms are the agreement mass and the corrective mass of a maximal coupling.

Two structural facts follow that the speedup law then quantifies. First, the bonus token: on a rejection the residual resample still yields one valid target token, and if the whole block of $g$ drafts is accepted the target pass has already computed the distribution for position $g+1$, so a free token is drawn there too. That is why Equation 2 has $g+1$ in the exponent rather than $g$: a block returns between one and $g+1$ tokens. Second, acceptance is where all the gain lives. A draft that never matches the target wastes every draft pass and the scheme degenerates to plain decoding plus overhead; a draft that matches perfectly returns $g+1$ tokens per target pass. Between these, the geometry of Equation 2 is governed entirely by the single number $a$.

Under the simplifying assumption that per-token acceptance events are independent with common probability $a$, Leviathan and colleagues derive the expected number of tokens produced by one draft block of length $g$ (the accepted prefix plus the one bonus token the target contributes on the first rejection or at the end):

If a draft pass costs a fraction $c$ of a target pass and a block requires $g$ draft passes plus one target pass, the wall-clock cost of a block is $(gc+1)$ target-pass units, so the speedup over plain target decoding is the ratio of tokens to time^[26]:

Equation 3 is the workhorse of the whole field, and Figure 3 replots it. Speedup rises steeply with acceptance and saturates: at $a\to 1$ the numerator approaches $g+1$ and the speedup approaches $(g+1)/(gc+1)$, its ceiling. The reported empirical acceptance rates for T5-XXL with a small T5 draft fall between roughly ^[1] 0.53 and 0.82, yielding the reported 2.3 to 3.4 times speedups on translation and summarization^[26]; Chen and colleagues report 2 to 2.5 times on a 70B Chinchilla-class model ^[2][19]. Later work raises acceptance or verifies more candidates per pass: SpecTr casts multi-candidate draft selection as optimal transport and reports an additional 1.37 times over baseline speculative decoding ^[7]; SpecInfer verifies a whole token tree in one pass ^[13]; Medusa attaches extra decoding heads and reports 2.4 times on Spec-Bench^[16][18]; the EAGLE line drafts in feature space and reports 3 to 6.5 times ^[15]; and DistillSpec aligns the draft to the target by distillation for a further 10 to 45 percent ^[14]. Every one of these keeps the lossless accept rule; they compete only on the acceptance rate $a$ and the cost ratio $c$.

It helps to organize this zoo by which term of Equation 3 each method attacks. The numerator, the expected tokens per pass, is raised either by increasing $a$ (better-aligned drafts: Medusa's extra heads^[16], EAGLE's feature-space drafting^[15], DistillSpec's distillation^[14]) or by verifying more candidates per pass so that the effective acceptance climbs (SpecTr's multi-draft optimal transport^[7], SpecInfer's token trees^[13]). The denominator, the cost per pass, is lowered by making the draft cheaper relative to the target, which is what a small distilled draft or a self-speculative early-exit head accomplishes. No lossless method touches the accept rule itself, because touching it would break the cancellation identity above and forfeit the losslessness that is the paradigm's entire selling point. This is precisely the constraint that speculative cascades relax, and seeing the lossless methods all pinned to one slice of the design space is what makes the extra degree of freedom visible.

2.3 Cascades and deferral theory

A cascade routes rather than verifies. Run the small model, compute a deferral signal, and if it clears a threshold, escalate to the large model^[3]. The expected per-query cost of a two-model cascade is

where $c_{\text{s}}$ and $c_{\text{l}}$ are the small- and large-model costs. Lowering $\mathrm{Pr}[\text{defer}]$ cuts cost but risks quality, so the deferral rule is where the design lives. Jitkrittum and colleagues^[6] characterize the Bayes-optimal deferral rule for a classification cascade and show it depends on the confidence of both the small and the large model and their error costs, and that confidence-only deferral, though often strong, can be provably suboptimal. Gupta and colleagues ^[10] carry deferral to generative models, show that sequence-level uncertainty suffers a length bias, and learn a rule on token-level quantiles. Related work tunes cascade confidence^[21] and trains models to be cascade-aware^[20]. The through-line is that a cascade is a lossy scheme by design: it accepts a bounded quality change to buy cost, exactly the freedom speculative decoding lacks. We connect these two research lines in our companion analysis of model routing and cascades, and to the bandit view of adaptive routing in bandit model routing.

The optimal deferral result deserves a closer look, because it foreshadows our Equation 12. For a two-model cascade with error costs, Jitkrittum and colleagues show that the Bayes-optimal decision to defer query $x$ is not a threshold on the small model's confidence alone; it compares the expected cost of answering with the small model against the expected cost of deferring, and the second quantity depends on how likely the large model is to be right, which is the large model's confidence ^[6]. When the large model is a consistent estimator of the true posterior, confidence-based deferral is close to optimal; when it is not, deferring on small-model uncertainty alone can be arbitrarily bad, because you spend the expensive model on queries it will also get wrong. The lesson, which the speculative-cascade analysis inherits, is that the right deferral signal is a comparison between the two models, not a property of one. In a speculative cascade that comparison is handed to you for free: the verifier already computes both $p(x)$ and $q(x)$, so their ratio is available at zero extra cost, which is why the optimal token-level rule is cheap to approximate ^[4].

The length-bias problem that Gupta and colleagues identify is the second foreshadowing ^[10]. If a cascade defers whenever a whole-sequence uncertainty score crosses a threshold, then long outputs, which accumulate more per-token uncertainty simply by being long, are deferred more often than their difficulty warrants, and short outputs escape scrutiny they deserve. The fix is to measure uncertainty at the token level and aggregate with a learned rule, which is exactly the granularity a speculative cascade operates at natively: it decides token by token, so it never conflates length with difficulty. FrugalGPT's own mechanism ^[3] combines three levers, prompt adaptation (shortening or batching prompts), model approximation (caching and small-model substitution), and the cascade itself, and its headline up to 98 percent cost reduction on the HEADLINES financial news task is the cascade lever operating on a workload where a small model suffices for most queries. Those savings are real but they are sequence-level and sequential, which is the inefficiency the coupling removes.

2.4 The two are duals, and the prior unification

Big Little Decoder ^[9] already sits between the paradigms: its fallback policy hands control to the large model when the small model is uncertain, and its rollback policy lets the large model overwrite bad small- model tokens, reporting 1.52 times lossless and 1.76 times at a one-point quality allowance. That quality allowance is a deferral knob in disguise. The decisive unification is due to Narasimhan and colleagues^[4]: they implement a cascade's deferral rule through speculative execution, so the large model verifies the draft in parallel rather than being invoked sequentially, and they characterize the optimal token-level deferral rule and approximate it with a plug-in estimator. On Gemma and T5 across summarization, translation, reasoning, coding, and QA, the resulting speculative cascades report better cost-quality trade-offs than both a sequential cascade and lossless speculative decoding^[5]. Consultant Decoding^[27] and the recent acceptance theory of Timor and colleagues ^[25] pursue related couplings. Our contribution is not the idea of combining the two; it is the compact frontier analysis, the coupling view, the closed-form block length, and the honest map of the boundary, which we develop next and relate to our inference-optimization survey and hybrid-inference economics.

The duality is worth stating as a correspondence, because it is what licenses a single analysis to cover both. In speculative decoding the small model is the draft and the large model is the verifier; in a cascade the small model is the primary responder and the large model is the fallback. Both schemes run the small model first and consult the large model to decide whether to trust it. They differ in three respects, and the speculative cascade takes the better answer to each. On granularity, speculative decoding decides per token while a classic cascade decides per sequence; the coupling keeps the per-token granularity. On timing, speculative decoding runs the large model's check in parallel while a cascade runs it sequentially after the small model finishes; the coupling keeps the parallel timing. On losslessness, speculative decoding is exact while a cascade is lossy; the coupling makes this a tunable choice rather than a fixed property, recovering exactness at one end and the cascade's cost savings at the other. Read this way, the speculative cascade is not a compromise between two techniques but the technique that dominates each on the axis where it was weak, which is exactly what Theorems 1 through 3 make precise.

2.5 The broader efficiency landscape

Draft-verify-escalate is one family in a crowded field, and situating it prevents overclaiming. The orthogonal levers act on the terms of the decode floor $bP/\text{BW}$ rather than on the number of target passes. Quantization lowers $b$, the bytes per parameter, so that each weight sweep moves fewer bytes; pruning and sparsity lower the effective $P$; and better hardware raises the bandwidth $\text{BW}$. None of these changes how many target passes a sequence requires, which is what draft-verify attacks, so they compose with it multiplicatively rather than competing. Distillation is special because it acts on both sides: a distilled model is a cheaper standalone target and a better-aligned draft, and DistillSpec exploits exactly the second role^[14]. Serving-system techniques such as continuous batching and paged attention ^[23] act on a different axis entirely, the batch size, and it is their success at climbing the roofline that defines the boundary of our regime. We restrict attention throughout to the number of target passes and the wall-clock cost of each, and we treat the rest of the stack as fixed context, which is the assumption a small-batch interactive deployment satisfies. Teams weighing these levers against each other will find the full comparison in our inference-optimization survey and the cost side in AI caching strategies.

2.6 A note on terminology and scope

The names in this literature overlap confusingly, so we fix usage for the rest of the paper. We use speculative decoding for any lossless draft-verify scheme whose accept rule preserves the target distribution, regardless of whether the draft is a separate small model, a set of extra heads, or a feature-space predictor. We use cascade for any scheme that routes between models of different sizes and is willing to change the output distribution to save cost, whether it routes per sequence or per token. We use speculative cascadefor the unified scheme that implements a cascade's deferral through a speculative verifier, which includes the exact and lossy operating points as its endpoints. When we say a scheme is lossless we mean its emitted distribution equals the target's exactly for every context, not merely that its quality metric is unchanged on some benchmark; the two are different claims, and conflating them is a common source of confusion in the practitioner literature.

Two scope decisions bound the paper. First, we consider two-model schemes throughout, deferring multi-tier chains to Section 9, because the two-model case already contains the whole conceptual content and the algebra stays readable. Second, we analyze the memory-bound decode regime exclusively, because it is the regime where the free-verification premise holds and where interactive latency is felt; the compute-bound, large-batch regime is a different problem with a different answer, and we say so rather than pretending the analysis covers it. Within these bounds the treatment is self-contained: a reader needs only the roofline fact of Section 2.1, the coupling identity of Section 3.2, and elementary calculus to follow every derivation.

3 A Unified Draft-Verify-Escalate Model

3.1 Setup and cost units

Fix a target model with next-token distribution $p(\cdot \mid h)$ given history $h$, and a draft model with distribution $q(\cdot \mid h)$. A draft pass produces one draft token and costs $c$ in units where a target pass costs one. Crucially, and this is the roofline fact of Section 2.1, a single target pass can score a block of $g+1$ candidate tokens at essentially the cost of scoring one, because it loads the weights once ^[11]. So the wall-clock cost of one draft-verify round with block length $g$ is $gc+1$ target-pass units regardless of how many draft tokens the target ends up examining. We call this the free-verification regime, and we return in Section 7 to what happens when batching breaks it.

Two modeling choices deserve comment. We charge the draft its full per-token cost $c$ for each of the $g$ proposals, which assumes the draft is itself memory-bound and autoregressive; a draft that produces its block with a single parallel head, as in blockwise or multi-head drafting^[8][16], has a smaller effective $c$ and shifts the optima accordingly, but the structure of the analysis is unchanged. We also treat the verify pass as exactly one target-pass unit independent of block width, which is the free-verification idealization; in reality a very wide block or a broad candidate tree adds a small compute term and a key-value memory term that we fold into Section 7's limitations. Within these choices the model is deliberately minimal, two cost parameters and one acceptance function, because the point is to expose the frontier's shape, not to predict a specific system's microseconds.

3.2 Verification is a coupling

What is the largest fraction of draft tokens a lossless rule can accept? A lossless round must emit a token distributed as $p$ while reusing the draft sample $x\sim q$ as often as possible. That is exactly a maximal coupling of $p$ and $q$: a joint distribution whose marginals are $p$ and $q$ and that maximizes the probability the two agree. The classical coupling inequality gives the answer in one line. For any coupling, $\mathrm{Pr}[X=Y]\le {\sum }_x\min (p(x),q(x))$, and the modified rejection-sampling rule attains it, so the maximal per-token acceptance probability is

where $\mathrm{TV}(p,q)=\frac{1}{2}{\sum }_x|p(x)-q(x)|$ is the total variation distance. Equation 5 is the lossless ceiling: no exact verifier can accept more than $1-\mathrm{TV}(p,q)$ of the time, and speculative decoding's accept rule hits it. This is the coupling view of verification, and it is why SpecTr's optimal-transport treatment of multi-draft selection is the right generalization: with $k$ candidate drafts the problem becomes optimal transport with a membership cost, and the greedy selector is $(1-1/e)$-optimal ^[7]. The acceptance rate $a$ in the Leviathan speedup law is not a free parameter; it is $1-\mathrm{TV}(p,q)$ averaged over contexts, which is why aligning the draft to the target, by distillation^[14] or cascade-aware training^[20], is the most direct lever on speed.

The coupling inequality that Equation 5 rests on has a one-line proof worth stating, because the same argument bounds the drift in Section 3.4. For any joint distribution with marginals $p$ and $q$, the event $\{X=Y=x\}$ has probability at most $\min (p(x),q(x))$, since it cannot exceed either marginal at $x$. Summing over $x$ gives $\mathrm{Pr}[X=Y]\le {\sum }_x\min (p(x),q(x))$, and ${\sum }_x\min (p,q)=1-\mathrm{TV}(p,q)$ follows from $\min (p,q)=\frac{1}{2}(p+q-|p-q|)$. The modified rejection-sampling rule attains the bound, so it is a maximal coupling and no lossless verifier can do better. This is the precise sense in which acceptance is not a knob the algorithm designer sets but a property of the model pair: it is one minus the average total variation distance between the two next-token distributions along the trajectory the pair actually produces.

A small numerical instance fixes intuition and is reused below. Suppose the target places probability $(0.5,0.3,0.2)$ on three tokens and the draft places $(0.6,0.3,0.1)$. Then the elementwise minimum is $(0.5,0.3,0.1)$, which sums to $0.9$, so $a_{\max }=0.9$ and $\mathrm{TV}=0.1$. The draft overproduces token one (wanting it with probability $0.6$ against the target's $0.5$) and underproduces token three; the lossless rule rejects exactly the excess $0.1$ mass on token one and repairs it by resampling token three from the residual. A block of five such tokens at $a=0.9$ returns, by Equation 2, about $(1-0.9^6)/0.1\approx 4.69$ tokens per target pass. We carry this instance into Section 3.4 to compute the drift that leniency would add.

3.3 A one-parameter family of deferral rules

Now relax losslessness. The lossless rule rejects $x$ whenever $q(x)>p(x)$, with rejection probability $1-\min (1,p(x)/q(x))$. A deferral rule generalizes this by being more willing to keep the draft token when the two models nearly agree, even if the draft slightly overproduces it. Introduce a leniency knob $\lambda \in [0,1]$ and an acceptance functional

where $r(x)\ge 0$ is a nonnegative slack that vanishes only where the draft never overproduces. At $\lambda =0$ the bracket is the exact rejection-sampling acceptance, so $a(0)=a_{\max }=1-\mathrm{TV}(p,q)$ and the round is lossless. As $\lambda$ rises, more draft tokens clear the bar, so $a(\lambda )$ increases monotonically toward one (Figure 4). At $\lambda =1$ with a suitable slack the rule keeps the draft token whenever the two models'disagreement is below a fixed level, which is a token-level confidence cascade in the sense of Gupta and colleagues ^[10]. The two paradigms are the endpoints of a single dial.

Monotonicity is easy to check and matters for the optimization in Section 4. The integrand in Equation 6, $\min (1,p(x)/q(x)+\lambda r(x))$, is nondecreasing in $\lambda$ pointwise because $r(x)\ge 0$, so $a(\lambda )$ is nondecreasing, and it is strictly increasing on the range where the argument has not yet saturated at one. It is also concave under a mild condition on $r$, because the $\min$ caps the growth as more tokens saturate, which is why the acceptance curves in Figure 4 bend over as $\lambda$ approaches one. The natural choice of slack is $r(x)=(q(x)-p(x){)}_{+}/q(x)$, the fractional amount by which the draft overproduces $x$, which makes $\lambda$ the fraction of the overproduced mass the rule is willing to forgive. With that choice, $\lambda =0$ forgives nothing and is lossless, and $\lambda =1$ forgives all of it, accepting the draft token whenever it is even slightly more likely under the draft than the target, which is a pure agreement cascade. This slack is also the one that makes the drift bound of Equation 7 tight, as we show next.

Why parameterize the family this way rather than some other? The design goal is that the two endpoints be exactly the two techniques we are unifying, and that the single scalar between them have a clear operational meaning. The forgiveness fraction achieves both: at one end it forgives nothing and is provably the lossless rule, at the other it forgives everything and is a token-level agreement cascade, and in between it is the fraction of the draft's overproduced probability mass the verifier is willing to let stand. A practitioner can therefore reason about the knob directly, it is the share of the model disagreement you are choosing to ignore, without needing to understand the underlying rejection-sampling test. Other parameterizations are possible, for instance a temperature on the accept probability or a learned per-token gate, and some may fit a given task better, but they lose either the clean endpoints or the single interpretable scalar. We choose the one that keeps both, because the value of a unifying framework is largely in its legibility, and a knob a team cannot interpret is a knob it will not tune.

3.4 The quality cost of leniency

Leniency is not free. Every draft token accepted that the lossless rule would have rejected is a token drawn from something closer to $q$ than to $p$, so the emitted distribution ${\tilde{p}}_{\lambda }$ drifts from $p$ toward $q$. The extra mass the rule wrongly keeps is exactly the slack it added, so a direct accounting bounds the per-token drift by the leniency times the region where the draft overproduces:

Equation 7 is the honest heart of the scheme: the drift is linear in $\lambda$ and capped by the same total variation distance that set the acceptance ceiling. A close draft (small $\mathrm{TV}$) both accepts more and drifts less per unit leniency, so draft quality is doubly rewarded. Figure 5 plots the bound for three draft-target distances. Over a sequence of $n$ emitted tokens the drifts accumulate sublinearly under standard mixing assumptions, and the deployed quality gap that a task metric measures is upper bounded by this distributional drift, which is why the speculative-cascade experiments can report better quality at fixed cost than a sequence cascade that discards a whole good response over one hard token^[4].

Equation 7 deserves its derivation, since it is the honesty of the whole scheme. The emitted distribution under leniency $\lambda$ keeps an extra slice of the draft's overproduced mass that the lossless rule would have rejected. On the overproduction set $\{x:q(x)>p(x)\}$ the lossless rule already forgives the agreement mass $p(x)$ and rejects the excess $q(x)-p(x)$; leniency forgives a fraction $\lambda$ of that excess and leaves it in the output rather than repairing it. The change in emitted mass at $x$ is therefore at most $\lambda (q(x)-p(x){)}_{+}$, and the total variation distance between the emitted and target distributions, being half the summed absolute change, is bounded by $\lambda {\sum }_{q>p}(q-p)=\lambda \mathrm{TV}(p,q)$, which is Equation 7. The bound is tight for the slack of Section 3.3, since that slack forgives exactly the fractional excess. Returning to the three-token instance of Section 3.2, where $\mathrm{TV}=0.1$, a leniency of $\lambda =0.3$ caps the per-token drift at $0.03$ in total variation: the draft's overproduction of token one is forgiven up to $0.3\times 0.1=0.03$, nudging the emitted distribution from $(0.5,0.3,0.2)$ toward $(0.53,0.3,0.17)$. In exchange, acceptance rises from $0.90$ toward the neighborhood of $0.93$, which by Equation 2 lengthens the expected block. That is the trade in miniature: three points of distributional drift for a few percent more acceptance, and Section 4 decides whether it is worth taking.

A word on how these per-token drifts accumulate over a full generation, because a reader might worry that a small per-token drift compounds into a large whole-sequence divergence. Under the standard assumption that the generation process mixes, that the influence of an early token's small perturbation decays as generation proceeds rather than amplifying, the sequence-level total variation grows sublinearly in the number of tokens rather than multiplicatively, so a per-token drift of a couple of points does not explode into a wholly different output. This is the same reason a slightly miscalibrated sampler does not produce nonsense over long outputs: local perturbations are absorbed. The assumption can fail for pathological models whose generations are chaotic in their early tokens, and for those the safe course is a smaller budget, but for the well-behaved models production systems deploy the sequence-level drift stays close to the per-token bound, which is why measuring the task metric at a handful of leniency settings (Section 8) is a sufficient safeguard rather than a per-token audit.

The pipeline that realizes this family is shown in Figure 6. It is one control loop: draft a block, verify in a single target pass, apply the $\lambda$-deferral rule per token, emit the accepted prefix, and re-anchor at the first escalation.

graph LR
    H["history h"] --> Q["draft model q<br/>propose block x1..xg<br/>cost g*c"]
    Q --> V["target model p<br/>score whole block<br/>ONE pass, cost 1"]
    V --> R{"deferral rule<br/>leniency lambda"}
    R -->|"accept xi"| K["keep draft token"]
    R -->|"escalate"| T["emit target token<br/>end block"]
    K --> O["append, continue block"]
    T --> A["re-anchor history"]
    O --> A
    A --> H

4 Latency-Optimal Coupling

We now derive the three results. Throughout, latency is measured in target-pass units and the acceptance rate is a function $a(\lambda )$ of the leniency through Equation 6.

4.1 The latency model

A round costs $gc+1$ target-pass units and yields $\mathbb{E}[\#\text{tokens}]=(1-a^{g+1})/(1-a)$ tokens (Equation 2, with $a=a(\lambda )$). The expected wall-clock time per emitted token is the reciprocal of the speedup:

Figure 7 plots $L$ against $\lambda$ at a fixed block length. The $\lambda =0$ endpoint is lossless speculative decoding; raising $\lambda$ lowers latency monotonically because $a(\lambda )$ rises, and the $\lambda =1$ endpoint is a token cascade. Latency alone is therefore minimized at maximum leniency; the interior optimum appears only when we charge for quality, which is Section 4.4.

It is worth being explicit about what $L$ does and does not include. It is the steady-state decode latency per emitted token, the quantity a user experiences as tokens-per-second once generation is underway; it excludes the prefill time to first token, which draft-verify does not touch, and it excludes queueing and scheduling effects, which belong to the serving layer. Because it is a per-token quantity, multiplying it by the response length gives the total decode time, so a technique that lowers $L$ by ten percent lowers total decode latency by ten percent regardless of response length. This linearity is why the modest per-token gains in the case study of Section 4.6 compound into meaningful wall-clock savings on the long outputs that reasoning and agent workloads produce, and it is why we optimize $L$ directly rather than some length-weighted aggregate.

4.2 Theorem 1: frontier dominance

Proof. A sequence cascade that defers a fraction $\delta$ of queries emits, per token, a mixture: with probability $1-\delta$ a draft token at latency $c$, and with probability $\delta$ it pays the draft pass and then the full target pass sequentially, latency $c+1$, because the escalation happens after the draft answer is scored. Its expected per-token latency is thus $L_{\text{casc}}=c+\delta$ and its cost is likewise $c+\delta$ in compute units. Build a speculative cascade that reproduces the same token-level accept decisions: keep the draft token exactly when the sequence cascade would not have escalated on that token. This matches the emitted distribution, hence the quality $Q$. But now the target pass that decides all of the block's escalations is a single parallel pass costing one unit shared across the whole block of $g$ tokens, not one sequential pass per deferred token. The per-token latency becomes $L_{\text{spec}}=(gc+1)/\mathbb{E}[\#\text{tokens}]$, which for any $g\ge 1$ and $a(\lambda )\in (0,1)$ is at most $c+\delta$ because the shared target pass is amortized over multiple accepted tokens rather than charged once per deferral. Cost obeys the same inequality since the target pass is loaded once. Equality holds only in the degenerate case $g=1$ with $a\to 0$, where the block never extends and there is nothing to amortize. $\square$

The proof is nothing more than the roofline observation cashed out as algebra: the sequence cascade pays for the target model's decision serially, once per hard token, while the speculative cascade pays for the identical decision once per block in the compute headroom that decoding leaves idle. Figure 10 in Section 5 draws the two frontiers this theorem separates.

The size of the gap is itself informative. Write the sequence cascade's per-token latency as $c+\delta$ and the speculative cascade's as $(gc+1)/N$ where $N=\mathbb{E}[\#\text{tokens}]$ is the expected block yield. Their ratio is $(gc+1)/(N(c+\delta ))$. At a representative operating point, $g=5$, $c=0.05$, $N=4$, and a cascade deferral rate $\delta =0.3$, the sequence cascade spends $0.35$ target-pass units per token while the speculative cascade spends $1.25/4=0.31$ units, and as the deferral rate rises the gap widens because the sequence cascade pays a full extra pass per deferral while the speculative cascade's cost is capped by the single shared verify. In the limit of a hard workload where $\delta \to 1$, the sequence cascade degenerates to running both models in full on every query, latency $1+c$, while the speculative cascade still amortizes its one verify pass across whatever prefix the draft gets right, so the advantage is largest exactly where cascades are most tempted, on hard traffic. This is the quantitative content behind the qualitative dominance, and it is why the reported speculative-cascade cost-quality curves separate most from the sequence cascade in the high-quality region where deferral is frequent^[4].

4.3 Theorem 2: the optimal block length

Fix the leniency and ask for the block length that minimizes latency, or equivalently maximizes the speedup $S(a,g,c)$ of Equation 3. Treating $g$ as continuous and differentiating $\log S=\log (1-a^{g+1})-\log (1-a)-\log (gc+1)$ with respect to $g$ gives the first-order condition

The left side is the marginal token gained by extending the block, which decays geometrically because a longer block is only useful if all earlier tokens were accepted; the right side is the marginal draft cost of one more proposal. Setting them equal defines the latency-optimal block length $g^{\star }(a,c)$. Equation 9 has no elementary closed form, but its solution is well approximated for small $c$ by

which makes the two comparative statics explicit: $g^{\star }$ rises as acceptance $a\to 1$ (the $1/(-\ln a)$ prefactor diverges, so high-acceptance drafts should propose long blocks) and falls as the draft grows relatively expensive (larger $c$ shrinks the logarithm). Figure 8 plots the exact $g^{\star }$ from Equation 9 against acceptance for three cost ratios, and Figure 9 shows the speedup curves whose peaks it locates. For the reported operating regime, acceptance 0.5 to 0.8^[1] and small cost ratios, the optimum is a handful of tokens, matching the block sizes used in practice^[17]. Tree verification^[13] effectively raises the per-pass token yield and so shifts $g^{\star }$ outward.

The approximation in Equation 10 comes from the small-cost limit of Equation 9. When $c$ is small the optimal block is long enough that $a^{g+1}$ is small, so $1-a^{g+1}\approx 1$and the left side of Equation 9 is approximately $-a^{g+1}\ln a$. The right side is approximately $c$ for a moderate block. Setting $-a^{g+1}\ln a\approx c$ and solving for $g$ gives $(g+1)\ln a\approx \ln (c/(-\ln a))$, hence $g+1\approx \ln ((-\ln a)/c)/(-\ln a)$, which is Equation 10. The two comparative statics are then transparent: the prefactor $1/(-\ln a)$ blows up as $a\to 1$, so a near-perfect draft should propose very long blocks, while a larger $c$ shrinks the logarithm, pulling the optimum in.

A worked instance: take a draft with acceptance $a=0.7$ and a cost ratio $c=0.05$, representative of a small distilled draft against a mid-sized target. Then $-\ln a=0.357$, and Equation 10 gives $g^{\star }\approx (1/0.357)\ln (0.357/0.05)-1=2.80\times 1.966-1\approx 4.5$, so a block of four or five tokens. Substituting back into Equation 3, the speedup at $g=5$ is $(1-0.7^6)/((0.3)(1.25))=(1-0.118)/0.375\approx 2.35$times, close to the peak of the $a=0.7$ curve in Figure 9 and consistent with the reported 2 to 3 times regime^[1][2]. Raising acceptance to $a=0.9$ moves the optimum to roughly nine tokens and the speedup past four times, which is the regime the feature-space drafters reach^[15]. The formula thus reproduces, from first principles, both the small block sizes and the speedup magnitudes the literature reports.

4.4 Theorem 3: the optimal deferral threshold

Now the main event. Choose the leniency to minimize expected latency subject to a budget on quality drift. Using Equation 8 for latency and Equation 7 for drift, and writing the budget as a maximum tolerated drift $\epsilon$, the program is

Form the Lagrangian $\mathcal{L}=L(\lambda )+\mu (\mathrm{TV}({\tilde{p}}_{\lambda },p)-\epsilon )$ with multiplier $\mu \ge 0$. Because both $L$ and the drift are smooth and monotone in $\lambda$ (latency decreasing, drift increasing), the Karush-Kuhn-Tucker stationarity condition $dL/d\lambda +\mu d\mathrm{TV}/d\lambda =0$ pins a unique interior optimum whenever the budget binds. Pushing the derivative through to the per-token decision, a draft token with context-conditional target and draft probabilities $p(x),q(x)$ should be accepted exactly when the marginal latency it saves exceeds the marginal drift it costs at the shadow price ${\mu }^{\star }$, which reduces to a single threshold on the draft-target disagreement:

Equation 12 is the optimal coupling. It says: accept the draft token unless the target underweights it by more than a single global factor $1/{\mu }^{\star }$, and set that factor by the quality budget. Three things follow immediately. First, at $\epsilon =0$ the budget forces ${\mu }^{\star }\to \infty$, the threshold becomes $p(x)/q(x)\ge 1$, and Equation 12 is exactly the lossless rejection-sampling rule: speculative decoding is the zero-budget special case of the optimal coupling. Second, the optimal rule is a per-token threshold on the same likelihood ratio the verifier already computes, so it costs nothing extra to evaluate, which is why the plug-in approximation of Narasimhan and colleagues is cheap and effective^[4]. Third, the shadow price ${\mu }^{\star }$ is a single scalar tuned to hit the quality target, exactly the one-dimensional dial a practitioner wants. The optimal rule depends on both models' probabilities, echoing the classification result of Jitkrittum and colleagues that the Bayes-optimal cascade deferral needs both confidences ^[6]; here that dependence falls out of the likelihood ratio automatically.

The KKT derivation is worth spelling out because it explains why the optimum is a single threshold rather than a per-token schedule. Stationarity requires $dL/d\lambda =-\mu d\mathrm{TV}/d\lambda$: at the optimum, one more unit of leniency lowers latency at exactly ${\mu }^{\star }$ times the rate it raises drift. Because both derivatives are integrals of per-token contributions over the same overproduction set, the equality can hold token by token only if each accepted token clears a common bar on its latency-per-drift ratio, and that ratio reduces to the likelihood ratio $p(x)/q(x)$ because a token's marginal contribution to acceptance and to drift are both proportional to its overproduction $(q(x)-p(x){)}_{+}$. The common bar is Equation 12. A per-token schedule would waste budget: it would spend drift on tokens whose latency payoff is below the going rate while refusing tokens whose payoff is above it, and reallocating toward the threshold strictly improves both objectives until the single bar is reached. This is the same exchange-rate logic that makes water-filling optimal in rate allocation, specialized to the drift-versus-latency trade.

A worked instance closes the loop. Keep the three-token distributions of Section 3.2, target $(0.5,0.3,0.2)$ and draft $(0.6,0.3,0.1)$, and suppose the task tolerates a per-token total variation drift of $\epsilon =0.03$. The only overproduced token is token one with excess $0.1$, so Equation 7 binds at ${\lambda }^{\star }\mathrm{TV}={\lambda }^{\star }\times 0.1=0.03$, giving ${\lambda }^{\star }=0.3$ and, through Equation 12, a threshold that accepts token one whenever the target underweights it by no more than the budgeted factor. Acceptance rises from $0.90$ to about $0.93$, the expected block at $g=5$ lengthens from $4.69$ to about $4.79$ tokens per pass, and the latency per token falls by roughly two percent, bought with three points of controlled drift. The gain looks small on one token because the pair already agrees on ninety percent of the mass; on a poorly aligned pair, where $\mathrm{TV}$ is large, the same budget buys far more acceptance, which is why the technique helps most exactly where lossless speculative decoding helps least.

4.5 Corollary: the interior dominates

Combining the three results, the endpoints are dominated. At $\lambda =0$ you have the best lossless latency but cannot reach a cost target below the acceptance ceiling; at $\lambda =1$ you have the lowest latency but unbounded drift. For any strictly positive, finite quality budget the optimal ${\lambda }^{\star }$ of Equation 12 sits in the interior and, by Theorem 1, its $(L,C,Q)$ point lies below the sequence- cascade frontier and to the low-latency side of the lossless point. That is the precise sense in which the coupled scheme beats either technique alone, and it is what the reported cost-quality improvements of speculative cascades measure^[4][5].

The corollary should be read with its conditions attached, because they are what keep the claim honest. The interior dominates only for a strictly positive, finite quality budget: with a zero budget the optimum sits at the lossless endpoint by construction, and with an unbounded budget it runs to the pure-cascade endpoint where drift is uncontrolled. It dominates only in the memory-bound regime, where Theorem 1's free-verification premise holds; in the compute-bound regime the construction that proves dominance loses its footing, as Section 7 details. And it dominates only within the leniency family of Equation 6, not against every conceivable scheme. Within those three conditions, however, the statement is exact and constructive: given any target quality below the model's own, the recipe of Equations 9 and 12 names a specific block length and deferral threshold whose operating point no sequence cascade and no lossless decoder can match on all of latency, cost, and quality at once. That constructiveness is what makes the result usable rather than merely reassuring.

4.6 A numerical case study

To see the three results act together, take a concrete pair and walk the whole procedure. Let the target be a mid-sized model and the draft a distilled model one twentieth its size, so the cost ratio is about $c=0.05$, and suppose distillation has aligned the draft well enough to give a lossless acceptance $a_0=0.75$, the high end of Leviathan's reported T5-XXL range ^[1]. Step one, block length: Equation 10 gives $g^{\star }=(1/0.288)\ln (0.288/0.05)-1\approx 3.47\times 1.75-1\approx 5.1$, so a block of five. Step two, lossless operating point: Equation 3 at $a=0.75,g=5,c=0.05$ yields a speedup of $(1-0.75^6)/((0.25)(1.25))=(1-0.178)/0.3125\approx 2.63$times, a lossless baseline squarely in the reported range. Step three, spend a quality budget: allow a per-token drift of $\epsilon =0.02$. If the draft-target total variation is $0.25$ (consistent with $a_0=0.75$), Equation 7 binds at ${\lambda }^{\star }=0.08$, lifting effective acceptance to roughly $0.79$ through Equation 6. Recomputing Equation 3 at $a=0.79$ raises the speedup to about $(1-0.79^6)/((0.21)(1.25))\approx 2.93$ times, an eleven percent latency improvement over the lossless point purchased with two points of controlled drift.

The case study makes the paper's claim tangible. The lossless corner delivers 2.63 times; opening the deferral knob to a small, measured budget delivers 2.93 times at a bounded quality cost; and a sequence cascade targeting the same quality would pay the full target latency on every deferred token rather than amortizing it across the block, landing above both on the latency axis by Theorem 1. All three numbers come from the same three equations, and none of them is a measurement of ours; they are what the cited speedup law and our derived optima imply for a pair whose acceptance and cost ratio sit in the published range ^[1][2][14]. A team can run exactly this arithmetic on their own measured $a_0$, $c$, and $\mathrm{TV}$ to predict their frontier before writing a line of serving code.

4.7 Sensitivity and robustness of the optima

Because the closed forms rest on an idealized model, it matters how sharp the optima are: a flat optimum tolerates estimation error in $a$ and $c$, while a sharp one punishes it. The block-length objective is fortunately flat near its peak. Around $g^{\star }$ the speedup of Equation 3 changes only in second order, because the first-order condition Equation 9 sets the derivative to zero there, so a block chosen one or two tokens away from the true optimum loses a few percent of speedup at most. This is visible in Figure 9, where each curve has a broad plateau rather than a spike, and it is why practical systems get away with a fixed block size across heterogeneous traffic: the penalty for a mismatched block is small. The lesson is to estimate $g^{\star }$ from Equation 10 and not agonize over precision.

The leniency optimum is sharper in one direction. Because latency falls quickly as leniency opens from zero (the acceptance curve of Figure 4 is steepest at small $\lambda$) while drift grows only linearly (Equation 7), the early part of the budget buys a lot of speed cheaply, and the marginal value of leniency declines as the budget is spent. This convex-benefit, linear-cost structure means the optimal operating point is not knife-edge: spending slightly less than the full budget sacrifices little speed, which is a comfortable place to be, because it lets a cautious team stay below its measured quality ceiling with a small margin at almost no latency cost. The dangerous direction is overspending, because once the acceptance curve saturates near one, additional leniency adds drift without adding speed, so the rule is to approach the budget from below and stop at the first sign the task metric moves, exactly the procedure Section 8 prescribes.

The one assumption whose violation genuinely threatens the optima is the independence of acceptance events, which we treat at length in Section 7. Positive autocorrelation, a rejected token making the next more likely to be rejected, makes long blocks less valuable than Equation 2 predicts, pulling the true optimal block shorter than $g^{\star }$; a local empirical sweep around the formula's estimate corrects for this cheaply. None of the comparative statics reverse under autocorrelation: more acceptance still favors longer blocks, and more draft cost still favors shorter ones. The analysis is thus robust in its qualitative claims and approximate in its exact numbers, which is the appropriate standing for a model whose purpose is to guide measurement rather than replace it.

5 The Published Record on the Frontier

The analysis makes falsifiable orderings, and the published numbers respect them. Figure 11 replots the reported wall-clock speedups of the major methods, each attributed to its source; Table 1 (Figure 12) classifies them by paradigm, losslessness, and reported gain; and Figure 10 places the paradigms on the cost-quality frontier that Theorem 1 separates.

5.1 A meta-analytic decomposition

The reported gains decompose exactly along the two levers the theory names, acceptance $a$ and the block/verify structure that turns acceptance into tokens per pass. Leviathan's 2.3 to 3.4 times^[1] and Chen's 2 to 2.5 times^[2] are pure acceptance-times-block gains at $\lambda =0$. SpecTr's additional 1.37 times^[7] comes from raising the effective acceptance through multi-candidate optimal-transport selection, that is, tightening the coupling of Equation 5. SpecInfer's tree verification^[13] raises tokens per pass, shifting $g^{\star }$ (Equation 9) outward. Medusa's 2.4 times^[16] and EAGLE-3's 3 to 6.5 times^[15] raise acceptance by drafting in feature space with model-internal signals. DistillSpec's 10 to 45 percent^[14] raises acceptance by shrinking $\mathrm{TV}(p,q)$ directly. Every lossless method is a point on the $\lambda =0$ slice, differentiated only by where it lands on Figure 3. The cost-oriented methods, FrugalGPT's up to 98 percent^[3] and Big Little Decoder's 1.76 times at a one- point allowance ^[9], are points at $\lambda >0$, and the speculative-cascade results^[4] are the interior optima that Section 4.5 predicts should dominate both.

The decomposition also explains a pattern that looks puzzling until read through the theory: why the biggest reported speedups belong to the methods that invest most in the draft. EAGLE drafts in the target's own feature space using its hidden states, which makes the draft's next-token distribution unusually close to the target's, driving acceptance high and the speedup to the 3 to 6.5 times range ^[15]; Medusa's multiple decoding heads play the same game more cheaply for 2.4 times ^[16]. In the language of Equation 5, both are shrinking $\mathrm{TV}(p,q)$ and so raising $a_{\max }$. This is the same lever DistillSpec pulls by explicit distillation ^[14], and it is why Section 8 names draft investment as the highest-impact engineering: every one of the field's speedup records is, underneath, a story about closing the distributional gap between draft and target. A speculative cascade adds one more lever the lossless methods cannot touch, the willingness to accept a bounded gap rather than close it, and stacks that lever on top of whatever alignment the draft already has.

Big Little Decoder deserves a second look because it is the historical bridge. Its fallback policy defers to the large model on small-model uncertainty, and its rollback policy lets the large model overwrite bad small-model tokens, reporting 1.52 times lossless and 1.76 times when a one-point quality drop is allowed^[9]. That one-point allowance is a deferral budget in all but name, and the jump from 1.52 to 1.76 times as the allowance opens is a small, real instance of the frontier this paper formalizes: a fixed budget of quality bought a measurable slice of latency. What the general analysis adds is the optimal way to spend that budget (Equation 12) and the guarantee that spending it through parallel verification rather than sequential fallback never costs extra latency (Theorem 1), which the rollback mechanism approximates but does not optimize.

5.2 Cross-confirmation of the key numbers

Because the argument rests on cited figures, we cross-checked the load-bearing ones against at least two independent sources. The speculative-decoding speedup law (Equations 2 and 3) appears in the original ICML paper^[1] and is restated in independent follow-on work and the ACL survey ^[17]. The T5-XXL 2.3 to 3.4 times and acceptance 0.53 to 0.82 figures are in the paper and its proceedings record^[26]. The Chinchilla 2 to 2.5 times is in the DeepMind paper abstract and secondary summaries^[2]. FrugalGPT's up to 98 percent cost reduction at matched GPT-4 quality appears in the arXiv version and the TMLR version^[3]. SpecTr's $(1-1/e)$-optimal selection and 2.13 times are in the NeurIPS paper^[7]. The memory-bound decode intensity near 0.5 FLOP per byte is in the roofline survey ^[11] and consistent with the transformer-scaling analysis ^[24] and the original roofline model ^[12]. Where a single number is reported as a range, we plot its midpoint and give the range in the caption.

A note on the standing of these numbers is warranted, because the paper leans on them. None of the speedups, acceptance rates, or cost reductions in this paper are our measurements; every one is drawn from a published source, and where a figure is load-bearing we required a second independent confirmation before using it. We treat a number as confirmed when the primary paper reports it and at least one independent venue, a survey, a proceedings record, or a separate group's restatement, reports the same value or a consistent one. Numbers we could not cross-confirm we either omitted or flagged in context, and we deliberately avoided combining figures from different sources into a single derived claim, because a composed number inherits both sources' conditions and is easy to misread as a measurement. The figures that plot equations are generated from the equations themselves and are labeled as analytical models; the one figure that plots published speedups (Figure 11) names each source in its caption and marks which methods are lossless. This is the standard we hold ourselves to, because an analysis paper that misrepresents the record it synthesizes is worse than useless, and the whole value of the frontier depends on the points that anchor it being real.

5.3 What the frontier predicts, and where the record could refute it

A frontier is only worth drawing if it makes predictions that could fail. Ours makes three, and we state how each could be refuted so a reader can hold the analysis accountable. First, it predicts a strict ordering: at any fixed quality below the target's, a speculative cascade should reach a lower latency and cost than a sequence cascade in the memory-bound regime (Theorem 1). This would be refuted by a measured sequence cascade that matches a speculative cascade's latency at equal quality on small-batch interactive traffic; the speculative- cascade results report the opposite ordering on Gemma and T5^[4], but a counterexample on a different pair would bound the claim. Second, it predicts that lossless methods cannot cross a quality-cost point that a lossy cascade reaches, because they are pinned to the $\lambda =0$ slice; a lossless method reported to match a cascade's cost at strictly lower quality-loss would refute the pinning, and we are aware of none. Third, it predicts that the advantage shrinks as batch size rises toward the ridge point; a speculative cascade that retained its full advantage at large batch would refute the roofline grounding, and this is the prediction we most want tested, because it is the boundary of the technique's usefulness.

The published record also disciplines what we may claim about magnitudes. The lossless speedups cluster in a 2 to 3 times band for classic drafts^[1][2] and climb to 3 to 6.5 times for feature-space drafters that push acceptance high^[15], and every one of those numbers is a point on the same Equation 3 surface at different $a$. A speculative cascade cannot conjure speed beyond what its acceptance and cost ratio permit; it can only reach the parts of the frontier that spending a quality budget opens up, which sit below and to the low-latency side of the lossless point but above the physical floor set by the draft's own cost. A claim of a speculative cascade delivering, say, ten times at high quality on a pair whose lossless acceptance is 0.7 would contradict the equations, and we would treat such a report as evidence of a measurement artifact or an undisclosed change to the model pair rather than a triumph of the method. This is the discipline the frontier imposes: it tells you not only what to expect but what to disbelieve.

Finally, the meta-analysis exposes a gap in the record that future work should close. Almost all reported numbers are either pure-latency (the speculative- decoding line) or pure-cost (the cascade line), and relatively few report the full three-way latency-cost-quality operating point the frontier is defined over. The speculative-cascade papers are the exception, reporting cost-quality curves^[4][5], but even they hold batch size fixed and so leave the roofline axis of Figure 2 unmeasured. A clean test of this paper would sweep batch size and quality budget jointly and trace where the coupling advantage predicted by Theorem 1 decays to zero. We flag this as the single most valuable experiment the theory invites, and we do not run it here, because this is an analysis, not an empirical study.

6 Discussion

The result is easy to misread as a claim that speculative cascades are simply better than the alternatives. They are not; they are better on a specific frontier under specific conditions, and the value of the analysis is in naming those conditions precisely. This section draws out the intuition and the boundaries the equations imply.

6.1 Why the interior wins

The deepest reason the coupled scheme beats both endpoints is that speculative decoding and cascades each waste a different resource, and the coupling wastes neither. Lossless speculative decoding wastes quality budget: it insists on distributional exactness even when the task would happily absorb a two-point drift, so it leaves acceptance, and therefore speed, on the table. A sequence cascade wastes bandwidth: it invokes the target sequentially on every hard query, paying a full weight sweep per deferral instead of amortizing one sweep across a block, so it leaves the roofline's idle compute unused. The interior operating point of Equation 12 spends exactly the quality budget the task allows and amortizes exactly the bandwidth the roofline offers, and there is no third resource left to waste. That is why the dominance in Theorem 1 is not a marginal improvement but a structural one: it removes two independent inefficiencies at once.

A second intuition is that the coupling turns a global routing decision into a local one without losing the routing. A sequence cascade must decide, before it sees how a response unfolds, whether the whole response is hard, and it pays for being wrong in either direction, deferring easy responses or committing to hard ones. The speculative cascade defers the decision to the token where the evidence actually arrives, the moment the draft and target disagree, and it makes that decision with the target's own probability in hand rather than a proxy confidence. Localizing the decision is what eliminates the length bias that Gupta and colleagues had to correct for at the sequence level^[10], and grounding it in both models' probabilities is what realizes the two-sided optimal rule of Jitkrittum and colleagues^[6] without a separate estimator.

6.2 The losslessness tax

The paper quantifies something practitioners feel but rarely price: the cost of insisting on losslessness. In the case study of Section 4.6 that cost was the gap between 2.63 times and 2.93 times, roughly a tenth of the latency, for a two-point drift the task could absorb. On a worse-aligned pair the tax is larger, because a high total variation distance both caps lossless acceptance low and gives leniency more room to raise it. This reframes the losslessness guarantee as a purchased property rather than a free default: you are paying for it in latency and cost, and the price is exactly the acceptance you forgo by refusing to spend quality. For many production systems that price is worth paying, because reproducibility and evaluation stability are real engineering goods. The point is only that it is a choice with a computable price, not a costless baseline, and the frontier lets you see the price before you commit.

6.3 Where this sits among efficiency methods

Speculative cascades are one lever among several, and they compose with most of the others rather than competing. Quantization shrinks $b$, the bytes per parameter, which lowers the decode floor for both draft and target and changes the cost ratio but not the structure of the frontier. Distillation produces better drafts, which raises acceptance and, as Section 4 showed, widens the whole usable frontier ^[14]. Paged and batched serving climbs the roofline toward the ridge^[23], which is the one composition that fights the coupling, because it erodes the free-verification premise; the two should be seen as governing different regions of the batch axis rather than as substitutes. Tree and multi-draft verification ^[13][7]raise the tokens verified per pass and shift $g^{\star }$ outward, composing cleanly with the deferral knob. The honest summary is that the coupling is orthogonal to most of the efficiency stack and antagonistic to exactly one part of it, large-batch throughput serving, which is precisely the regime Section 7 excludes. Readers building an end-to-end inference stack will find the broader menu in our inference-optimization survey and the economics in hybrid-inference economics.

6.4 Implications for reasoning and agents

The coupling interacts with two trends the field is moving toward. Long chain-of-thought reasoning multiplies the number of decode steps per user turn, which multiplies the value of any per-token latency reduction, so the interior operating point is worth more on a reasoning workload than on a short-answer one; the caveat is that reasoning traces mix easy connective tokens with hard inferential leaps, so the acceptance rate is highly non-uniform and a fixed leniency is a compromise a position-aware schedule could beat. Agent loops raise a sharper issue: they interleave free-form generation, where high leniency is safe, with structured tool arguments, where a single wrong token breaks a call and no drift is acceptable. The natural design drops to $\lambda =0$ on structured spans and opens the knob on prose, which is the decoding analogue of constrained generation, treated in our constrained-decoding semantics paper. The broader agent-serving context, where latency compounds across many model calls in a loop, is exactly where interactive inference lives, and it is developed in our agentic systems work.

6.5 The deferral rule as a first-class serving parameter

A practical consequence of the analysis is that the deferral leniency should be treated as a serving parameter on the same footing as batch size, temperature, or maximum tokens, not as a fixed property baked into the decoding algorithm. Speculative decoding hides its accept rule as an implementation detail because it has no free parameter to expose; the rule is fixed by the losslessness requirement. Once losslessness becomes a choice, the shadow price of Equation 12 is a knob a serving system can and should surface, defaulting to zero for callers that need exactness and opening for callers that have declared a quality budget. This reframes a body of work that currently lives inside decoding kernels as a policy the serving layer owns, and it suggests an interface: a caller specifies a tolerated quality drift alongside its prompt, and the system picks the leniency and block length that hit that drift at minimum latency. The measurement discipline for setting and monitoring such a parameter is exactly the eval-and-observe loop we describe in LLMOps and observability, and treating it as a governed parameter rather than a hidden constant is what lets a team reason about it at all.

Seen this way, the paper is as much about interface design as about mathematics. The three results, dominance, optimal block, optimal threshold, jointly say that a serving system with two exposed knobs (block length and quality budget) can offer every operating point on the frontier and always sit at its optimum, from strictly lossless at one extreme to aggressive cost savings at the other, with a single scalar in between selecting exactly how much quality to trade. That is a considerably simpler contract than asking users to choose between two entirely different inference stacks, and it is the practical payoff of showing the two stacks were one stack all along.

6.6 A unifying picture

Stepping back, the result is an instance of a recurring pattern in systems research: two techniques that grew up solving adjacent problems turn out to be special cases of a single parameterized mechanism, and the parameter that connects them is exactly the resource one technique refused to spend and the other spent freely. Here that resource is distributional fidelity. Speculative decoding spends none of it and buys latency up to a ceiling; sequence cascades spend a lot of it, coarsely, and buy cost; the speculative cascade spends precisely the right amount, per token, and buys both. The mechanism that makes the spending precise is the verifier that both techniques already contain, one using it to guarantee exactness and the other to decide routing, revealed to be the same object viewed through different accept rules. Once the accept rule is seen as a free parameter rather than a fixed law, the whole design space between the two techniques opens, and the optimization in Section 4 simply finds the best point in it.

This is a useful lens for the rest of the inference stack too. Several other pairings in efficiency research, exact versus approximate attention, full versus quantized weights, dense versus sparse activation, have the same shape: a fidelity knob that most deployments leave pinned at the exact end out of caution. The discipline this paper models, name the fidelity resource, bound its cost, and set it by a measured budget rather than by default, applies to those pairings as readily as to draft-verify decoding. We do not develop them here, but we note that the frontier method generalizes beyond its subject, and that the habit of treating an exactness guarantee as a purchased property with a computable price is worth carrying into any efficiency decision.

6 Threats to Validity and Limitations

This is an analysis paper, and its results are only as good as their assumptions. We state them plainly.

The i.i.d. acceptance assumption is false in detail. Equations 2, 3, 8, and 9 inherit Leviathan's assumption that per-token acceptance events are independent with a common rate $a$^[1]. Real acceptance is autocorrelated: a rejected token often signals a hard region where the next tokens are also likely to be rejected, and acceptance varies systematically with position and content^[25]. The closed forms for $g^{\star }$ are therefore first approximations to be refined by a local empirical sweep, not exact operating points. The qualitative comparative statics (more acceptance means longer optimal blocks; more draft cost means shorter) are robust to the assumption; the exact numbers are not. The direction of the error is predictable: positive autocorrelation makes the tail of a long block less likely to survive than the geometric model assumes, so the true optimal block is a little shorter and the true speedup a little lower than Equations 3 and 9 predict. A team should read the closed forms as an initial estimate and confirm with a short empirical sweep, treating the formula as a way to start the search near the answer rather than as the answer itself.

The free-verification premise fails as batch size grows. The entire advantage in Theorem 1 comes from the target verify pass being nearly free, which holds only in the memory-bound region of the roofline^[12][11]. As batch size rises past the ridge point, or with very long shared prefixes, decode becomes compute bound, weight loads are amortized across many sequences, and the verify pass starts competing for saturated compute. In that regime a sequence cascade that simply avoids invoking the large model saves real compute, and the coupling advantage shrinks toward zero. Speculative cascades are a small-batch, latency- sensitive, interactive-serving technique; we make no claim about the high- throughput batch regime that systems like vLLM target^[23].

The scheme is lossy for any positive budget. Equation 7 bounds the drift, but the bound is positive whenever $\lambda >0$. Unlike plain speculative decoding, a speculative cascade does not preserve the target distribution, so it must not be used where distributional equivalence is required, for reproducible evaluation, for regression testing against a reference model, or where a policy mandates that the deployed model's outputs match a certified reference. In those settings the only admissible operating point is $\lambda =0$, which is ordinary speculative decoding. This is not a minor footnote but a category boundary: it places speculative cascades firmly in the same family as classical cascades and other quality-trading techniques, and separates them from the losslessness guarantee that is speculative decoding's distinctive promise. A reader who wants the big model's exact distribution at reduced latency should reach for plain speculative decoding and treat this paper as an account of what they give up by refusing to spend quality, namely the extra acceptance the drift budget would have bought.

The quality budget needs a task metric. Equation 12 sets the shadow price so that the total variation drift hits a ceiling $\epsilon$, but total variation is a distributional surrogate for what a user cares about, which is a downstream task metric. The link from drift to task quality is monotone under mild assumptions but not calibrated, so $\epsilon$ must be set by measuring the task metric across a small sweep of ${\mu }^{\star }$, not read off the bound. This is the same caution Gupta and colleagues raise about sequence-level uncertainty proxies ^[10].

The cost model ignores memory and tree overheads. We charge a block $gc+1$ units and nothing else. Real systems pay for key-value cache growth, for managing and pruning candidate trees^[13], and for the draft model's own memory footprint, which competes with the target for bandwidth. These are second-order for a small draft and a handful-length block but grow with tree width, and a full systems model would include them. The most important omission is the key-value cache: a rejected block still consumed cache slots for its speculative tokens, and a system that speculates aggressively must budget memory for tokens it may discard, which interacts with the paged-memory management that high-throughput serving relies on ^[23]. Our cost model is deliberately a latency model, not a memory model, and a deployment near its memory ceiling should treat the block length as constrained by cache pressure as well as by Equation 9.

Optimality is within the stated family. Theorem 3 finds the best member of the leniency family of Equation 6 under the drift bound of Equation 7. A different deferral parameterization, a learned rule on richer features ^[10], a confidence-tuned rule^[21], or a verifier-free scheme could do better on a given task; our claim is the structure of the optimum inside this family and its endpoints, not global optimality over all possible schemes.

7 Practical Implications

The analysis reduces to a short decision procedure for an infrastructure team, drawn in Figure 13.

graph TD
    A["Need to accelerate decoding"] --> B{"Losslessness required?<br/>(reproducibility, policy)"}
    B -->|yes| C["Speculative decoding, lambda = 0<br/>tune draft alignment and g*"]
    B -->|no| D{"Small-batch / latency-bound<br/>serving regime?"}
    D -->|no, large batch| E["Sequence cascade or batch;<br/>verify is not free here"]
    D -->|yes| F["Speculative cascade<br/>set g near g* (Eq 9)"]
    F --> G["Measure task metric vs mu*;<br/>pick smallest quality budget<br/>that hits the cost target"]
    C --> H["Distill the draft to raise a<br/>(shrinks TV, widens budget)"]
    F --> H

Step one: measure two numbers. The acceptance rate $a$ of your draft against your target on representative traffic, and the cost ratio $c$. Everything else keys off these. Acceptance is $1-\mathrm{TV}(p,q)$ averaged over your traffic (Equation 5), so it is a property of the specific model pair and domain, not a universal constant. Measuring it is cheap: run the draft and target side by side on a sample of real prompts and record how often the lossless accept rule keeps the draft token. Do not borrow an acceptance number from a paper on a different pair or a different domain, because a draft that aligns well on one distribution can align poorly on yours, and the acceptance rate is the single input that most determines your entire frontier.

Step two: decide whether you can be lossy. If reproducibility, evaluation, or policy requires the served distribution to equal a reference, stay at $\lambda =0$: plain speculative decoding, which still buys the reported 2 to 3 times ^[1][2]. If you have measurable, tolerable quality slack, you may open the deferral knob.

Step three: check the regime. If you serve at large batch and decode is compute bound, the free-verification premise (Section 7) does not hold and a plain cascade or better batching may beat coupling. If you serve small-batch interactive traffic, the coupling advantage of Theorem 1 is real.

Step four: set the block length near $g^{\star }$ from Equation 9 or its approximation Equation 10, then sweep a small neighborhood because acceptance is not truly i.i.d. (Section 7).

Step five: turn the one dial. Sweep the shadow price ${\mu }^{\star }$ of Equation 12 and watch your actual task metric, not the total variation surrogate. Stop at the smallest quality budget that hits your cost or latency target. This is the operating point the paper says is optimal within the family.

A worked reading of these steps: a team serving an interactive assistant measures a draft acceptance of $0.7$ and a cost ratio of $0.06$, decides it can tolerate a small, evaluated quality drift because its outputs are not compared against a fixed reference, confirms it serves at batch sizes in the low single digits (memory bound), sets the block length to five from Equation 10, and then sweeps the shadow price while watching its own answer-quality metric until the metric just begins to move, stopping one notch before. That final operating point is a lossy speculative cascade tuned to the largest safe budget, and by the case study of Section 4.6 it should sit ten to fifteen percent below the lossless latency at a quality cost the team has measured rather than guessed. The whole procedure is arithmetic on three measured numbers plus one metric sweep, and it can be done before committing to an implementation.

Above all, invest in the draft. Because acceptance is the master variable in every equation, the highest-impact engineering is raising it: distill the draft against the target on on-policy data^[14], or train the pair to be cascade-aware^[20]. A better draft accepts more at zero budget and drifts less per unit leniency (Equation 7), so it widens the whole usable frontier at once. Teams building this into a serving stack will find the relevant tooling in platform engineering and the measurement discipline in LLMOps and observability; the cost-governance framing connects to our agentic FinOps work, and product teams shipping latency-sensitive features should read GenAI apps.

One caution completes the procedure: measure quality on the leniency-adjusted output, not on the lossless output, and measure it continuously, because the safe budget is not a constant. Acceptance and the draft-target distance drift as traffic changes, so a leniency that was safe last month can become too aggressive this month if the input distribution has moved and the draft has fallen out of alignment. The right operational posture treats the quality budget the way a reliability team treats an error budget: set it explicitly, monitor the actual task metric against it, and pull the leniency back automatically if the metric approaches the ceiling. This turns the static optimum of Section 4 into a governed control loop, and it is the only way to run a lossy scheme responsibly in production, where the alternative, setting a leniency once and forgetting it, risks a slow, unmonitored quality regression that no single request would reveal.

8 Open Problems and Future Work

Multi-tier coupling. Everything here is two models, a draft and a target. A real deployment often has three or more tiers, and the optimal coupling of a chain, where an intermediate model is both a verifier for the tier below and a draft for the tier above, is open. The frontier likely generalizes, but the block-length and threshold optima couple across tiers: a token accepted at the middle tier need never reach the top, so the acceptance rates compose and the quality budget must be allocated across the chain rather than set once. A naive design applies Equation 12 independently at each tier, but the budgets interact, because drift introduced at a lower tier is carried upward, and the joint allocation that minimizes end-to-end latency under a single end-to-end quality budget is a genuinely harder optimization than the two-model case we solve here. The classification-cascade literature has studied multi-stage deferral^[3], and combining that with the speculative verifier is the natural next step.

Batch-aware leniency. The free-verification premise degrades continuously with batch size (Section 7). A leniency schedule $\lambda (B)$ that opens the knob at small batch and closes it as batching climbs toward the ridge point would track the frontier as load varies, but requires a roofline-aware cost model inside the serving loop^[23].

Learned couplings beyond the likelihood ratio. Equation 12 is optimal within the leniency family, but the family is parameterized by a single scalar slack. A learned per-token deferral on richer features^[10][21] could dominate it on a given task at the cost of the clean guarantee; characterizing that trade is open.

Coupling in agent loops. Agentic systems interleave generation with tool calls, so the acceptance rate varies sharply between free-form reasoning spans and structured tool arguments. A coupling that raises leniency on prose and drops to $\lambda =0$ on structured spans would be the decoding analogue of the constrained-generation work in our constrained-decoding semantics paper, and connects to reasoning-cost trade-offs in the cost of reasoning.

Verifier-free deferral. The whole scheme assumes a target verify pass. Consultant Decoding ^[27] and related work ask whether the target can be consulted less often than once per block, which would change the cost model's constant term and move the frontier.

Acceptance under distribution shift. The acceptance rate that drives every equation is measured on some reference traffic, but production traffic drifts, and a draft that was well aligned at deployment can degrade silently as the input distribution moves. The recent theory of when a draft is accepted ^[25] is a start, but an online estimator that tracks acceptance and re-solves for the block length and leniency as traffic shifts, without a costly re-measurement pass, is open. The natural design monitors the empirical accept rate in the serving loop and nudges the two knobs toward their re-derived optima, which turns the static analysis of this paper into a controller; the stability and convergence of that controller are unstudied.

Position-aware and content-aware leniency. Section 6.4 noted that reasoning traces and structured spans have sharply different safe leniencies. A principled schedule would set the deferral threshold as a function of a lightweight predictor of local difficulty or output type, spending budget where it is cheap and conserving it where a single wrong token is costly. This is the token-level analog of the learned deferral rules that outperform fixed thresholds at the sequence level ^[10][21], and characterizing the optimal schedule under a global budget is a clean extension of Equation 12 from a scalar shadow price to a state-dependent one.

Frequently Asked Questions

What is a speculative cascade in one sentence?

A two-model scheme in which a small draft proposes a block of tokens, a large target scores the whole block in one memory-bandwidth-bound pass, and a tunable per-token deferral rule decides whether to keep each cheap draft token or escalate to the target. Zero leniency recovers lossless speculative decoding^[1]; opening the knob recovers a token-level cascade^[10]; the interior beats both ^[4].

Why does verifying many tokens cost about the same as generating one?

Because batch-one decoding is memory-bandwidth bound: producing a token requires streaming the full weights from memory, and that streaming, not the arithmetic, sets the time ^[11]. The arithmetic intensity is about half a FLOP per byte, far below any accelerator's roofline ridge^[12], so scoring a whole block on already-resident weights is nearly free. That is the entire engine of draft-verify decoding. The corollary matters as much as the fact: the free lunch exists only while the workload is memory bound. Push batch size high enough that each weight sweep is shared across many sequences and the accelerator crosses its roofline ridge into the compute-bound region, where the extra arithmetic of verifying a block is no longer absorbed by idle units, so the same physics that makes the scheme work at small batch predicts it stops helping at large batch.

Is a speculative cascade lossless?

Only at zero leniency. Exact speculative decoding is lossless by an algebraic identity of its accept rule ^[2]. Any positive leniency accepts draft tokens the lossless rule would reject, so the output distribution drifts toward the draft, bounded by Equation 7 at $\lambda \mathrm{TV}(p,q)$. Speculative cascades are therefore in the lossy family with classical cascades ^[3], not the lossless family.

What is the optimal coupling, precisely?

Two settings. The block length $g^{\star }$ from Equation 9, which balances the geometric benefit of a longer block against wasted draft passes; and the deferral threshold from Equation 12, a per-token bound on the target-to-draft likelihood ratio at a single shadow price ${\mu }^{\star }$ fixed by the quality budget. At zero budget the threshold becomes the exact lossless rule.

How is this different from FrugalGPT?

FrugalGPT routes whole responses sequentially: run the cheap model, score the full answer, and re-run the expensive model if the score is low^[3]. That pays both models' latency back to back and forces an entire response to the big model over one hard token. A speculative cascade routes per token in parallel: the target already scores the block, so the deferral is free and localized, and it never pays the sequential penalty (Theorem 1). The two also differ in how they measure difficulty: FrugalGPT learns a scorer over the whole response, which inherits the length bias that afflicts sequence-level uncertainty ^[10], whereas a speculative cascade reads difficulty off the target's own per-token probability, which is available for free from the verify pass and carries no length bias because it is local. The savings FrugalGPT reports, up to 98 percent on a workload where a small model suffices for most queries ^[3], are real, but they are sequence-level and sequential, which is precisely the inefficiency the coupling removes.

How is this different from ordinary speculative decoding?

Ordinary speculative decoding has one accept rule, chosen to be lossless, so its speedup is capped by the draft's acceptance rate and it cannot spend quality for cost ^[1]. A speculative cascade replaces that single rule with a one-parameter family (Equation 6) and lets you trade a bounded amount of quality for extra acceptance, latency, and cost.

When should I use plain speculative decoding instead?

When losslessness is required, for reproducibility, evaluation against a reference, or policy, or when you have no measured quality slack to spend. Plain speculative decoding is the safe default and still delivers 2 to 3 times^[2]. The cascade earns its keep only with a measured, tolerable quality budget and a target the lossless speedup cannot reach.

When does a cascade without speculation win?

In the compute-bound regime. At large batch or with long shared prefixes, decode climbs past the roofline ridge and the verify pass is no longer free^[11][23]. A classical cascade that avoids invoking the large model then saves real compute, and the coupling advantage shrinks. Speculative cascades are a small-batch, latency-sensitive technique. The practical dividing line is whether your serving is dominated by many concurrent sequences sharing each weight load, in which case you are compute bound and a plain cascade or better batching is the right tool, or by a few latency-critical streams, in which case you are memory bound and the coupling wins. Many production systems live in both regimes at different times of day, which argues for a batch-aware policy that opens the deferral knob under light load and closes it under heavy load, an extension we flag as open in Section 9.

How do I pick the block length?

Measure acceptance $a$ and cost ratio $c$, then use Equation 9 (or its approximation Equation 10): the optimum grows with acceptance and shrinks with draft cost, landing at a handful of tokens for the reported regime ^[17]. Then sweep a small neighborhood because real acceptance is not i.i.d. ^[25].

What single change most improves a speculative cascade?

Raise the draft's acceptance rate, because it is the master variable in every equation. The cleanest lever is distilling the draft against the target on on-policy data, reported to buy 10 to 45 percent^[14]; a closer draft accepts more at zero budget and drifts less per unit leniency, widening the whole frontier. Two things follow. First, the same distillation effort helps a lossless deployment and a lossy one equally, so it is a safe investment before you have decided how aggressive to be. Second, because acceptance is a property of the specific model pair on your specific traffic, a draft that a benchmark reports as excellent may accept poorly on your domain; measure it on your own inputs rather than trusting a leaderboard.

Does the analysis assume anything about the model architecture?

Very little. The only architectural facts it uses are that decoding is autoregressive, so tokens are produced one at a time and a block can be verified in parallel, and that at small batch the target's forward pass is memory-bandwidth bound, so verifying a block costs about one weight sweep^[11][12]. Both hold for standard transformer decoders ^[22] and for most current serving setups at interactive batch sizes. The analysis does not depend on the number of layers, the attention variant, the vocabulary size, or the specific draft mechanism, which is why the same equations describe classic small-model drafts, multi-head drafters, and feature-space drafters alike; those differ only in the acceptance rate and cost ratio they achieve, which are the two inputs the model takes.

How does this interact with long reasoning or agent workloads?

Favorably in magnitude, with a caveat in uniformity. Long chain-of-thought and multi-step agent loops multiply the number of sequential decode steps per user action, and because the per-token latency saving is linear in output length, a ten percent per-token improvement becomes a ten percent saving on a very long trace, which is where it matters most. The caveat is that these workloads mix easy tokens (connective prose, boilerplate) with hard ones (an inferential leap, a tool argument), so a single fixed leniency is a compromise, and on structured tool arguments you should drop to zero leniency because one wrong token breaks the call. The right design is content-aware, an open problem we flag in Section 9, and the constrained-generation side of it is developed in our constrained-decoding semantics paper.

Glossary