Without speculation · serial target calls
Each new token requires another target pass
Target pass 1 … France is → Paris
Target pass 2 … is Paris → .
Target pass 3 … Paris . → The
3 sequential target-model steps to emit “Paris . The”
How speculative decoding works
The draft model proposes several future tokens cheaply, and the target model verifies them in parallel. The final emitted text is the longest accepted prefix, plus a corrected token if a mismatch occurs.
1 · verified context
Start from the current prefix
The capital of France is
The target model has already produced the anchor/bonus token is. Now we want the next few tokens after it.
→
2 · cheap drafting
Draft model proposes a block
Paris . It is
Small/cheap drafter or lightweight draft head
Goal: propose multiple candidate next tokens fast
→
3 · target verification
Target model checks the draft in parallel
Paris . It is
The target model accepts the longest matching prefix. Once a token is rejected, later draft tokens in that round are discarded.
→
4 · output of the round
Emit accepted tokens and continue
The capital of France is Paris . The
The system outputs the accepted prefix, then the target model supplies a corrected token at the rejection point, and the next speculative round begins.
Core idea: instead of asking the target model for one token at a time, speculative decoding uses a cheap drafter to guess several tokens ahead and lets the expensive target model verify them in one go.
Dimension key used throughout
B = batch size
n = verified context length
γ = proposal length / draft block length
|V| = vocabulary size
n = verified context length
γ = proposal length / draft block length
|V| = vocabulary size
k = target hidden size in EAGLE-3 notation
d = draft hidden size in DFlash/DSpark notation
m = number of target layers used as context
d = draft hidden size in DFlash/DSpark notation
m = number of target layers used as context
r = sequential-head rank/state size
DSpark default Markov rank: r = 256
hk ∈ ℝd, Uk ∈ ℝ|V|
DSpark default Markov rank: r = 256
hk ∈ ℝd, Uk ∈ ℝ|V|
Shapes below omit batch B for readability. Add a leading B dimension in batched serving.
Why speculative decoding can be faster
Autoregressive decoding normally makes the target model wait for its own previous token before starting the next step. A drafter breaks that dependency on the expensive path: it guesses a block first, then the target scores all draft positions together with causal attention.
With speculation · one block verification
The drafter exposes parallel work
Drafter is → Paris . It is
after “is” Paris accept
after “Paris” . accept
after “.” The reject “It”
after “It” … discard
after final “is” … discard
Causal masking keeps each position from seeing future draft tokens. 1 target forward pass
Serving batch · a second axis of parallelism
Verify several requests and several positions together
The server can stack the verification blocks for B requests. The target then runs dense GPU operations over a tensor with a request dimension and a token-position dimension. Padding is shown for clarity; production systems may instead pack variable-length blocks.
Request 1
is Paris . It is
Request 2
was founded in 1890 PAD
Request B
is 42 . PAD PAD
B × (γ + 1) × d one dense target-model verification batch
Fewer serial steps Several accepted tokens can advance each request after one verifier call.
Better GPU utilization More token positions turn small decoding operations into larger matrix operations.
Same target distribution Verification accepts only tokens consistent with the target model and corrects the first mismatch.
It is not automatically γ times faster. The drafter has a cost, rejected suffix work is wasted, and larger verification batches consume memory. Speedup is best when drafts are cheap, acceptance is high, and the hardware executes the wider verifier batch much faster than the equivalent sequence of target-model calls.
EAGLE / EAGLE-3
sequential / tree drafter target hidden featuresUses target-model hidden states from already verified tokens. EAGLE-3 fuses low, middle, and high target-layer features, then directly predicts token logits through a lightweight decoder and the target LM head.
l, m, h ∈ ℝᵏ
[l; m; h] ∈ ℝ³ᵏ → FC → g ∈ ℝk
draft output a ∈ ℝk → LM head → ℝ|V|
[l; m; h] ∈ ℝ³ᵏ → FC → g ∈ ℝk
draft output a ∈ ℝk → LM head → ℝ|V|
DFlash
parallel drafter KV injectionExtracts selected target-layer hidden states from the verified prefix, projects them into the draft hidden space, and injects them as extra keys/values into every draft layer.
H(li) ∈ ℝn×d
concat m layers: ℝⁿˣᵐᵈ
Wc: ℝd×md → Hctx ∈ ℝn×d
draft slots: ℝγˣᵈ → logits ℝγˣ|V|
concat m layers: ℝⁿˣᵐᵈ
Wc: ℝd×md → Hctx ∈ ℝn×d
draft slots: ℝγˣᵈ → logits ℝγˣ|V|
DSpark
parallel + tiny sequential confidence schedulerReuses the DFlash-like context-injected parallel backbone, then adds a lightweight Markov or RNN head so each sampled slot can depend on previously sampled draft tokens.
parallel: hk ∈ ℝd, Uk ∈ ℝ|V|
Markov: W1[xk−1] ∈ ℝr, W2 ∈ ℝr×|V|
bias Bk ∈ ℝ|V|, final logits Uₖ+Bₖ
Markov: W1[xk−1] ∈ ℝr, W2 ∈ ℝr×|V|
bias Bk ∈ ℝ|V|, final logits Uₖ+Bₖ
Run the example
Choose a method and step through one speculative decoding round. New “feature view” steps show the actual inputs and dimensions used by the draft model.
EAGLE / EAGLE-3
Fuses target hidden states, then drafts sequentially and can verify a tree.
draft accepted rejected dropped
1 / 6
Input feature comparison
| Question | EAGLE / EAGLE-3 | DFlash | DSpark |
|---|---|---|---|
| What is fed to the drafter? | Fused target features for verified tokens plus token embeddings for sampled/draft tokens. | Anchor token embedding + mask embeddings, with target hidden features injected as KV context. | Same DFlash-like parallel input, then previous sampled token features feed a Markov/RNN sequential head. |
| Uses target hidden states? | Yes. EAGLE-3 fuses low/mid/high layers for tokens already processed by the target. | Yes. Selected target layers are concatenated, projected, and injected into draft-layer K/V. | Yes, inherited from the DFlash-style backbone; DSpark also uses draft hidden states hₖ for confidence. |
| Hidden states for unverified draft tokens? | Not available from the target. EAGLE-3 feeds back its own draft output a as a proxy feature. | Not available from the target. Future slots are mask positions inside the draft model. | Not available from the target. Future slots are mask/anchor positions plus sequential bias from sampled draft tokens. |
| Output shape | One or more draft logits vectors in ℝ|V|, sequential/tree-expanded. | γ logits vectors: ℝγˣ|V| in one pass. | γ adjusted logits vectors: Uₖ+Bk ∈ ℝ|V| sampled left-to-right. |
Compact algorithm sketches with shapes
EAGLE-3-style
target pass over verified prefix:
li, mi, hi ∈ ℝk
gᵢ = FC([lᵢ;mᵢ;hᵢ]) ∈ ℝᵏ
draft step:
input: g for verified tokens
+ e(anchor) or previous a as proxy
decoder output a ∈ ℝk
logits = LM_head(a) ∈ ℝ|V|
sample token
feed a back for deeper draft stepsDFlash
target context:
H^(l₁)...H(lm) ∈ ℝn×d
Hctx = RMSNorm(Wc concat(H)) ∈ ℝⁿˣᵈ
draft input:
[E(anchor), E(MASK) ...] ∈ ℝγˣᵈ
each draft layer:
K,V = concat(project(Hctx), project(Hd))
output logits ∈ ℝγ×|V|DSpark
parallel backbone:
hk ∈ ℝd, Uk ∈ ℝ|V|
Markov head:
q = W1[xk−1] ∈ ℝr
Bₖ = q W2 ∈ ℝ|V|
pₖ = softmax(Uₖ + Bₖ)
confidence:
cₖ = sigmoid(wᵀ[hₖ; q])
scheduler chooses prefix length ℓReferences
- Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv:2401.15077.
- Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2025. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv:2503.01840.
- Jian Chen, Yesheng Liang, and Zhijian Liu. 2026. DFlash: Block Diffusion for Flash Speculative Decoding. arXiv:2602.06036.
- Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong, et al. 2026. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. DeepSeek-AI.