Lookahead

less than 1 minute read

Last blog about Medusa was going longer than I expected. So I will write separete blogs about lookahead and EAGLE1/2

Lookahead is from LMSYS, and I mainly read from this blog and this post

1 Jacob Iteration Algorithm

This is an algorithm to solve nonlinear equations iteratively. Details here. It breaks down the matrix to a diagonal one (easy to revert it by taking reciprocal of all diagonal elements) and L/U. Alt text

The idea is to change auto-regression decoding into a nonliear equations system and solve all the tokens by iteration Alt text

2 Lookahead decoding

So we can solve all m tokens in k iterations with parellel decoding. If $k«m$ then we can achieve speed up with lookahead decoding. The paper is here Alt text

Tags:

Categories:

Updated:

You May Also Enjoy

K9S and Kubeadm

April 30 2025

After playing K8S for couple of weeks and started to deploy K8S and debugging network issues

Sliding Window Attention

April 27 2025

I was debugging a sliding window attention bug and it was fixed by this PR. I helped on the review and get it merged.

Eagle 1/2/3 + HASS

April 26 2025

Speculative Decoding w Eagles 0 Medusa review Zhihu explains Medusa building the tree attention has $\Sigma_{i=1}^N\Pi_{j=1}^i{C_i}$ branches ($N$ head and $...