Predictive Speculative KV Replication for Bursty LLM Inference

 (jwlabs.vercel.app)

41 points | by shreybirmiwal 3 days ago ago

5 comments