Rendered at 23:58:29 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
ak_t 35 minutes ago [-]
You don't have to send every single request twice, just the ones that are haven't returned in time. Wait until some threshold, such as your p95 latency, and send your backup request after that. Return whichever request comes back first, and it should cut your tail latency without doubling your cost, since it only duplicates the small % of requests at the tail.
Sending two identical parallel requests is the classic approach. But, logically speaking, it should also double the cost.
I would send a second request if the first request fails to return the first token within, say, 1 second. Then there's a chance the first request is stalling, which is an infrequent event.
I wonder if higher-availability tiers of LLM providers do a similar thing internally.
ImPostingOnHN 1 hours ago [-]
Token caching might help here, but if it returns the same result, faster, for the same price as priority, seems good
dvaplima 2 hours ago [-]
Nice turn around, does anyone has a benchmark regarding other types of requests (priority vs send twice) other than voice/call? Or the tests already test that?
eigenblake 3 hours ago [-]
I love this. Simple. Useful. To the point. If AI was used, I can't tell because it is clearly representing the author's beliefs.
thousand_nights 2 hours ago [-]
agreed, reads like a breath of fresh air, no fluff
It's a bit unintuitive, but they key idea is roughly 'If you're working on stale load data (as always), best of 2 strikes the right balance between distributing load evenly and giving more work to less loaded hosts'. If you do 'best of k', you end up with herd behavior, overloading one host. 'best of 1' sends too much traffic to slow hosts.
ramon156 2 hours ago [-]
for a tier thats twice the cost i would expect >2x the speed. somewhere 5-10x
e.g. 1.40m would become 0.30s.
do people really pay for these priority plans?
behnamoh 1 hours ago [-]
This is half the story; you should show performance per dollar. I doubt your 2x approach would fare well against the priority if you consider the costs.
ImPostingOnHN 1 hours ago [-]
The article mentions that the priority tier costs 2x normal, so the costs of running normal twice should be fine.
moffkalast 2 hours ago [-]
If you want a controllable and predictable system, host it yourself. APIs will always have outages, delays and breaking changes every so often. That's the price you pay for not doing it properly and outsourcing your job.
Google calls this a 'hedged request': https://cacm.acm.org/research/the-tail-at-scale/
I would send a second request if the first request fails to return the first token within, say, 1 second. Then there's a chance the first request is stalling, which is an infrequent event.
I wonder if higher-availability tiers of LLM providers do a similar thing internally.
It's a bit unintuitive, but they key idea is roughly 'If you're working on stale load data (as always), best of 2 strikes the right balance between distributing load evenly and giving more work to less loaded hosts'. If you do 'best of k', you end up with herd behavior, overloading one host. 'best of 1' sends too much traffic to slow hosts.
e.g. 1.40m would become 0.30s.
do people really pay for these priority plans?