Systems · Distributed — 2026
An inference control plane that treats batching, routing, admission and rollout as one scheduling problem — and holds 16× the goodput of fixed batching once the fleet is past saturation.
Shipped — public, 159 tests, CI exercises a live fleet on every push
Image slot
Halcyon — goodput against offered load for three scheduling strategies
1600×1000 · the goodput-vs-load chart, or the architecture diagram
Role
Solo build — scheduler, simulator, benchmarks, rollout controller
Year
2026
Stack
- TypeScript
- Node.js
- Discrete-event simulation
- Peak-EWMA routing
- Kubernetes
The problem
Inference service time is affine rather than linear in batch size — a fixed launch and weight-paging cost plus a marginal per-token cost — so serving one request at a time burns most of a GPU on overhead that was paid for and thrown away. Batching recovers the throughput and immediately introduces queueing delay, and a fixed batch size or timeout is a number that is correct at exactly one arrival rate. Meanwhile replicas are not interchangeable, a latency-aware balancer will happily send more traffic to the replica that is failing fastest, and a new model version still has to ship without betting the user base on it. These are usually four separate systems that fight each other.
Approach
Made batch size emergent rather than configured: earliest-deadline-first admission against an online cost model, so requests that provably cannot meet their deadline are shed in microseconds instead of consuming capacity and missing anyway.
Scored replicas with peak-EWMA under power-of-two-choices, with outcome-based circuit breaking, so a replica that fails in two milliseconds stops looking attractive to the load balancer.
Replaced the static concurrency cap with a gradient-based limiter that infers queueing from latency, alongside per-tenant token buckets and priority-graded shedding.
Gated progressive rollout on evidence: sticky-hash traffic splitting with a controller that promotes or rolls back a canary by comparing it against a concurrent baseline rather than against a dashboard.
Benchmarked it in a fully seeded discrete-event simulation — same arrival trace, same device model, three schedulers — then verified the same behaviour against a real running fleet in CI.
Results
- goodput past saturation
16×
goodput past saturation
fixed batching at 260 req/s
- goodput past saturation
83.1/s
goodput past saturation
5.2/s under fixed batching
- p95 past saturation
1,020 ms
p95 past saturation
2,618 ms under fixed batching
- urgent SLO met at saturation
41.0%
urgent SLO met at saturation
27.9% under fixed batching
- tests, plus a live fleet load test in CI
159
tests, plus a live fleet load test in CI
Below saturation Halcyon and fixed batching are indistinguishable, and the repository reports that rather than hiding it — deadline awareness earns nothing when there is slack for everyone. It earns everything at the edge, which is where systems actually break.
Image slot
Halcyon control plane architecture
1600×1000
Image slot
Canary rollout decision trail under sustained load
1600×1000
What it came to
The result worth keeping came from the rollout controller catching something nobody had written a test for. A canary looked twice as fast as the baseline at 25% of traffic and was rolled back at 60%, because one canary replica was absorbing 60% of the load while two baseline replicas shared the rest. The canary was never slower — it was under-provisioned, and that is invisible until you ramp. A rollout validated at 1% and then promoted would have shipped it.