Tracon

/ the problem

Serving schedulers treat requests as independent one-shot inferences. Agent workloads are not: they are dependency graphs with sessions, and tool execution rather than model time dominates end-to-end latency — so one slow tool call head-of-line-blocks everything queued behind it, and a session's warm context has locality that schedulers currently ignore.

/ approach

• Measured the workload before designing for it: an exporter turns real agent transcripts into a content-free trace — shapes, sizes, ids and timings survive; prompts, arguments, paths and outputs do not
• Characterisation against that corpus: 79% of busy time is tool execution, 3.7% of calls hold 75% of the tool time, and 74% of prompts arrive while the system is already busy
• One C++ decision kernel compiled twice — a pybind11 module the simulator calls in-process, and a Go gRPC service via cgo — with identical decisions verified across both transports
• Every policy evaluated in a trace-driven discrete-event simulation against a FIFO plus dynamic-batching baseline, with methodology and hardware stated next to each number

/ results

• Dependency- and context-aware scheduling cuts p95 turn latency against FIFO at 16× replicated load, with no oracle knowledge
• Simulator reproduces observed turn latency at 0.0% median and 0.1% p90 replay error
• Scheduling decisions cost 1–52 µs in-process and 215–417 µs over localhost gRPC, by queue depth
• The accompanying failure study replicates on an independent 94,059-call public corpus, and reports which of its own findings did not survive

/ architecture

/ links