The ops dashboard showed something that did not look like inference. Every GPU in the cluster ramped up to about 100 watts, sat there briefly, and dropped back, on a regular beat. Utilization read zero. No compute process started or stopped. The RTX 3080s went from about 20 watts to between 86 and 100. The 3070s went from about 20 to between 50 and 70. CPU package watts did not move at all.
The GPUs outside the cluster, the AMD lanes and the embedding fleet, were flat at their idle floor the whole time. Whatever this was, it only touched the sixteen cards that serve our 262B model.
Reading the chart wrong
The first read was “every few minutes.” That turned out to be the chart, not the hardware. The node detail page draws its power sparkline over a five minute window by default, and every card showed five humps. Five humps in five minutes is one per minute.
A one minute period on a cluster nobody was using is a strong hint. People do not send requests on a metronome. Software does.
Counting the requests
The coordinator log answered it. Since the model was last launched, the coordinator had served about 1,863 requests over 31 hours. Of those, 1,848 were the same thing: a completion with a four-token prompt, arriving once every 60.78 seconds. About 15 were real traffic.
That signature belongs to our cluster sentinel, a small service on adhara that watches the production coordinator and relaunches it if it dies. Once a minute it sent a real chat completion, the word “ping” with a one-token output limit.
It has to be a real completion, and that part is not a mistake. The sentinel exists because of an earlier incident where the coordinator went zombie: it answered its health endpoint happily for ten hours while completing nothing. A monitor that only asks “are you up” would have watched that outage and reported green the whole time. So the sentinel asks the model to actually generate a token.
Why a tiny request is expensive
The probe itself is cheap. It is a four-token prompt evaluation that takes about 0.65 seconds. But it runs through the whole pipeline, which means every one of the sixteen cards does a slice of work. And an NVIDIA card holding a live CUDA context does not drop back to idle clocks when its kernel finishes. Each card sat at raised power for roughly 20 seconds after every probe.
So about 1 percent compute duty became about 33 percent power duty, on every card, all day.
We measured it in a window with no real traffic. The sixteen cards have a summed idle floor of 328 watts. The actual mean was 545 watts. That is 217 watts continuous, 66 percent over the floor, for a health check. It works out to about 3.7 watt-hours per probe and about 5.2 kilowatt-hours per day.
To make sure the probe was the only source, we folded the power telemetry on the 60.78 second period. A single periodic source shows up as one contiguous hot slice and a flat remainder, and that is exactly what came out: about 20 seconds near 1,000 watts, then a dead flat 337 watts for the rest of the cycle. Two other timers on those boxes run every two minutes, a GPU health watchdog and a power cap job. They contributed nothing visible.
One trap we avoided on purpose: we did not diagnose this with an nvidia-smi sampling loop. On these boxes that collides with the monitoring agent’s own NVML poll and fakes GPU drops on the dashboard. The telemetry we already collect was enough.
The part that stung
While tracing this we found that the sentinel had been disarmed for three weeks. It had been stood down in late August, on a day of heavy forge work, with an empty disarm file, and the script honors an empty file forever. A disarmed sentinel does not restore anything, but it still probes and logs its verdict.
So for three weeks the 217 watts bought us a log file. No protection, full price.
Proving it
We stopped the service. Within one minute, total cluster GPU draw fell from 551 watts to 332. The hottest card went from 96 watts to 36. The coordinator stayed healthy. That drop was the fix for the night and also the proof that nothing else was involved.
The fix: tier the probe
The real completion is still the only honest test for a zombie, so we kept it and changed how often it runs. The sentinel still wakes every 60 seconds, but on most cycles it only runs checks that never touch a GPU: a TCP connect, a GET on the health endpoint, and a kernel read of the RDMA queue pair count on the fabric. The real completion runs on a slow tier, every 600 seconds by default. It also runs immediately whenever a free check looks dirty or the last real verdict was not healthy, so any failure streak goes straight back to a real probe every minute.
Kill authority did not move. Only real completion verdicts count toward a restore. A skipped cycle touches no counter.
Cost scales linearly with the real probe rate:
- every 60 seconds: about 217 watts
- every 300 seconds: about 43 watts
- every 600 seconds: about 22 watts
- every 1,800 seconds: about 7 watts
The trade is detection time for one failure class. A crash, a refused connection, or a hung socket fails the free check on the next cycle and keeps the old detection path. The zombie that answers health checks but never completes now waits up to one interval before the confirm path starts, roughly 13 minutes worst case at 600 seconds. The sentinel was built to prevent a ten hour outage. Thirteen minutes is a price we will pay.
The test suite went from 159 to 195 tests. We mutation-tested every new guard, and two of the new tests turned out to pass even with the exact behavior they claimed to pin deleted. Both were rewritten until a deliberate break made them fail.
After
We deployed it armed. The first two real probes landed 601 seconds apart with nothing between them. Per-minute cluster GPU draw over the next quarter hour read 326, 582 (the startup probe), then a flat run near 326, then 523 when the scheduled probe arrived, then 328 again. The hottest card sat at 35 watts in quiet minutes and 94 in probe minutes. Averaged over the interval, the residual cost is about 20 watts, down from 217.
We also changed how the sentinel gets stood down. A disarm now takes a token with a time limit, and the sentinel re-arms itself when it expires. An empty file is exactly how we lost three weeks.
Lessons
Price a synthetic monitor in watts, not just latency. On a GPU pipeline the energy per probe is set by how long the cards hold their clocks afterward, times every card in the pipeline. The probe’s own compute time barely matters. The only lever is probe rate.
Put everything that can be answered without a GPU on the fast loop. Keep real inference on a slow one, and escalate to fast real probes only after something looks wrong.
Use the hardware you are not investigating as a control group. The flat non-cluster GPUs told us this was cluster traffic before we read a single log line.
A monitor that is switched off should cost nothing. If disarmed still means probing, a forgotten disarm is pure waste. Make stand-downs expire.