The box went dark at 05:15 and stayed that way for three and a half hours, until I could physically unplug it in the morning.
It was a backup inference node in the lab’s AMD fleet: four older gaming cards on a 2013-era desktop motherboard that fans its PCIe lanes out through a PLX switch chip. The box is supposed to recover on its own. It didn’t.
The symptom was specific. Under load, and sometimes near idle, the SMU (the card’s power-management microcontroller) on one GPU would stop answering commands. The driver would try a GPU reset, but the auto-selected reset path on this GPU family works by commanding the SMU, and the SMU was dead. The reset stalled, the kernel hung, and all telemetry vanished.
The signature detail was in the logs: the failure lived at the same PCI address every time, across many boots.
I spent the next two weeks chasing the card.
The thermal theory
My first instinct was heat. I swapped the fans for hygiene, but the crash temperatures were benign, and a sibling box with an earlier incident had already taught me that this card family could crash at idle in a cool garage. An idle recurrence is about the cheapest disproof of a load or thermal theory there is. The fans were fine. The card was still dying at the same address.
The driver hardening
Next I looked at the software path. The default reset relied on the SMU, so I forced the SMU-independent reset method, disabled the GPU family’s deepest sleep state, and forced-enable recovery. I rebooted and verified the settings.
Two days later, the box hung again: same signature, same address.
This was correct engineering. The reset path was sound for a healthy card, but a software reset path inside a hung kernel can’t save you. If the SMU is unresponsive, the driver is just shouting into the void. The box remained dark.
The watchdog stop-gap
The chipset watchdog was locked off by the BIOS and I couldn’t reach it, so instead I drove the board’s Super-I/O watchdog: separate silicon wired to the reset line that the BIOS lock doesn’t touch.
I armed it and let it fire. The box hard-reset itself.
I built three layers of recovery: a hardware watchdog on kernel hang, a kernel panic reboot, and a health-probe timer for the “service wedged but kernel alive” case.
That collapsed the recovery window from 3.5 hours to about 6 minutes. It automates the power-cycle, but it doesn’t fix anything. The card still died. The slot still ate it.
The turn
I had to prove the card was innocent, so I moved the suspect card to a different slot. It ran clean.
Then I moved a known-good card into the suspect slot, and it went dark there.
All four cards were good. The killer was the slot, specifically the downstream port of the board’s PCIe switch, physically the second slot from the top.
Physical slot order doesn’t track bus order on a PLX board. The map had to be built by experiment, not read off the silkscreen. That slot was failing to maintain the link under the electrical stress of the SMU hang.
The slot dies
A few days later, the slot died completely. It wouldn’t train a link at all, and it survived a CMOS reset and every BIOS lever I had. The board was dead in that lane.
I pulled the fourth card and ran the box on three.
The fix I keep not making
The plan was a used server-class board and CPU with native PCIe lanes wired straight to the CPU, no PLX switch in the path to lose a slot to again. I priced one at around $150 and figured that would close this out properly.
I still haven’t bought it. The box has been rock solid on three cards ever since the slot died, and the CPU, old as it is, keeps pace with the newer X99 boards in the fleet on inference. Losing a slot cost a quarter of the card count and none of the uptime. The transplant sits on the shelf as an option I might never need.
Fleet-wide prevention
The same GPU family’s deep-idle state is implicated in these SMU hangs, so I rolled out a clock-floor pin to all six AMD boxes. It costs about 8 to 10 watts of extra idle power per card, and it keeps the SMU out of the state where it goes unresponsive.
The watchdog recovery stack now covers the whole fleet: prevention on one layer, recovery on another.
The rules
Swap-test the slot, not just the card. A fault that follows the slot is invisible if you only ever move cards. The swap in both directions is the experiment: suspect card elsewhere, known-good card into the suspect slot.
Localize the fault to one PCI address across multiple boots before accepting any mechanism theory. The address was in every crash log from day one. The fan and driver theories were built before I ever asked whether the fault moved with the card.
A watchdog is a stop-gap, not a fix. It automates the power-cycle while you hunt the real fault. Deploy it anyway. 3.5 hours of darkness became 6 minutes.
A degraded system that holds is a legitimate outcome, not an unfinished job. Three working cards beat four with an intermittent hang. If the stable configuration isn’t costing you anything you’re actually using, the “real fix” can sit on the shelf.