This month we added another dual-Xeon box to the lab to take CPU-heavy model work off the machines that serve inference. It needed a fast private path to Deneb, the storage server that holds the model library, so we ran a direct 40-gigabit cable card to card between them. No switch, no fabric, just two ports and a piece of copper.
The link came up at 10 gigabits.
Both cards are 40-gigabit capable. The identical card model sits in Deneb and runs 40 gigabits into the rack switch on its other port. The cable is the same kind we use elsewhere in the room. And still, on every boot, the new server’s end trained at 10 and stayed there.
Wrong theory one: the cable
Neither end could read the cable’s ID chip. One side reported that the cable had no EEPROM. The other said the I2C bus was constantly busy. Forcing both ends to 40 gigabits with autonegotiation off would not bring up a link at all: thirty seconds of nothing, then back down.
That looked conclusive, and we wrote it up twice. The cable is bad. Buy a known good one.
I kept refusing it, for one cheap reason. The same cable model is running 40 gigabits out of Deneb into the switch right now. If we had a bad batch, that link would be sick too.
So we stopped staring at the broken link and compared it to the healthy ones. Every working card-to-card leg in the lab, the five that fan out from Adhara, runs with autonegotiation on and the far end advertising 40000baseCR4. Ours had autonegotiation off and advertised only the 10-gigabit modes.
Wrong theory two: autonegotiation
One command at each end, autonegotiation on, and the link jumped to 40000Mb/s. Four parallel iperf3 processes measured 39.4 gigabits per second with zero retransmits.
That parallelism matters, by the way. A single iperf3 process on this version is single-threaded and tops out around 20 gigabits, which had made an earlier measurement look like a hard 20-gigabit ceiling that did not exist.
We persisted the setting at both ends, wrote it up, and called it done.
Then we rebooted the box, because the entire point of that day’s work was proving the configuration survived a restart. Everything else came back: the static address, the route over the link, the RAID volume, the file mounts, the agent. The link came back at 10 gigabits with autonegotiation off at both ends.
And now nothing would move it. Not a udev rule. Not a systemd one-shot after the network came up. Not a watcher loop running on both hosts. Not applying at both ends inside the same second, or with the ports administratively down, or bouncing them afterward. Eight variations, all 10 gigabits.
The part we did to ourselves
Here is the piece worth carrying away, because it cost the most.
When the forced-40G attempt failed on day one, we backed it out with a command that sets the speed to 10 gigabits and turns autonegotiation off. On this driver, that narrows the port’s advertised mode list down to the 10-gigabit modes, and it stays narrowed until that machine reboots. Worse, a bare autonegotiation-on does not widen it again. It keeps whatever list is already there. Every watcher and boot hook we built was issuing a command that could not have worked.
There was a second trap sitting right next to it. The driver prints “Auto-negotiation: on” or “off”, which reads exactly like a setting you control. It is not. It is status, reporting whether negotiation completed. We spent a day trying to write to a field that only reports.
Once we restored the full default mode list at both ends and I reseated the cable, the link trained 40 gigabits in about three seconds. Then the next reboot dropped it to 10 again. The narrowed list had been a real problem, but it was never the cause.
The tell
Buried in the kernel log, once per boot, on every boot: “Unsupported cable detected.” A firmware event, raised by the card about its own port.
Two days of notes showed the shape of it. The link only ever reached 40 gigabits right after a hot re-plug of the cable, and it only ever died at a reboot. A re-plug logged no event at all. And the cable is not marginal: when it runs, it runs at better than 39 gigabits with zero retransmits.
Wrong theory three: firmware
Both boxes run a firmware version slightly behind the newest available, so the obvious next move was to flash.
The research said no. Going back through the release notes across this card generation, nothing documents how third-party cables are handled, or why boot would differ from hot-plug. The one boot-related fix people point to in the newer firmware applies only to ports configured for InfiniBand, and ours are fixed to Ethernet. The older firmware would have dropped a fix we already have.
Every community case matching our pattern ended the same way: buy a validated cable. Thirty-four sources, and the answer was always a purchase.
The free test
One thing nobody had tried was separating the firmware from the boot.
Unloading and reloading the card’s driver performs a full firmware restart without a POST. If the cable rejection fires at every firmware init, it should fire on a reload too. If it only fires at boot, then the trigger lives somewhere in the pre-OS path.
About two seconds after the driver came back: no event, autonegotiation complete, 40000Mb/s, far end advertising 40000baseCR4. The link retrained itself.
That is the whole answer. The firmware accepts this cable. Only a real boot rejects it. So the thing doing the rejecting is the card’s own boot ROM, which initializes the port during POST, classes the cable as unsupported, and leaves it at 10 gigabits for the remainder of that boot.
The fix
In the BIOS, under slot disablement, set that card’s slot to Boot Driver Disabled. The operating system still sees the card normally. The only thing skipped is the card’s option ROM at POST, which matters only if you boot from that card, and we do not.
The next boot logged zero events. The port came up at 40000Mb/s with autonegotiation complete and the far end advertising 40000baseCR4. Four iperf3 processes measured 39.5 gigabits per second with zero retransmits. Jumbo frames, the route, the file mounts and the agent all came back healthy.
No flash. No purchase. One setting.
Lessons
A status field that looks like a setting will cost you a day. We built a udev rule, a systemd unit and two watcher loops against a read-only readout.
The safe revert was not safe. The command we used to back out cleanly narrowed the port’s capabilities for the rest of the boot, and the obvious undo could not widen them again. Know what your cleanup leaves behind.
Reboot before you write it down. “Fixed” held until the first restart, twice. If persistence is the thing you are claiming, then the test is a restart, not a working link.
When every source says buy the part, look for the test nobody ran. The literature’s answer here was a validated cable. The test that actually split the problem, a firmware restart versus a real boot, was free and took seconds.
Refusing the first answer was right both times, and it was right for a cheap reason: another link in the same room was already using the same part happily. Compare the healthy thing against the sick thing before you condemn a component.
One postscript. The storage server carries the same card, the same firmware and the same cable type on another port. That one is scheduled.