Delilah died on a Friday afternoon, eleven years old, in my arms under my desk. That was her spot. Cancer. A tumor on her front leg we’d had removed a year ago came back meaner than the first time. An in-home appointment was already scheduled for Monday, because my wife is the realist in this house and knew it was close to time. Delilah didn’t wait for the appointment. She was at my feet, where she always was, and it took about thirty seconds, and I was there for all of them. I held her and heard her last breaths, and there is no version of that I would trade, and it wrecked me anyway.
Some people process grief by sitting still. I am not one of those people.
The night everything broke
That evening I did what my hands wanted to do: I tore the server tent apart. New shelves. A new-to-me Dell R730 that had been waiting for a home. Every cable out, every box moved, two machines relocated back into the house. If you’ve ever done a full physical reorganization of a working cluster at night, while grieving, you already know what happened next: everything broke, and it broke in the most instructive ways possible.
The coordinator node started refusing SSH: auth succeeding, sessions dying
instantly, both ends of the connection blaming each other for the reset.
That turned out to be a jumbo-frame black hole: MTU-9000 endpoints with a
recabled path between them that silently ate any frame over 1500 bytes.
Small packets fine, big packets gone. Ping lies to you; ping -f -l 8000
tells the truth. That one’s in the knowledge base now as a permanent house
rule: after any recabling, big-ping the fabric pair-wise before trusting it.
The NAS came back wearing an IP address I’d retired weeks ago: a new NIC had reshuffled interface enumeration and resurrected a stale config. A GPU that had been quietly cooking on one dead fan chose that same afternoon to fall off the PCIe bus entirely. By 2 a.m. everything was green again, and I had learned more about my own network in one night than in the prior month.
Five install attempts
Then there was the new server. The R730 fought me like it was being paid to. The desktop installer black-screened on the GPU three separate ways: nouveau, Xorg hanging on a dual-VGA setup, safe graphics only sometimes safe. The RAID controller presented the boot SSD as invisible until I converted modes, and the Lifecycle Controller wedged itself so thoroughly that the only cure was the old Dell exorcism: both power cords out, hold the power button, let the flea power drain, count to sixty, start over.
Attempt five: Ubuntu Server ISO, text-mode installer, no graphics stack to hang. It installed in minutes. There’s a lesson there about fighting a graphical installer for a machine that will never have a monitor again, and I intend to keep ignoring it slightly slower each time.
Naming her
Every machine in this fleet is named for a star. Rigel, Deneb, Altair, Sirius, Vega, Polaris. The naming scheme was never supposed to carry any weight. It was just a convention, until a heavy 2U server needed a name the same day I lost my dog, and the rack slot happened to be right next to Deneb.
The dog stars were the obvious place to look. Sirius, the Dog Star and the brightest in the sky, was already spent on a VM. But Canis Major, the Great Dog, has a second-brightest star: Adhara. And Adhara has a property I can’t stop thinking about. In visible light she’s unremarkable; you’ve never picked her out of the sky. But in extreme ultraviolet, Adhara is the single brightest star there is. The brightest light in the sky, in a spectrum your eyes can’t see.
Eleven years of a dog under the desk, and now she’s gone, and the room is full of her anyway. Brightest in a light you can’t see. Yeah. That’s the one.
What she does now
Adhara-the-server came online at 3 a.m.: forty-eight Xeon threads, an RTX 3080 someone forgot to mention was inside, and a 40-gigabit point-to-point link to Deneb that moved its first model files the same night. A 400GB checkpoint that used to take an hour now lands in nine minutes.
And here is the part I didn’t plan, the part I only noticed the next morning. Like every node in this fleet, her idle CPU cycles go to World Community Grid. The work unit she picked up first, minutes after joining, before I’d decided anything else about her, was from the Mapping Cancer Markers project. Forty-eight threads, around the clock, working on the disease that took the dog she’s named for.
I didn’t script that. The scheduler just handed it to her. Some machines get a job; this one got an inheritance.
The tent is quiet now, the fabric is clean, and the Great Dog watches it from the rack next to Deneb. Good girl.