Today at work I got to enjoy another round of “Operation: Donkey Punch”, a term I picked up from my hacking friends. It’s where, after fighting with a device to make a change through standard means and failing repeatedly because the standard means are absurdly dumb or limited by policy or design, you tear it apart, make the change while it’s not looking, and then put it back together.
In this example, my department got a pair of Dell DGX devices — basically a set of ARM cores and a high-end NVIDIA GPU with lots of memory. They run a custom version of Ubuntu and are sold as desktop AI accelerators. They’re about the size of a Mac Mini. Team wants to try running models locally.
Thing is, the Setup Wizard expects you, the customer who just shelled out $5000 each, to be in a SOHO office environment, and it makes bad assumptions that don’t work in the Enterprise. My office has a MITM web filter that decrypts web traffic, sniffs it, and encrypts it with our own CA certificate. Every device that needs web access (which is mostly HTTPS these days) must have this cert installed or the device will trust nothing.
Despite being on the lab wired network, the Setup Wizard kept giving me the prompt to select a WiFi AP; that’s because even with correctly configured Ethernet, when it tries to call home to see if it’s truly on the Internet, it fails the cert trust and falls back to demanding WiFi. We can’t use WiFi in the lab; security policy.
There’s no widget to add a cert. Can’t even login on ssh or an alternate terminal. Completely locked out until Setup Wizard finishes. I tried every way to make it work.
Frustrated, I decided it was time to void the warranty. I opened the case, removed the storage, attached it to my workstation, copied the cert file to the right folder, simulated what update-ca-certificate does to “install” the cert, reinstalled the storage into the DGX, and powered it up. Restarted the Setup Wizard, and at the point where it would’ve asked for WiFi, it went directly to downloading system updates and finishing.
Fist up. Big ol’ punch to the head. Take it, bitch.