This is part one of a probably extremely long series of tales of when I've made the mistake of believing AI without applying sufficient thought. This time, it's about rebooting my server.
First a caveat. I'm not an AI zealot. I don't think it's right that the AI companies have stolen vast amounts of copyrighted material to populate their models, and I don't think it's right that they are wantonly destroying the planet with their data centres. Nor do I agree with the financial stupidity of the AI bubble. As an aside, it's a complete waste of resources to use the technology to produces deep-fakes for consumption on social media, even if they're amusing for 10 seconds. However, the AI cat is out of the bag. At some point I'll get round to writing a blog post about my thoughts on AI in general and LLMs in particular.
Anyway, moving on to this particular AI-related "incident" at my own personal data centre behind my sofa.
Last weekend, I was on holiday in the very southwest of the UK, near Lamorna Cove, for those of you who know Cornwall and the St. Just peninsula in particular. This is approximately 250 miles (400 kilometres) from home. My server sent me an alarming email, leading to a short but annoying chain of events. The email stated that one of the disks in my main server's ZFS RAID array had failed, or rather had disappeared. This particular 1TB of storage was NVMe (the other four are SATA), and is the oldest of the five. Nevertheless, unlike the others it's from a quality manufacturer and wasn't expected to fail any time soon.
I don't know much about ZFS, but I do know that the Linux kernel can occasionally abandon a disk if it doesn't respond in a timely manner, so my first hope was that this was a transient problem; the kind of thing that happens once a year or less. Armed with an unhealthy heap of confirmation bias, I asked Google Gemini if this could be the case, and whether it was safe to reboot the server. Bear in mind that at this point the server was happily working away, albeit with a degraded disk array. You can probably imagine what's coming next.
Needless to say, my home server does not have ILO (Integrated Lights Out) or equivalent technology which provides a completely separate environment with which to remotely manage the device including its power state. Using my VPN, I was able to ssh to the server. I verified that the disk had been removed from service, but couldn't see anything I understood in the dmesg kernel output. Anyway, in for a penny in for a pound, and AI said it would be fine, so what could possibly go wrong? I rebooted.
It hung on shutdown.
The only thing of any real importance running on my server is email. There are five mail boxes spread across four domains, and a number of forwarders for people who no longer hold accounts. They are all family or friends. Given that I had no intention of curtailing my holiday and driving for 6 hours to investigate the problem, I had to message them all and explain what had happened. They were all very understanding of the upcoming 36 hour outage. The happy ending is that holding the power button down to power off, and then rebooting, fixed the problem as originally intended. The error was indeed transient, and ZFS automatically "resilvered" the array in about 5 minutes.
The moral of the story? I'm not sure really, but I wouldn't have triggered the failed remote reboot unless AI had given me the confidence to do so. That's not to say it wasn't completely my fault!