As some of you may know we had a bit of a snafu the other week. We run several virtual machines on our server to mitigate problems should one of those VMs suddenly decide to stop working or get hacked or die or destroy it's filesystem... Which is exactly what happened!

We had two 3TB SATA drives in raid1 which was used as storage for the Minecraft VM and the email VM. Back then, the web server was also placed on the Minecraft VM. Then the ext4 file system of the Minecraft server died when I tried cleaning up some old ARK server instances (which have been running in their own VM for a really long time now). A disk check recovered most of the files and directories, but but them all in lost+found which means each directory and file got placed out of context. Instead of having a path like:
Code:
/dir_A/sub_dir_1
/dir_A/sub_dir_2
/dir_A/sub_dir_3
/dir_B/sub_dir_1
/dir_B/sub_dir_2
/dir_B/sub_dir_3
etc
We got this:
Code:
/random_dir/files
/random_sub_dir/files
/random_sub_dir/files
ect
We do have backups of the email and Minecraft servers so it's not too bad. But I didn't want to take any chances and replace the two 3TB SATA drives. I've bought two 6TB SAS drives.
Seagate Enterprise capacity wdbmma0060hnc-ersn 3.5 HDD 7200rpm SAS 12 GB/S 256 MB cache 8,9 cm 3,5 inch 24 x 7 512 Native BLK
Name:  Seagate Enterprise drive.jpg
Views: 900
Size:  38.0 KB
2x EUR 240,93

While waiting for my order to arrive I set up a new VM dedicated as just a web server and restored the email server. I decided to go with Ubuntu 18.04 as we ran Ubuntu 16.04 before which was fine. Unfortunately Ubuntu 18.04 seemed to hang every few days. No crashes, it just froze with 1 CPU core at 100% load. So I decided to upgrade VMWare as well which is a bit of a hassle. For this to work I had to create a custom ISO of the VMWare ESXi installer that had drivers on it for our raid controller. Having done that I pulled the USB stick with the old VMWare ESXi installation on it. Then created a boot USB stick and put in a different stick in the server to install to. Installation failed a few times. So... I swapped installer USB disk and destination stick. Installation still failed. I dug around and found another USB stick to install ESXi on. It's a Sandisk 32GB stick. One of those really tiny ones that are barely larger than the USB plug itself. Installation succeeded! Then create virtual switches, import all the remaining VMs, set them to auto-start and we've been running happily ever since.

The new 6TB SAS drives arrived, I put them in the server. And nothing. The drives weren't detected by the raid controller, at all. It seems the controller is too old, or the drives lack proper backward compatibility, whatever. It doesn't work.

So I've also ordered a new SAS raid controller:
Fujitsu SAS Controller psas cp400i 12 GB/S 8 poort based on LSI sas3008
Name:  Fujitsu SAS Controller psas cp400i.jpg
Views: 678
Size:  14.8 KB
EUR 216,72

Then added some cables and decided to also replace the case fans, 5 total, as those have been running 24/7 for the past 7 years or so.

The new raid controller should arrive this week. Assuming it works and we can get a working raid1 array with the new 6TB SAS drives, I can then create a new VM for Minecraft, restore the back-ups and get that part up and running again.

So yea, snafu