Server hardware stuff

[InsaneJ]

Last night the server crashed. Turns out the Areca raid controller is slowly going bad. After restarting the server the controller wasn't recognized any longer. After another restart the card re-appeared.

As was announced on the front page I've removed 64GB of RAM from the server to be send to Jiro along with parts for the server he's building. When I started the server after that the Areca raid controller was gone from the system again. Another restart and it was back. Being able to reproduce that kind of behavior is never a good sign.

I've got back-ups of the complete VMs that run on the server but still it would suck if the raid controller would die completely before a replacement arrives. I've been looking for a second hand Areca controller. That's going to cost about 200 euro. For now I'm just going to hope that the controller will continue to function as we're not looking to be spending extra money on hobbies right now. We're in the process of moving and we've got quite a lot of expenses to cover in that regard. If it'll last a couple more months I'll be happy to replace it in the summer.

Anyway. Important thing to note is that the server got a big performance bump by taking out half the RAM. The reason for this is that one 64GB kit was rated for 2133MHz while the other is a 3000MHz kit which was running at 2133MHz. The new AMD CPUs benefit greatly from faster RAM which can't be said for Intel CPUs unfortunately. Right now the new TnFC server seems to be running about 25% faster than before. Or rather, it's using 25% less time per tick. Which is nice

[InsaneJ]

As some of you may have noticed the server reboots every few days now. Reason for this is the Areca 1680 controller which is very much ready to be replaced. I checked the purchase receipt for it and I've had this controller since April of 2010 and is has been running 24/7 ever since. That makes it a pretty awesome piece of hardware in my book

To replace the 1680 I have ordered a second hand ARECA ARC-1880DI-IX-12 SAS 6G 12 Port RAID Controller for $230. Two years ago this card still sold for $790 new. According to Areca support the ARC-1880 should serve as a drop-in replacement for the ARC-1680 card. I'll make backups before upgrading, but if all goes well it shouldn't be much more work than replacing the card, booting VMware ESXi and (maybe) importing the raid volumes and then the VMs on those volumes. If all goes well...

[InsaneJ]

The new raid controller arrived today. Customs screwed me over and added a 53 euro import fee....

In the next few days I'll be moving all the VMs from the Areca controller to the LSI one. In case the newer Areca card doesn't work with the current drives and configured raid sets and volumes, I can simply re-create those and then migrate the VMs there instead of having to recover from back-ups.

This means things will probably slow down a bit as everything has to run of two SAS drives for a bit. I'll also test the new card to see if it actually works. And when I'm satisfied it has a good chance of working I'll swap the old Areca for the newer one.

[InsaneJ]

It's been a while since the last update

Nothing much has changed hardware-wise, though we've been experiencing some problems with the new Vanilla server. The Java Virtual Machine(JVM) crashes every so often. For those that don't know Minecraft runs on Java which means it's code is processed through the JVM which translates Java code to machine code that your processor can understand. This way the developers of Minecraft (or in our case, Spigot) only have to write their code once and it can run on many platforms. Windows, Linux, FreeBSD, Mac. If it has Java installed on it, it can probably run the code.

So the JVM crashing is a very bad thing. In the past few months the file system of the Dynmap hard drive had gotten corrupted. I've even re-rendered Dynmaps for TFC and TnFC which took weeks. All of our Dynmap tiles (images) reside on a single WD Purple drive. The drive itself reports no errors. What I'm going to do now is move the virtual hard drive in which all Dynmaps files resides from the WD Purple drive to one of the raid arrays. Then I'm going to replace the WD Purple drive with an old hard drive I have lying around. Then move the virtual hard drive to that 'new' physical drive and see if that changes anything.

Technically I could do all these changes while the server is running. But to be on the safe side I prefer to bring the server down when swapping parts and handling cables. This means there will be some down time in our future so don't worry if the servers or site is down, it's probably me working on something.

[InsaneJ]

All Dynmap files are now running of a raid 5 array of WD red drives. I've removed the WD purple's volumes and configurations from the server and raid controller, it is no longer mounted or being used. We'll see if that makes any difference.

[InsaneJ]

My plans for the upcoming weeks is to upgrade the server again. We're still having intermittent stability issues with the current server and it's getting old having to deal with those.

What I'm thinking of doing is replacing the 1920X Threadripper with an AMD 3900X or perhaps a 3950X on an X570 motherboard with 128GB of memory. Rough estimate:

• 700 for the memory
• 300 for the motherboard
• 450 or 775 for the CPU
• 75 for CPU cooler

So somewhere between 1500 and 1750 euros. I haven't decided on the exact memory type (ECC/non-ECC) and motherboard yet. I'm currently looking into VRM benchmarks.

The CPU will be a lot faster clock-for-clock so along with the extra memory the new server will offer enough capacity to run additional game servers like the new Technode Firmacraft for 1.12 pack.

As always. If you're feeling generous, you know where the donate button is

[InsaneJ]

Alright, I bit the bullet and ordered the following parts.


Gigabyte Aorus X570 MASTER
Excellent VRM setup. Only beat by boards costing several hundred euro more. It has two ethernet connections: 1gbit Intel and 2.5gbit Realtek. The Intel nic will be dedicated for the PFSense router while the Realtek nic will handle all the LAN traffic. The motherboard has support for ECC memory but I opted to go with a non-ecc kit (see below) because the kits on the "supported list" only went up to 16GB sticks and we need 32 to get to 128GB RAM total.
€379

Spoiler!

AMD Ryzen 9 3950X
16 cores / 32 threads. The fastest mainstream CPU currently available. Not much to say here. The only way to get an even faster CPU is to go with a high-end desktop system based on Threadripper or server grade Epyc. Intel just isn't an option anymore at this point.
€779

Spoiler!

Corsair Vengeance LPX CMK64GX4M2D3600C18
Two kits of 2x 32GB for a total of 128GB. This kit is on the "supported list" of the motherboard. Although it's rated at 3600MHz, when running 4 sticks of 32GB the memory speed will drop to 2666. I could have gone with a (very) slightly cheaper kit of 4x 32GB 2666, but with this kit I potentially have the option to run the memory at lower timings.
€668 (2x 334)

Spoiler!

Noctua NH-D15
One of the best air coolers on the market.
€89

Spoiler!

VGA0419
I've also bought a low power (2W) PCI-e graphics card. It's basically one of those onboard VGA adapters you see on server boards, except now on a PCI-e card. It's only meant to display the BIOS and text-based terminals. It's been running in the current server for almost a week now. Saves a bit on power and it was the last part I could swap out without having to replace motherboard/cpu/ram.
€8 on ebay

Spoiler!

All in all these parts cost a bit more than I initially thought they would coming in at a little over 1900 euro. The parts should arrive June 26th. Looking forward to assembling the new server