Page 2 of 5 FirstFirst 1234 ... LastLast
Results 11 to 20 of 46
  1. #11
    As some of you may know we had a bit of a snafu the other week. We run several virtual machines on our server to mitigate problems should one of those VMs suddenly decide to stop working or get hacked or die or destroy it's filesystem... Which is exactly what happened!

    We had two 3TB SATA drives in raid1 which was used as storage for the Minecraft VM and the email VM. Back then, the web server was also placed on the Minecraft VM. Then the ext4 file system of the Minecraft server died when I tried cleaning up some old ARK server instances (which have been running in their own VM for a really long time now). A disk check recovered most of the files and directories, but but them all in lost+found which means each directory and file got placed out of context. Instead of having a path like:
    Code:
    /dir_A/sub_dir_1
    /dir_A/sub_dir_2
    /dir_A/sub_dir_3
    /dir_B/sub_dir_1
    /dir_B/sub_dir_2
    /dir_B/sub_dir_3
    etc
    We got this:
    Code:
    /random_dir/files
    /random_sub_dir/files
    /random_sub_dir/files
    ect
    We do have backups of the email and Minecraft servers so it's not too bad. But I didn't want to take any chances and replace the two 3TB SATA drives. I've bought two 6TB SAS drives.
    Seagate Enterprise capacity wdbmma0060hnc-ersn 3.5 HDD 7200rpm SAS 12 GB/S 256 MB cache 8,9 cm 3,5 inch 24 x 7 512 Native BLK
    Name:  Seagate Enterprise drive.jpg
Views: 900
Size:  38.0 KB
    2x EUR 240,93

    While waiting for my order to arrive I set up a new VM dedicated as just a web server and restored the email server. I decided to go with Ubuntu 18.04 as we ran Ubuntu 16.04 before which was fine. Unfortunately Ubuntu 18.04 seemed to hang every few days. No crashes, it just froze with 1 CPU core at 100% load. So I decided to upgrade VMWare as well which is a bit of a hassle. For this to work I had to create a custom ISO of the VMWare ESXi installer that had drivers on it for our raid controller. Having done that I pulled the USB stick with the old VMWare ESXi installation on it. Then created a boot USB stick and put in a different stick in the server to install to. Installation failed a few times. So... I swapped installer USB disk and destination stick. Installation still failed. I dug around and found another USB stick to install ESXi on. It's a Sandisk 32GB stick. One of those really tiny ones that are barely larger than the USB plug itself. Installation succeeded! Then create virtual switches, import all the remaining VMs, set them to auto-start and we've been running happily ever since.

    The new 6TB SAS drives arrived, I put them in the server. And nothing. The drives weren't detected by the raid controller, at all. It seems the controller is too old, or the drives lack proper backward compatibility, whatever. It doesn't work.

    So I've also ordered a new SAS raid controller:
    Fujitsu SAS Controller psas cp400i 12 GB/S 8 poort based on LSI sas3008
    Name:  Fujitsu SAS Controller psas cp400i.jpg
Views: 678
Size:  14.8 KB
    EUR 216,72

    Then added some cables and decided to also replace the case fans, 5 total, as those have been running 24/7 for the past 7 years or so.

    The new raid controller should arrive this week. Assuming it works and we can get a working raid1 array with the new 6TB SAS drives, I can then create a new VM for Minecraft, restore the back-ups and get that part up and running again.

    So yea, snafu

  2. #12
    Quote Originally Posted by InsaneJ View Post
    So yea, snafu
    EUR 698.58 + unlisted amounts in cables and other parts for maintenance. More often than not J is backing the majority, if not all of the costs. We appreciate all of our donors that help

  3. #13
    Oh my, i hope everything will go well as planned.

  4. #14
    Quote Originally Posted by Vikusha View Post
    Oh my, i hope everything will go well as planned.
    It didn't.

    I got the new raid controller today and installed it in the server. Replaced the fans while I was at it. Turned on the server, raid controller was detected, but it didn't detect the two SAS drives.

    Checking further it doesn't seem the drives are spinning up. So far I've used two different sets of cables and a convertor block. None of those seem to work. So either both drives are DOA, there's a problem with the power supply (not likely since currently the server is running 11 other hard drives both SAS and SATA), or these Seagate drives need something else entirely.

    First thing I tried was a simple adapter block:
    5Gbps SFF 8482 SAS to SATA 180 Degree Angle Adapter Converter Straight Head
    Spoiler!


    Then I tried a dedicted cable thinking perhaps the drives weren't detected because I mixed SAS and SATA on 1 SFF8087 connector (which splits to 4 SATA connectors). This is the cable:
    StarTech. com sas808782p50 Intern Serial-attached SCSI Mini SAS kabels – SF-8087 op 4 x SFF-8482 50 cm
    Spoiler!


    When that didn't work either I tried another raid controller with it's own cable. This one:
    CableDeconn SFF-8643 interne Mini SAS HD auf (4) 29pin SFF-8482 Stecker mit SAS 15pol Power Port 12 GB/s Kabel 1M
    Spoiler!


    Perhaps it's the SATA power to 15-pin SAS power that's causing the problems. Not sure why it would since it's pretty straightforward. Remember we're already running four 300GB SAS drives without problems. But since the new drives aren't spinning up I'm leaning in that direction right now.

    If anyone has any bright ideas, I'm all ears :B

  5. #15

  6. #16
    Svert linked me a forum thread where someone experienced something similar. In the end the solution was to use silver cables forged by elves in moonlight. So I've ordered another set of cables:
    Adaptec ACK-I-HDmSAS-4SAS-SB-.8M
    Spoiler!

    We'll see if that makes any difference.

  7. #17
    The new cable works!

    I plugged it in just now. The drives spun up and are recognized by the raid controller. I'll try doing the raid setup and VM stuff tonight so we can get the TFC servers up and running.

  8. #18
    Moderator Elbe97's Avatar
    Join Date
    Apr 2015
    Location
    Prague, Czech Republic
    Posts
    261
    So the silver cable forged by Elves in moonlight worked Awesome! Good news for TFC
    My common IGNs: Elbe97 / Elborax / Elbor

    THe FouR HoRSeMeN oF THe aPoCaLYPSe:

  9. #19
    Quote Originally Posted by Jiro_89 View Post
    EUR 698.58 + unlisted amounts in cables and other parts for maintenance. More often than not J is backing the majority, if not all of the costs. We appreciate all of our donors that help
    Well, it looks like Jiro is planning on doing something similar. Not sure if I'm speaking out of turn here. Jiro, if you don't like me talking about it just yet, just remove my post

    Anyway. The ideas that are floating around are something like this:
    Jiro is going to buy the current server's motherboard + CPU + CPU cooler + 64GB RAM + two 1.5TB hard drives from me. With those funds I'm going to upgrade our current server to an AMD Threadripper system. Then the parts are going to move around the world and will be placed into a machine that Jiro is going to host in the US. This effectively means we'll be moving the ARK servers to the US where most of our players are from (I think). While we're not getting more RAM for our servers overall right away, this does allow room for growth.

    Another plan floating around is that Sverf is going to be buying a new motherboard + CPU for his server that currently hosts the vanilla Minecraft servers. I'll be donating 32GB of RAM, a 24-port raid controller + cables and two 3TB drives for that build.

    The Jiro-plan is going to cost around 750 euro. Sverf will be spending around 350 euro. So if you're feeling generous you know where the donate button is

  10. #20
    Quote Originally Posted by InsaneJ View Post
    In this thread I'll try to keep track of what goes on with the server hardware wise.

    The last publicly documented server change was this: [completed] New server plans (These are no longer 'new' plans)
    We upgraded the server to a 6-core/12-thread Intel Core i7 5820K CPU with 64GB of RAM.

    Unfortunately that setup gave us trouble running VMWare ESXi which resulted in an unstable server. It took a few weeks to track down the exact cause. The problem was with the CPU. If you ever do any PC building: it's almost never the CPU that's causing stability issues. Unless you do overclocking. But that isn't the case here.

    I decided to upgrade the CPU to a 14-core/28-thread Intel Xeon E5 2680 v4 CPU with 96GB RAM after that. This is what we are currently running all our servers on. That took care of the stability issues.

    Then the next issue was with server performance. Or to be more precise: disk performance. The servers are running of two 7200 rpm SATA drives. And even though they are connected to an Areca 1680i raid controller with 4GB cache and a dual core 1.2GHz PowerPC cpu, it's not enough to run all the additional servers we're now running. It used to be:
    • website
    • email
    • Minecraft, 10 or so instances.

    And now we added:
    • ARK Survival Evolved, 3 modded instances.

    Disk I/O was lagging behind and that caused some noticeable performance issues.

    So I purchased an Icy dock Tough Armor 4 x 2.5" mobile rack for 1 x 5.25" device bay. In this dock, I have placed three second hand 300GB 10K rpm SAS drives. These disks are meant to offset the disk I/O that's been hammering the OS drives. I also swapped out the four 80GB Intel Postville SSDs and replaced those with two new Samsung 850 Pro 256GB SSDs.

    The additional drives worked well for a few weeks. Unfortunately this morning at around 5:02AM one of those three drives failed. This is not a big deal since they were running in raid-5. However it does mean I now have to move the virtual machines that were running on those drives back to the other disks they were on before. Which means we may experience some slow downs in the time to come.

    The 10K rpm SAS drives were second hand with no warranty so the faulty drive will have to be replaced by buying another. We just had a beautiful baby girl and with all the stuff we need for that I'm not allowed (haha ) to spend more money on my hobby. So if anyone wants to help out by donating (see the front page). Those SAS drives cost about $55 each:
    HP 300GB 6G SAS 10K 2.5 inch

    At any rate the servers will continue to run. The ARK servers are running on two Samsung 850 PRO SSDs. It's just everything else that will get a performance hit now that they have to share the slower storage.
    Hi InsaneJ,
    I have a small job as web developper and realised I got a not in use 860 evo 256gb laying around, which I can sell to you for cheap. If i'm right you live in the netherlands aswell. I would also suggest to run the website on a cheap hosting service to make some space on the ssd's for ark and mc. Website hosting is really cheap atm. I can also host something for you for free if youre intrested. So contact me if you want anything,
    Greetings Bram.
    Last edited by bram_dc; 23rd October 2018 at 00:36.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •