Results 1 to 10 of 10
  1. #1

    Exclamation TFC servers down!

    All the TFC servers and Dynmaps have been down for at least 15 mins.

    I could play fine this morning so its probably not a problem on my side.

  2. #2
    Cake! InsaneJ's Avatar
    Join Date
    Jan 2012
    Location
    Cakeville
    Posts
    4,685
    Blog Entries
    21
    Servers are back.

  3. #3
    Servers are down again.
    Is there something going on? The TFC game servers have been down quite a lot lately.

  4. #4
    Cake! InsaneJ's Avatar
    Join Date
    Jan 2012
    Location
    Cakeville
    Posts
    4,685
    Blog Entries
    21
    We've had some server troubles the past week or so. I've been trying to track it down but so far it hasn't been easy.

    Currently I have replaced the power supply, motherboard, CPU and graphics card all without any significant change. The server still crashes or freezes from time to time.

    I should receive new RAM today or maybe tomorrow. If that doesn't resolve the issue then there are only two more options I can think of:
    • the raid controller
    • OS and/or software


    In case of the raid controller, that's a pain. It's an 800 euro controller. So if that's the problem then we're going to have to change the structure of our servers to use a different kind of disk redundancy. Also we'll be switching from Xen virtualization to VMWare since it's easier to maintain (for me at least ). I can convert the Xen virtual machines to VMWare virtual machines. It's not hard to do it just takes time. Right now we have 3 virtual machines running. One for the website/forums, one for the email server and one for the game servers. The reason we separate those is so we can assign a certain amount of resources to each virtual machine. This means that if the website gets overloaded for whatever reason, it shouldn't affect the game servers much. Also if one of the servers is hacked it will make it more difficult to also get to the other servers, at least we hope it does. Which depending on the hack you can never know beforehand really.

    If it's the OS and/or software then we need to reinstall everything. This is the worst case scenario really but also rather unlikely at this point. If it does come down to this then it'll be a while before we have everything back up.

    Right now I'm still hoping changing the RAM will solve the issue. Although when I ran a memtest a couple of days ago it didn't find any problems there. So yeah, it might just end up being the raid controller and costing us a ton of work. But those are the pleasures of hosting things yourself and not having to play a ton of money for it. Although replacing hardware isn't cheap either

    Oh, btw. I've restarted the servers. They should be up momentarily.

  5. #5
    quick note. when the ark servers go down i typically cannot access the website/forums either. has happened the last few times the server crashed. i could not get on to report it.

  6. #6
    Cake! InsaneJ's Avatar
    Join Date
    Jan 2012
    Location
    Cakeville
    Posts
    4,685
    Blog Entries
    21
    This is what leads us to believe it's a hardware related issue. If it were software only the game server should crash and not the web server. Unless the issue is with Xen...

    The real difficulty is that we haven't found any leads to what's causing this. Normally if there's a hardware problem it's fairly easy to reproduce. But so far we haven't been able to do that. So the only option we have at this point is to just replace everything and see if the crashes stop.

  7. #7
    Once you've swapped out all of the hardware...you'll have enough parts to build another server.

  8. #8
    Cake! InsaneJ's Avatar
    Join Date
    Jan 2012
    Location
    Cakeville
    Posts
    4,685
    Blog Entries
    21
    Yes, a second server I intend to sell

    While replacing the server RAM a few minutes ago I noticed something. The battery backup module of the raid controller had bulged. This can happen when batteries are starting to fail.

    As you can see, the green side of the battery is curved and it no longer fits entirely inside it's plastic bracket.
    Name:  potato quality.jpg
Views: 41
Size:  147.6 KB

    While the raid controller doesn't mention any battery problems. This one is next on my list if the new RAM doesn't change anything.

    Just FYI. This particular battery backup module goes for aroun $130,- That's one crazy expensive 3.7v battery. Perhaps I should get one from a Duracell power pack

  9. #9
    Minecraft Addict DOM's Avatar
    Join Date
    Nov 2011
    Location
    KSC
    Posts
    765
    Blog Entries
    28
    Quote Originally Posted by InsaneJ View Post
    This particular battery backup module goes for aroun $130
    Holy cow! Is it made from solid gold? I would just fit a voltage regulator on a 9V if that's the case.

  10. #10

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •