PDA

View Full Version : TFC servers down!



CorpseDrop
8th July 2016, 05:39
All the TFC servers and Dynmaps have been down for at least 15 mins.

I could play fine this morning so its probably not a problem on my side.

InsaneJ
8th July 2016, 08:28
Servers are back.

CorpseDrop
15th July 2016, 02:33
Servers are down again.
Is there something going on? The TFC game servers have been down quite a lot lately.

InsaneJ
15th July 2016, 08:42
We've had some server troubles the past week or so. I've been trying to track it down but so far it hasn't been easy.

Currently I have replaced the power supply, motherboard, CPU and graphics card all without any significant change. The server still crashes or freezes from time to time.

I should receive new RAM today or maybe tomorrow. If that doesn't resolve the issue then there are only two more options I can think of:

the raid controller
OS and/or software


In case of the raid controller, that's a pain. It's an 800 euro controller. So if that's the problem then we're going to have to change the structure of our servers to use a different kind of disk redundancy. Also we'll be switching from Xen virtualization to VMWare since it's easier to maintain (for me at least ;)). I can convert the Xen virtual machines to VMWare virtual machines. It's not hard to do it just takes time. Right now we have 3 virtual machines running. One for the website/forums, one for the email server and one for the game servers. The reason we separate those is so we can assign a certain amount of resources to each virtual machine. This means that if the website gets overloaded for whatever reason, it shouldn't affect the game servers much. Also if one of the servers is hacked it will make it more difficult to also get to the other servers, at least we hope it does. Which depending on the hack you can never know beforehand really.

If it's the OS and/or software then we need to reinstall everything. This is the worst case scenario really but also rather unlikely at this point. If it does come down to this then it'll be a while before we have everything back up.

Right now I'm still hoping changing the RAM will solve the issue. Although when I ran a memtest a couple of days ago it didn't find any problems there. So yeah, it might just end up being the raid controller and costing us a ton of work. But those are the pleasures of hosting things yourself and not having to play a ton of money for it. Although replacing hardware isn't cheap either ;)

Oh, btw. I've restarted the servers. They should be up momentarily.

Dourne
15th July 2016, 08:59
quick note. when the ark servers go down i typically cannot access the website/forums either. has happened the last few times the server crashed. i could not get on to report it.

InsaneJ
15th July 2016, 09:05
This is what leads us to believe it's a hardware related issue. If it were software only the game server should crash and not the web server. Unless the issue is with Xen...

The real difficulty is that we haven't found any leads to what's causing this. Normally if there's a hardware problem it's fairly easy to reproduce. But so far we haven't been able to do that. So the only option we have at this point is to just replace everything and see if the crashes stop.

TheRealPoker
15th July 2016, 16:48
Once you've swapped out all of the hardware...you'll have enough parts to build another server.:cool:

InsaneJ
15th July 2016, 20:02
Yes, a second server I intend to sell :)

While replacing the server RAM a few minutes ago I noticed something. The battery backup module of the raid controller had bulged. This can happen when batteries are starting to fail.

As you can see, the green side of the battery is curved and it no longer fits entirely inside it's plastic bracket.
3017

While the raid controller doesn't mention any battery problems. This one is next on my list if the new RAM doesn't change anything.

Just FYI. This particular battery backup module goes for aroun $130,- That's one crazy expensive 3.7v battery. Perhaps I should get one from a Duracell power pack (https://hardforum.com/threads/arc-6120-replacement-battery.1846962/) ;)

DOM
15th July 2016, 23:37
This particular battery backup module goes for aroun $130

Holy cow! Is it made from solid gold? I would just fit a voltage regulator on a 9V if that's the case.

CorpseDrop
16th July 2016, 02:21
Down again :(