Page 2 of 2 FirstFirst 12
Results 11 to 17 of 17
  1. #11
    The timeouts still seem to be occurring.

  2. #12
    Just an updated to frequency: At the moment we've experienced 5 in the last 10 minutes. When the population goes over or around 30 the chance of the timeout seems to increase exponentially.

  3. #13
    Sverf
    Guest
    It seems purely related to amount of players on the server, more players, the more it happens.
    However the logs have no indication at all as to why. The Netty selector warnings do not happen before every mass DC and with the tweaks to the Linux kernel the xen-netfront issue from the first post is not the cause either. Im not sure where to look further at this time.

    One last crazy idea I have was to replace the netty class files bundled with the minecraft jar with a newer version that apparently fixes the netty select bug. No idea if this can be done however..

  4. #14
    Sverf
    Guest
    Browsing through last nights log (15:20 CET yesterday - 9:00 CET today) we had mass disonnects (more then 10 players dc'ing) at:

    15:28:04
    16:39:23
    18:12:51
    19:09:39
    19:16:07
    19:33:13
    19:39:40
    19:46:21
    19:52:32
    19:58:43
    20:04:24
    20:10:09
    20:15:56
    20:21:50
    20:27:57
    20:33:53
    20:45:15
    20:51:03
    20:56:50
    05:03:12
    05:09:19
    05:15:25
    07:13:10
    08:43:24

    Between 1900 and 2100 CET it was particularly bad.

  5. #15
    We're still running Java 7 right?
    Code:
    java version "1.7.0_71"
    Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
    Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)
    Perhaps we could try running Java 8 first? I ran a local instance of the server on my PC using Java 8. It starts up normally as far as I could tell.

  6. #16
    Sverf
    Guest
    We can switch the default java implementation per useraccount, so yeah, we sure can test the TFC server with java 8.

  7. #17
    To try to keep things running smoothly we are going to enforce some restrictions. These restrictions are on a per town basis. Meaning each town may have:
    • 4 mature animals of each kind. This means: 4 cows, 4 pigs, 4 horses, etc. All mature, not counting young animals.
    • 1 plot of berry bushes. This means 16x16 = 256 bushes. They may be in more than one plot, but 256 max.
    • 2 plots of tilled soil farmland. This means 512 blocks. They may be in more than two plots, but 512 max.
    • 1 of each kind of fruit tree. (red apple, green apple, banana, peach, etc.)
    • 1 of each kind of anvil. (copper, bronze, wrought iron, steel, etc.)

    Again, to be absolutely clear. This limits are per town. If your town is very large and isn't getting enough food, please let us know. Also if anyone needs help removing large amounts of tilled soil, fruit trees or berry bushes you may contact staff to help you.

    Now for some technical stuff explaining why and how we came up with the new restrictions.
    We have made quite a lot of progress since the last update. We used Warmroast, a Java profiler aimed towards Minecraft, to track down what is keeping the server occupied. It seems that during a mass disconnect the server is busy processing a lot of tile entities. The server has upward of 15.000 tile entities loaded during peak hours. In the screenshot below you can see that net.minecraft.world.WorldServer.func_72939_s() is using 82.79% CPU time during a mass disconnect on the server. It looks like it's going through a very large array and deleting items from it. If this takes too long (1200) it dies, which seems to be when the mass discoonect occurs.

    Unfortunately this is a vanilla Minecraft function. This means that only Forge could potentially create and optimization for this to better deal with large amounts of tile entities. But even if they did, it would mean a new version of Forge which we can't use since Cauldron, which our server runs on, isn't being maintained any longer. Cauldron is essentially Minecraft_server + Forge + Bukkit + magic glue code.

    So that leaves us with two possible options. Perhaps the TFC devs could do something to optimize server performance. Or we try to reduce the amount of tile entities on the server. During peak hours we've seen that the majority of tile entities are made up of farmland and fruit trees. Below is an example of this:


    Going through what the server is doing we discovered that about 10.000.000 blocks are formed on the server each hour. That's about 2700 blocks each second. These are most likely snow blocks and the like. However each of these blocks forming meant one more record in the Prism data base which logs everything that happens on the server. We have change the Prism configuration to no longer include these naturally forming blocks which aren't necessary for grief prevention and roll-back. This saves us a considerable database overhead.

    We have installed a mod that optimizes performance. It's quite a scary mod to run since it changes a lot of the internal workings of the server.This is the summary:
    • Ticks multiple worlds at the same time in different threads
    • Ticks multiple entities and tile entities within worlds at the same time in different threads
    • Performs movement updates in network threads, so players will move smoothly even if TPS is bad
    • Performs chunk loading and chunk generation asynchronously - no lag spikes from players logging in or teleporting
    • Improved collision code which can handle thousands of mobs in a single block space without terrible TPS. Will still break clients.
    • Many more small tweaks to minecraft's internals and other mods, both to improve performance and ensure that multithreading doesn't break everything.
    Attached Images Attached Images   

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •