Jump to content

Trent

Member
  • Posts

    171
  • Joined

  • Last visited

  • Days Won

    30

Posts posted by Trent

  1. Now that we're back up and should be be stable, I'll talk about the issues that happened this week.

     

    Web Proxy

    The issues started last weekend when we started seeing the web panel have issues loading. We saw the web proxy lagging and worked to address that, but it was unclear what the underlying problem was. In the middle of the week, we were able to make a fix that helped the web proxy perform normally, but that caused the issue to go one step deeper into the Minehut system.

     

    APIs

    After the web proxy stopped being the point of pain, the issue presented itself in the APIs. Both the network and web APIs are critical to Minehut running and handle internal communication between Minehut services. We saw these services timing out, which meant that the web panel and game servers were having issues, as well as not being able to login to Minehut and join the lobby. We spent 2 days debugging bottle necks here and found a few pain points, which we patched yesterday.

     

    Redis

    Once the API issues were cleared up, we got to the root of the problem. Redis is a cache that stores a lot of data, and is where our APIs retrieve data when responding with server status, MOTD, and how many servers are online. We found 2 issues with this that combined were catastrophic:

    1. Redis is a cache and isn't meant to be persistent data, but none of the data in redis was ever clearing.
    2. An endpoint that hits redis would pull every object in redis and iterate over it, which is not how a key value store is supposed to work.

    rediscount2.PNG

    Graph of objects in redis. This should be a flat line with old objects expiring as new ones are being created.

    Redis Fix

    The above means that there were millions of objects in redis that weren't being cleared, and were being calculated every time an endpoint was called. This isn't something new, and has been building up for awhile. This morning we saw it hit the tipping point of multiple services that rely on redis to all fail. It took us a little bit to track down the issue, and then we attempted to start redis on a more powerful box and flush it without doing a full network restart. That helped resolve the problem, but left us in a broken state as our cache doesn't rebuild itself safely in that case. It took a couple of restarts to get the new redis configured correctly and in a healthy state, and now we're finally there.

     

    Other Things

    • Fixed ram not allocating correctly when upgrading player slots.

     

    It was a rough week, but we should be out of it for a bit. The issues that we did today will just delay the problem again for awhile, which is why we're working on rebuilding pieces of Minehut to allow us to continue to scale to 4000+ players.

  2. 8 minutes ago, Gonzalo506 said:

    If i change to 10 slots and then buy 20 slots for 50c/d, will i have 2gb of ram or just 1gb of ram? That got me confused

    20 slots isn't a purchasable plan. It goes 15 -> 25 if you're getting them with credits. The plans are listed in the top post.

  3. 12 hours ago, Splatez07 said:

    😥 may have to switch to a paid host because of this 😥

    I mean you can still upgrade to 2gb with us if you'd like. 2gb is a lot to be giving away for free and you can't find that anywhere else.

     

    16 minutes ago, EdyMCcraft said:

    Nah, switching to a paid host.The server dies to like 50 blocks placed.Or 30 blocks of a block that has gravity..

    Nah, byne Minehut, had a good time..

    9 hours ago, CookizSkript_ said:

    Whyyyy?

    My server now dies from 50 blocks moved

    We added a ton of nodes to our backend a few minutes ago to help spread out the load more. Should help with lag in game.

     

    6 hours ago, Guest not exacty said:

    i dont like how the player slots got cut down..

     

    They weren't. You still have 20 if your server was up before this post.

     

     

  4. 11 minutes ago, Merfy said:

    Is there like a list of the 117,907  that keeps their 20 slots?

     

    24 minutes ago, Trent said:

    If your server was started after the 20 slot change went into effect on May 20th and before this post was made, your server will have 20 slots, but 1gb of ram.

     

  5. Crazy Growth

    Over the last few months, we've grown at an unexpected rate. Take a look at the milestones below for when we broke our concurrent player records:

    Spoiler

     

    1057 players - June 10, 2018

    1208 players - January 12, 2019

    1257 players - January 13, 2019

    1275 players - February 25, 2019

    1326 players - March 10, 2019

    1353 players - March 16, 2019

    1441 players - March 17, 2019

    1516 players - April 13, 2019

    1736 players - April 14, 2019

    2063 players - May 4, 2019

    2271 players - May 5, 2019

    2469 players - May 24, 2019

    3231 players - May 25, 2019

    3374 players - May 26, 2019

    3593 players - June 2, 2019

    3761 players - June 9, 2019

    3876 players - June 23, 2019

     

    A year ago when SLG took over Minehut, there were 100k registered users and 1100 players. Now, we regularly have 3500 players on during the week and we broke 400k registered users last week. Even in the last month since we increased the default ram to 2gb and slots to 20, we went from 2500 concurrent users to 3500! That being said, we have to make some changes to make sure we have room to continue to grow.

     

    Today's Changes

    • Our free plan is back to 1gb of ram and 10 player slots
    • If your server was started after the 20 slot change went into effect on May 20th and before this post was made, your server will have 20 slots, but 1gb of ram.
      • If you change back to 10 slots on the server.properties, you won't have the option to go back to 20 for free, and there's no way to go back from there.

    The process for flagging servers to have 20 slots was automatic and was applied to 117,907 servers. 

     

    Updated Plans

    image.png

     

    Web Panel Issues

    This is another piece of Minehut that is hitting limits with how many players we have. We need to make some changes to let it run more smoothly at this high capacity. A temporary patch was applied to try to buy us some time, but we're working on doing a proper fix asap.

     

     

    • Like 3
    • Thanks 3
    • Confused 1
    • Sad 2
  6. 1 hour ago, Jah.Stuul said:

    Is there a way to play on a server on 1.14.2 for optifine? Every time I try to play with optifine on the server it takes me to the lobby>

    You can install ProtocolSupport to allow older versions to join your server.

  7. This combat update announced today is pretty crazy. Can't wait to see how it changes the PVP community and if it gets more competitive again. And for when we bring it to Minehut and half the people are like yay and the other half think we ruined their server.

     

     

  8. 5 hours ago, ORASDemo5 said:

    How can I have a server running 1.13?

    If you install the plugin ProtocolSupport, you can allow versions back to 1.8 to connect.

     

    4 hours ago, Swagalich1 said:

    What do you mean by "simply restart"? Do you mean delete and restart the whole world? If not then how can I make my 1.14.2 server into a 1.14.3 server? Thanks. 🙂

    We didn't shut down servers to make this change, so any servers that were running when we did, were still running on Spigot 1.14.2. If you wanted to get Spigot 1.14.3, you just had to restart your server.

  9.  

    Check out that thread for info. We run default Spigot and Bungeecord. Spigot is a fork of Craftbukkit, which is an implementation of the vanilla server that Bukkit, the API, can hook into so plugins can work. Spigot is run by md_5, who used to be a moderator in the Bukkit community until he was fired for being inactive (because he started Spigot while Bukkit was still a thing). The Bukkit community no longer updates Craftbukkit since some copyright issues a few years ago. Many of the people that started Bukkit went on to work at Mojang, such as Dinnerbone, Grum, and EvilSeph (no longer there). Check out https://dinnerbone.com/blog/2013/01/06/story-bone-and-bukkit/

    • Like 2
  10.  

    Minehut is now on 1.14.3. The process for this update was to just restart lobbies and proxies, which kicks all players but doesn't turn off player servers. For your server to run on 1.14.3, it will need to simply restart. Until then, your server might still be on 1.14.2.

     

    Issues

    If you're noticing any issues with plugins or your server, please report them here https://forums.minehut.com/forum/11-community-support/

     

    Technical Information

    Spigot built as of https://hub.spigotmc.org/stash/projects/SPIGOT/repos/craftbukkit/commits/e8b39d430a77c2e53656b4c7bc2ed4b4a2b48a00

    Protocol Support https://github.com/ProtocolSupport/ProtocolSupport/pull/1118

    Bungee as of https://github.com/SpigotMC/BungeeCord/commit/941d7f7262979d7cc402b62f65c57bd1294ec6d6

  11. Hey, we'll be updating to 1.14.3 soon. We're waiting on a few third party programs we use to update.

     

    • Spigot updated
    • Bungeecord updated
    • ProtocolSupport hasn't updated yet (lets previous versions join)

    Once ProtocolSupport updates, we'll update all of Minehut to 1.14.3 and support versions back to 1.8. If we updated now, we would only support 1.14.3 and no previous versions, so we're going to wait a little bit 🙂

    • Like 2
  12. Screen Shot 2019-06-19 at 09.56.29.png

     

    Your eyes can relax, dark theme is now available! This theme was developed by Super League's design team built on top of the popular Borx theme. Please let us know if you find any issues or have any suggestions and post them over in the suggestions category. To enable, scroll to the bottom of any page, click the themes dropdown, and toggle between light and dark.

     

    Spoiler

    Screen Shot 2019-06-19 at 09.48.11.png

     

     

    Auto Reply to new DM

    All users can now setup auto replies to direct messages. You can enable this on the account settings page under "Auto Reply to PMs". This will send a response to the first message of a new conversation from any member. See the below spoiler for how I have mine setup.

     

    Spoiler

    Screen Shot 2019-06-19 at 09.56.55.png

     

    • Like 1
  13. This is a more technical update, but I'm sure some people will be interested in this 🙂

     

    We're now running PaperSpigot again instead of regular Spigot. It's a "high performance" fork of Spigot (which is a high performance fork of Craftbukkit) that helps make servers run a bit more smoothly. You can check out their documentation at the link below.

    Upstream projecthttps://papermc.io/

    Our forkhttps://github.com/Minehut/Paper/tree/ver/1.14

     

    The only change we made to Paper was to rotate log files. This means that once your logs/latest.log gets to 10mb, it will be compressed and renamed to log<timestamp>.tar.gz. There's a limit to 7 .gz files to be rotated out. Once you hit 8, the oldest will be deleted. This will prevent servers that spam logs from automatically hibernating.

     

    This will also fix ProtocolSupportStuff not functioning correctly, as well as a few other plugins such as ProtocolLib that hook directly into PaperSpigot.

     

    EDIT: Paper was causing issues with not telling the panel the server was done starting (even though it was). Until we can fix this, we're back on Spigot 1.14.2. I rebuilt from BuildTools so we have the latest fixes.

    • Like 3
×
×
  • Create New...