Trent

Now that we're back up and should be be stable, I'll talk about the issues that happened this week. Web Proxy The issues started last weekend when we started seeing the web panel have issues loading. We saw the web proxy lagging and worked to address that, but it was unclear what the underlying problem was. In the middle of the week, we were able to make a fix that helped the web proxy perform normally, but that caused the issue to go one step deeper into the Minehut system. APIs After the web proxy stopped being the point of pain, the issue presented itself in the APIs. Both the network and web APIs are critical to Minehut running and handle internal communication between Minehut services. We saw these services timing out, which meant that the web panel and game servers were having issues, as well as not being able to login to Minehut and join the lobby. We spent 2 days debugging bottle necks here and found a few pain points, which we patched yesterday. Redis Once the API issues were cleared up, we got to the root of the problem. Redis is a cache that stores a lot of data, and is where our APIs retrieve data when responding with server status, MOTD, and how many servers are online. We found 2 issues with this that combined were catastrophic: Redis is a cache and isn't meant to be persistent data, but none of the data in redis was ever clearing. An endpoint that hits redis would pull every object in redis and iterate over it, which is not how a key value store is supposed to work. Graph of objects in redis. This should be a flat line with old objects expiring as new ones are being created. Redis Fix The above means that there were millions of objects in redis that weren't being cleared, and were being calculated every time an endpoint was called. This isn't something new, and has been building up for awhile. This morning we saw it hit the tipping point of multiple services that rely on redis to all fail. It took us a little bit to track down the issue, and then we attempted to start redis on a more powerful box and flush it without doing a full network restart. That helped resolve the problem, but left us in a broken state as our cache doesn't rebuild itself safely in that case. It took a couple of restarts to get the new redis configured correctly and in a healthy state, and now we're finally there. Other Things Fixed ram not allocating correctly when upgrading player slots. It was a rough week, but we should be out of it for a bit. The issues that we did today will just delay the problem again for awhile, which is why we're working on rebuilding pieces of Minehut to allow us to continue to scale to 4000+ players.

20 slots isn't a purchasable plan. It goes 15 -> 25 if you're getting them with credits. The plans are listed in the top post.

Startup issues should be better. We had host nodes a bit overloaded this morning, so we added a bunch more and are seeing CPU usage much healthier.

I mean you can still upgrade to 2gb with us if you'd like. 2gb is a lot to be giving away for free and you can't find that anywhere else. We added a ton of nodes to our backend a few minutes ago to help spread out the load more. Should help with lag in game. They weren't. You still have 20 if your server was up before this post.

Crazy Growth Over the last few months, we've grown at an unexpected rate. Take a look at the milestones below for when we broke our concurrent player records: A year ago when SLG took over Minehut, there were 100k registered users and 1100 players. Now, we regularly have 3500 players on during the week and we broke 400k registered users last week. Even in the last month since we increased the default ram to 2gb and slots to 20, we went from 2500 concurrent users to 3500! That being said, we have to make some changes to make sure we have room to continue to grow. Today's Changes Our free plan is back to 1gb of ram and 10 player slots If your server was started after the 20 slot change went into effect on May 20th and before this post was made, your server will have 20 slots, but 1gb of ram. If you change back to 10 slots on the server.properties, you won't have the option to go back to 20 for free, and there's no way to go back from there. The process for flagging servers to have 20 slots was automatic and was applied to 117,907 servers. Updated Plans Web Panel Issues This is another piece of Minehut that is hitting limits with how many players we have. We need to make some changes to let it run more smoothly at this high capacity. A temporary patch was applied to try to buy us some time, but we're working on doing a proper fix asap.

You can install ProtocolSupport to allow older versions to join your server.

I worry about how this will change what hacked clients might be able to do, as well as how hard it will be able to catch people with reach hacks.

This combat update announced today is pretty crazy. Can't wait to see how it changes the PVP community and if it gets more competitive again. And for when we bring it to Minehut and half the people are like yay and the other half think we ruined their server.

If you install the plugin ProtocolSupport, you can allow versions back to 1.8 to connect. We didn't shut down servers to make this change, so any servers that were running when we did, were still running on Spigot 1.14.2. If you wanted to get Spigot 1.14.3, you just had to restart your server.

Check out that thread for info. We run default Spigot and Bungeecord. Spigot is a fork of Craftbukkit, which is an implementation of the vanilla server that Bukkit, the API, can hook into so plugins can work. Spigot is run by md_5, who used to be a moderator in the Bukkit community until he was fired for being inactive (because he started Spigot while Bukkit was still a thing). The Bukkit community no longer updates Craftbukkit since some copyright issues a few years ago. Many of the people that started Bukkit went on to work at Mojang, such as Dinnerbone, Grum, and EvilSeph (no longer there). Check out https://dinnerbone.com/blog/2013/01/06/story-bone-and-bukkit/

Minehut is now on 1.14.3. The process for this update was to just restart lobbies and proxies, which kicks all players but doesn't turn off player servers. For your server to run on 1.14.3, it will need to simply restart. Until then, your server might still be on 1.14.2. Issues If you're noticing any issues with plugins or your server, please report them here https://forums.minehut.com/forum/11-community-support/ Technical Information Spigot built as of https://hub.spigotmc.org/stash/projects/SPIGOT/repos/craftbukkit/commits/e8b39d430a77c2e53656b4c7bc2ed4b4a2b48a00 Protocol Support https://github.com/ProtocolSupport/ProtocolSupport/pull/1118 Bungee as of https://github.com/SpigotMC/BungeeCord/commit/941d7f7262979d7cc402b62f65c57bd1294ec6d6

Hey, we'll be updating to 1.14.3 soon. We're waiting on a few third party programs we use to update. Spigot updated Bungeecord updated ProtocolSupport hasn't updated yet (lets previous versions join) Once ProtocolSupport updates, we'll update all of Minehut to 1.14.3 and support versions back to 1.8. If we updated now, we would only support 1.14.3 and no previous versions, so we're going to wait a little bit

That doesn't trump SECURITY REASONS

He said no :c

I reached out to the author.

Your eyes can relax, dark theme is now available! This theme was developed by Super League's design team built on top of the popular Borx theme. Please let us know if you find any issues or have any suggestions and post them over in the suggestions category. To enable, scroll to the bottom of any page, click the themes dropdown, and toggle between light and dark. Auto Reply to new DM All users can now setup auto replies to direct messages. You can enable this on the account settings page under "Auto Reply to PMs". This will send a response to the first message of a new conversation from any member. See the below spoiler for how I have mine setup.

This is a more technical update, but I'm sure some people will be interested in this We're now running PaperSpigot again instead of regular Spigot. It's a "high performance" fork of Spigot (which is a high performance fork of Craftbukkit) that helps make servers run a bit more smoothly. You can check out their documentation at the link below. Upstream project: https://papermc.io/ Our fork: https://github.com/Minehut/Paper/tree/ver/1.14 The only change we made to Paper was to rotate log files. This means that once your logs/latest.log gets to 10mb, it will be compressed and renamed to log<timestamp>.tar.gz. There's a limit to 7 .gz files to be rotated out. Once you hit 8, the oldest will be deleted. This will prevent servers that spam logs from automatically hibernating. This will also fix ProtocolSupportStuff not functioning correctly, as well as a few other plugins such as ProtocolLib that hook directly into PaperSpigot. EDIT: Paper was causing issues with not telling the panel the server was done starting (even though it was). Until we can fix this, we're back on Spigot 1.14.2. I rebuilt from BuildTools so we have the latest fixes.

It's disabled for security reasons for the foreseeable future.

You'll need to OP yourself https://superleague.zendesk.com/hc/en-us/articles/360015767354-How-do-I-OP-myself-

This one is awesome. I love how much you can do in so few lines.

What about how to store kills? I feel like this script is tough without its counterpart.

What about the ability to change the lore too?

Check out the FAQ https://superleague.zendesk.com/hc/en-us/articles/360015767054-Downloading-and-uploading-survival-guide and if you still have issues, post what they are here and multiple people will attempt to help