An Update on Recent Service Issues

[Trent]

Hey everyone, we wanted to provide a detailed update on what has happened over the last few days. This doesn't come just from me, this comes from everyone here at Super League. 

 

Incident Overview 

We strive to provide you with the best free Minecraft server network in the world. We shoot for excellence, and it hurts when we fall short, and over the past 36 hours, we’ve fallen short. First, we want to apologize to everyone who has been affected in any way due to the several compounding issues yesterday and today – We are deeply sorry and are hard at work to resolve them and keep them from happening again. I'll explain what happened and what we've done to prevent it from happening again in the future.

Yesterday (Wednesday) afternoon we noticed our file servers (where we store servers) started experiencing issues and were overloaded with requests. This caused some servers to be unable to complete their normal save operations resulting in some servers not saving successfully, and losing progress. Once we pinned down the file server software as the root cause of the issue, we reverted back to the last stable version which put us back in a stable state.  

Today (Thursday), we first noticed an issue this afternoon that our server despinner (the process that shuts down servers when they are idle) was errored out. This resulted in no servers saving / stopping, but users were still able to start their servers. We very quickly reached our max server cap and then no new servers were able to start. This is where you can see the plateau in online servers but a steady decline in players in the below graph. 
 

Player count in dark blue, server count in cyan both graphed by the second 
 

After we patched the server despinner and attempted to come back online, (when the graph starts to increase in both players and servers again), we noticed another issue with our file server reporting it was overloaded. In an attempt to prevent any data from being lost like we knew happened last night, we prevented any new servers from starting as well as turned back off the server despinner process, which meant no servers would save and fail because of issues with the file server. We did some updates to help the file server, and then slowly let servers save while we didn't let any new ones start. During this, we made the call to completely wipe our existing storage server and go with a backup. 

Last August, we migrated hosts. We made the call during the migration to save servers both on our new host but also to our old host as a backup incase we had issues post migration. When a server starts, it checks to see if it's in our new file server. If not, it checks if it's in the old file server, if not, it creates a new server from our template. Luckily, we never turned this off. This means that every time a server shuts down, it saves both to our old host and to our new host. We also take a nightly backup of both hosts, with the most recent backup taken at 230AM EST. This allowed us to make the decision to wipe our current file server of all data since some recent server saves to our new file server were corrupted but were saved properly to our old file server.
 

Data Loss / Backup Info 

Now that we're back online, this should mean that servers that failed to save yesterday (no way to know if this was you) but hadn't yet started again will not see any data loss when they come online next time, which will reduce the number of servers that lost data. This also means that anyone that noticed data lost today for the first time might have a backup available that we can try to restore. If you first noticed data loss last night (before approximately 230AM EST), it's highly unlikely we have a backup of your server

If you experienced data loss at any time over the last few days, please make sure to write into support to see if we can do anything to help even if it's unlikely we have a backup. Our support form is at https://minehut.com/support/form

 

New Patch 

While we were offline today, we wrote a patch that updated the server shutdown process to not give up if it fails. It will now always wait for the server to successfully save. If this patch would've been in place before yesterday, no one would've experienced data loss, but we still would have had issues with servers starting and stopping until we fixed the other file server issue. This patch coupled with another one that we rolled out over winter break has drastically reduced the risk of data loss on servers in the future. 

All of us here at Minehut are sorry for the string of issues that have plagued your servers for the past day. We know Minecraft and the worlds you create within your server are a much-needed escape for many of you, and we are sorry that we haven't delivered on our service quality recently. For Minehut users who lost any data, please reach out to us immediately so we can work with you to determine if your data can be recovered and to work to regain your trust. Please contact us at https://minehut.com/support/form, allow additional time as we respond to all customer service inquires related to this issue.

 

Thank you for your patience and understanding.

- Super League Gaming, Inc.

   

 

[GoldBoi69]

cool

[tisdfdffg]

thanks trent:) we appreciate all you do 

[not_3than]

nice

[mybeary]

when next zomb

[tigero9]

We're glad you do everything you can!

Edited February 12, 2021 by tigero9

[assdsaas]

At least you fixed it early, instead of letting it become a bigger problem.

[mekbturtle]

maintenence bad

Edited February 12, 2021 by Migins

[CoolProgrammer]

@Trent Thank you so much for the efforts you put into letting us know the issue. We all appreciate your hardwork, keep it up.

[Rufen]

6 minutes ago, mybeary said:

when next zomb

inf

[mybeary]

Just now, Rufen said:

inf

why next zomb

 

[MikeTheAngel]

10 minutes ago, mybeary said:

why next zomb

 

inf

 

[RandomGuyInMask]

Stuffs like this happen, and it can't be helped. Sometimes, it's not even the staffs' fault, but there will always be folks who'll give you guys the flack for not meeting their expectations, which is to be expected. 

I'm pretty sure that you staffs have done your best to mitigate the problem and to keep it from turning worse. Although my words do not speak for everyone who use Minehut, I'd like to say thanks for tackling the issues with alacrity and being honest with us, if that means anything. 

[Centrect]

thanks for saving us yet again trent

[Guest Damie]

Thank you kind sir

[Tistay]

yes

[CuteCatsXXL]

very epic Trent! 

[Guest Appreciate It!!]

Just wanted to say thanks for all you and the entire team has done.... greatly appreciated.  My server was not effected with data loss but I totally understand what other people are going through.  **Always /dl world (world name)!!!!!!!**

[Tdawg]

9 hours ago, Trent said:

While we were offline today, we wrote a patch that updated the server shutdown process to not give up if it fails. It will now always wait for the server to successfully save. If this patch would've been in place before yesterday, no one would've experienced data loss, but we still would have had issues with servers starting and stopping until we fixed the other file server issue. This patch coupled with another one that we rolled out over winter break has drastically reduced the risk of data loss on servers in the future. 

 

You said on the discord on 10/20/20 that you did this same thing:
We can more easily retry failed uploads and better detect when a server is ready to be put into hibernation. We also now give the servers as much time as they need to save. Previously, servers were limited to 10 minutes for the entire save process to complete. Now, if you have a large server that takes a while to upload and compress, your server container will stay active until that process completes, even if it takes an hour or more. https://discordapp.com/channels/239599059415859200/240269653358805003/767974713141493761

What happened to that?

[Guest Help me pls]

hello my name in discord / minecraft is loloo777 and i can't log in and that always works when dan says the website says minehut login
Login to your Minehut account.

Hm ... looks like we couldn't log you in. And I did everything right help; DDD

[ZeDarkOne]

You don't have to say sorry people should know things break at some point but can be fixed.

[Mischa]

Hi,

This week I noticed that our survival server had lost some big progress in a specific big chunk where we built our structures and where we were mostly active. The terrain is completely reset to its original natural landscape from before we started building, while outside the chunk of lost progress there still exist remainders of the lost structures (these parts were completed later than the core structures), which are literally cut off from the the original chunk with the builds as seen on the screenshot added down below. The fences still look like they're attached to the other fences of the missing chunk. This seems weird as it doesn't seem like a matter of overall issues with the server safe file (I'd expect that the later builds would also not exist in that case), but rather a matter of issues with the chunks. Now I'd like to know if this was caused by the problem described up top? If so, could it somehow be fixed? I'd hate to see the progress being lost like that. If it definitly can't be fixed, then I too would like to know so. Then I know I can move on and start over elsewhere with what was left without wondering whether what was lost can be recovered. 

I hope the issue can be fixed. Thanks in advance. 

[Certy]

R.I.P Data Loss. Thanks for the update 

@Trent

[Guest Test]

Hiiii

[KeemanGaming]

My server was deleted and the last time I played was 2 days ago, 2/10/2021, If you could find a way to fix it I would be appreciate it, my server is: SMPRSTwo, thanks!