Server Technical Issues Tonight

A very quick post just to explain why the server was crashing a lot tonight, as it was something that I was quickly able to spot and fix, and is something that I had not expected, and thought the monitoring should have picked up but did not.

In short, the /home partition on the CentOS host had reached 100% capacity. The server runs in /home and at some point tonight the /home partition hit 100% usage.

The obvious question is why didn't out monitoring pick it up, after all I've made a bit of a song and dance about it being so good. And in short it's because the server still had a lot of disk space available, however it was on a different partition, so we had 50GB "Free" but not that the MC Server could write to.

What I've done for now is delete a few old files (Well, download them and delete them from the server) and this has included the hub world map, which as far as I'm aware was not being used at all, so that world will now be totally blank.

Longer term one option is to simply expand the partition, but to be honest that's not something I'd like to do if I can avoid it, as it can cause problems, and is realistically going to be a few hours outage to back everything up before starting, but might be what's required. I'm hoping I've bought us a little bit of time before this becomes a major issue again.

In terms of world sizes, this is the current stats:

Flatlands - 29.8GB
HubWorld (Pre-Downloaded) - 1.55GB
MasterBuilderWorld - 4.13GB
PlotWorld - 18.5GB
StaffWorld - 3.3GB
World - 19.6GB
World_nether - 5.33GB
world_the_end - 4.71GB

I don't want to be wiping worlds at the moment unless there is no other choice, especially as I know when I suggested having a rolling main world people weren't so keen on it, but it's just to give everyone an idea on the current scale and why this is such an issue as we continue to grow.

I'm hoping in the longer term some of these worlds will be split out into their own servers, which will help with the issue but it is just something to be aware of.

Apologies again for the instability tonight, I'm hoping I've bought us a few days at least to start properly sorting this, but I will keep a close eye on things over the coming days.

@DragonSlayer2189#2266 yeah I had forgot Linux partitions itself this way. I assumed 150gb disk would be enough but didn't account for it to partition chunks of it away from the home mount.

And none of that data is for anything other than the world files themselves.

Core protect currently for everything is taking up around 24gb since I took over hosting the server.

@DragonSlayer2189#2274 the core protect data is less of a concern tbh. And I'd rather only be deleting data when we wipe those worlds or parts of the world's.

As for the maps the answer is to purge the maps properly and more often...

Longer term I've got some solutions as I'd like to be splitting things like the plot world and staff worlds to be their own distinct minecraft servers but thats a little way off. Shorter term I will need to see how quickly the server files are growing again and see if I need to expand the disk or if I can wait it out.