TotalFreedom Network Temporary Outage

  • Current eta for restore is going to be the end of next weekend. Unfortunately I've only just got home and have commitments this week and the damage from what I've seen is substantial. Will update if this changes but so far I can't see me getting to look at this until the end of Friday and it'll be a day or two to fix the mess.

    Are there any backups we can get running in the meantime? At the very least for the Freedom gamemode. Skyblock may be more wanted by some people but if we play on a backup our progress we make will be reset when the fixed version comes online, so I'd be against making a backup of that running unless all regulars on it are fine with just playing for the sake of playing and not expect progress saves.

    According to quantum mechanics, unless something is observed, there is an equal chance of it both being and not being there. Hence it is said to be in superposition, until observed. However we are somehow fully certain, despite never having directly observed, that you indeed have no bitches. Your bitchlessness has broken the rules quantum mechanics had established. Indeed an impressive feat.

  • Current eta for restore is going to be the end of next weekend. Unfortunately I've only just got home and have commitments this week and the damage from what I've seen is substantial. Will update if this changes but so far I can't see me getting to look at this until the end of Friday and it'll be a day or two to fix the mess.

    Are there any backups we can get running in the meantime? At the very least for the Freedom gamemode. Skyblock may be more wanted by some people but if we play on a backup our progress we make will be reset when the fixed version comes online, so I'd be against making a backup of that running unless all regulars on it are fine with just playing for the sake of playing and not expect progress saves.

    multiple problems with that 1) our backups are so large it would be incredibly painful to do a full restore from one iirc, 2) the whole issue is that due to some players abusing their newfound permissions once they got them, a lot of network wide stuff is all fucked up, as such it is unclear just how much is fucked and so it’s risky to try and have any segments of the server open to the public.

    You can live without TF for a week, go play hypixel or something

    there’s also a small server I’ve heard of under the ip plex.us.org but it’s probably lame

  • Just as an update, I've identified the individual who I believe has caused this, and further investigation is ongoing in that regard.

    I have also now been able to work through all of the changes that were made to the Bungee itself and think I've repaired any damage that was caused. I'm currently working through updating plugins and the bungee itself before that comes back online.

    The Minecraft servers themselves are going to require a bit more work as some were seriously damaged as a result of this incident, I'm hoping to get some basic functionality up and running later today.

  • Just as an update, I've identified the individual who I believe has caused this, and further investigation is ongoing in that regard.

    I have also now been able to work through all of the changes that were made to the Bungee itself and think I've repaired any damage that was caused. I'm currently working through updating plugins and the bungee itself before that comes back online.

    The Minecraft servers themselves are going to require a bit more work as some were seriously damaged as a result of this incident, I'm hoping to get some basic functionality up and running later today.

    That’s good.

    meow-upscaled.png

  • some were seriously damaged as a result of this incident

    Still can't understand why the instinct of some people isn't "Wait I'm not supposed to touch this, better not get in trouble" instead of "Oh wait I can grief this lol"

    TotalFreedom's Executive Community & Marketing Manager

  • Just as an update, I've identified the individual who I believe has caused this, and further investigation is ongoing in that regard.

    I have also now been able to work through all of the changes that were made to the Bungee itself and think I've repaired any damage that was caused. I'm currently working through updating plugins and the bungee itself before that comes back online.

    The Minecraft servers themselves are going to require a bit more work as some were seriously damaged as a result of this incident, I'm hoping to get some basic functionality up and running later today.

    well who tf was it

  • Just as an update, I've identified the individual who I believe has caused this, and further investigation is ongoing in that regard.

    I have also now been able to work through all of the changes that were made to the Bungee itself and think I've repaired any damage that was caused. I'm currently working through updating plugins and the bungee itself before that comes back online.

    The Minecraft servers themselves are going to require a bit more work as some were seriously damaged as a result of this incident, I'm hoping to get some basic functionality up and running later today.

    well who tf was it

    It was obviously Scutix. /s

  • Just as an update, I've identified the individual who I believe has caused this, and further investigation is ongoing in that regard.

    I have also now been able to work through all of the changes that were made to the Bungee itself and think I've repaired any damage that was caused. I'm currently working through updating plugins and the bungee itself before that comes back online.

    The Minecraft servers themselves are going to require a bit more work as some were seriously damaged as a result of this incident, I'm hoping to get some basic functionality up and running later today.

    well who tf was it

    As the post says, it's being investigated further.

    Patrolling the Mojave almost makes you wish for a nuclear winter.

  • im confused, isnt it very public knowledge who caused this? or do you mean who ran the command

    as I think it isn’t super clear from the comments of this post I will explain. Basically after the permissions all fucked up on the events server the problem was contained only within the events server, but then one user on the events server added the * permission node to the default network manager group, fucking up permissions all across the network. Presumably, after this happened, that same user, and probably some others, went around to the other servers and fucked stuff up further, now that they had full unrestricted permissions on the entire network

  • im confused, isnt it very public knowledge who caused this? or do you mean who ran the command

    as I think it isn’t super clear from the comments of this post I will explain. Basically after the permissions all fucked up on the events server the problem was contained only within the events server, but then one user on the events server added the * permission node to the default network manager group, fucking up permissions all across the network. Presumably, after this happened, that same user, and probably some others, went around to the other servers and fucked stuff up further, now that they had full unrestricted permissions on the entire network

    Thanks for explaining, but it’s fucking ridiculous to act like that one individual was the reason for this outage. We fucked up, they found it. Our fuck up is the main cause, not the individual. We failed in our role to make sure shit like this doesn’t happen; I appreciate that the user did fuck up too and they should be banned, but the majority of blame lies with us so let’s not pretend.

  • im confused, isnt it very public knowledge who caused this? or do you mean who ran the command

    as I think it isn’t super clear from the comments of this post I will explain. Basically after the permissions all fucked up on the events server the problem was contained only within the events server, but then one user on the events server added the * permission node to the default network manager group, fucking up permissions all across the network. Presumably, after this happened, that same user, and probably some others, went around to the other servers and fucked stuff up further, now that they had full unrestricted permissions on the entire network

    im confused, isnt it very public knowledge who caused this? or do you mean who ran the command

    as I think it isn’t super clear from the comments of this post I will explain. Basically after the permissions all fucked up on the events server the problem was contained only within the events server, but then one user on the events server added the * permission node to the default network manager group, fucking up permissions all across the network. Presumably, after this happened, that same user, and probably some others, went around to the other servers and fucked stuff up further, now that they had full unrestricted permissions on the entire network

    oh brother that might have been me, i thought it would only add the permissions to events-02, not the entire darn network, sorry :( (btw after that the only thing i really used was /adminchat, some others might have done the other bad things) sorry folks, i did a cosmically sized oops

    Edited once, last by SuperRyn (August 6, 2022 at 6:18 PM).

  • as I think it isn’t super clear from the comments of this post I will explain. Basically after the permissions all fucked up on the events server the problem was contained only within the events server, but then one user on the events server added the * permission node to the default network manager group, fucking up permissions all across the network. Presumably, after this happened, that same user, and probably some others, went around to the other servers and fucked stuff up further, now that they had full unrestricted permissions on the entire network

    Thanks for explaining, but it’s fucking ridiculous to act like that one individual was the reason for this outage. We fucked up, they found it. Our fuck up is the main cause, not the individual. We failed in our role to make sure shit like this doesn’t happen; I appreciate that the user did fuck up too and they should be banned, but the majority of blame lies with us so let’s not pretend.

    yeah I agree, this has been extensively talked about already behind the scenes though, and while I do not know if any sort of punishment will happen for the individual who caused this issue, the underlying issues which caused this to happen have been addressed privately

  • oh brother that might have been me, i thought it would only add the permissions to events-02, not the entire darn network, sorry :( (btw after that the only thing i really used was /adminchat, some others might have done the other bad things) sorry folks, i did a cosmically sized oops

    You shouldn't have touched it at all...

    ピバラ。

  • Just to update.

    We've now got Hub, Freedom and Skyblock Game Modes fully back up and running inside the network. The hub has a minor snag with the compass selector where it's not able to reach other servers, I need to figure out exactly what's causing that as it's a bit weird, guessing it's where I've moved the server.

    I've moved the Hub-03 server into the panel so it's now running on MC-01 rather than its own VPS, this was on the card for a little while but this felt like the time to do it given I understand the Hub world itself was damaged so has been restored to the 1.1 build from the MB Team.

    From what I can tell Skyblock didn't have any damage caused as permissions are set up there locally rather than through the network so the * permissions didn't get used or abused there, same with the Freedom game mode.

    The events server has now been removed, and there will not be an events server for the time being. In the future should there be a requirement for an events server, it will be managed by the Systems Administration team rather than being permitted to be managed or run by the events team due to this issue.

    We're making some final checks and updating Discord perms, but once that's done we should be good to welcome you back online.

    Wild1145

    Network Owner at TotalFreedom

    Managing Director at ATLAS Media Group Ltd.

    Founder & Owner at MastodonApp.UK