For those interested, I've saved the spreadsheet and will be keeping that up to date. I will update this thread as well, but it is already getting a bit hard to follow.
For those interested the spreadsheet can be seen here - https://atlasmediadev-my.share…jwslCwC9OgHl7iCQ?e=7j4tdC
@Erin#18285 Some of it was, but it didn't really make a dent. I'm currently working through it 70000000 rows at a time, which tends at the moment to be about a couple of days. Current plan is to remove everything from before April iirc.
For anyone interested, I've plotted the data on to a chart. This one shows a forecast of what the storage sizes would look like over the coming months unless we do something.
And this shows the total storage use by month to date.
The reason for doing this now, is because we're finding that Freedom-01 is now so large, it breaks the backup solution which means it's iffy when TF does / does not get backed up... That's why we're keen to start looking at ways to split the worlds out and hopefully prevent this from getting too much worse, along with purging CoreProtect data from the server and ageing it off, which I don't like, but unfortunately needs to be done.
The short answer is no.
We ban people for their actions... Not their potential / past actions.
If there is a player who has an open permban request, then they can be sanctioned, when they break the rules online.
I've already warned admins for this sort of thing before, and I'll absolutely start suspending admins that I find banning players for no reason, which this would be classed as.
Under exceptional circumstances where the player poses a genuine threat to the stability of the server, authority sits with the executive ban manager, lead developer or owner of the server to issue / approve the banning of players where open indef ban requests are outstanding.
I won't lock this thread now because I'm sure there's more discussion to be had, but that's the policy as it stands unless someone can make a really good case for banning players for no reason.
I'd like you all to join me in congratulating DragonSlayer2189 as being appointed our new ECD for TotalFreedom. As a long standing member of the Master Builder team, and being heavily involved in a number of project I'm sure they'll do an excellent job.
Congrats.
@UnderTails#17965 I've laid out the options that the community has... You can't create options where there are none. I won't sacrifice the stability and performance of the server over this.
The community can continue with gravity disabled and various blocks that are currently blocked re enabled. Or you can go back to how it was before the latest tfm release and have gravity with none of the blocks which can cause such lag through falling disabled.
Those are the only options here as I said.
Also for clarity your options are.
1) We continue to disable gravity and enable all blocks in world edit that currently are or should be disabled.
2) We undo the community decision to allow these blocks and re enable gravity.
Those are basically the two options we have at this point. We know it causes lag and greif which is challenging to restore, and we know there is a current bug where sand and gravel is allowed when it should never have been.
I was asked to disable gravity in order to enable blocks like concrete powder and such, if the community has changed its mind then that's fine. But I do want to make sure we're clear with the actual options here.
Could I ask folks who reply to this to reply in detail beyond the usual vouch / object. There's a lot to discuss here and this will be reversing an already approved suggestion so I'd like to see some strong reasoning behind doing so.
I'll post my own response to this later.
@Folfy_Blue#17802 so for clarification the reason game mode is disabled in vanilla pre dates spectator mode and is more because the gamemode command can change other players game modes as does the essentials version. As tfm takes command priority it just is easier there to re write the command.
And all this since Feb I think it would be... Cracking work chaps.
Yeah, in theory this shouldn't be too hard to do, I do something similar in the social media feed channel on Discord through Microsoft Power Automate so it pulls the picture of the twitter and display name and populates that.
I wanted to give everyone a bit of an update, while we are still progressing through the investigation and there is a lot of information still to be confirmed or unknown, I wanted to give some information based on what we have found so far.
The dev server was at some point previously (I believe around Feb 2021) cloned fully from the Freedom-01 server at the time, as Freedom-02 has been as well. The reasoning behind this was to allow us to create an accurate non-production environment that the development team could test and experiment in without it risking the impact of players on the live servers.
Due to failings on my part some steps that should have been taken to roll various credentials and sanitise / destroy some data, was neglected and is ultimately why we are here today. We are working to put steps into place to ensure that this doesn’t happen again, and in the interests of clarity, access to the production servers are significantly tighter controlled, with access to Freedom-01 and Freedom-02 currently only being granted to Steven, Fleek and myself where we are the 3 who actively require such access.
We know that the attackers had gained unauthorised and illegal access to our infrastructure, which from all indications and investigations so far, suggest was limited to just our development environment. The credentials which were breached were only valid on the development server and we have seen no current indications of any of our other infrastructure being in any way accessed or compromised.
From what we have seen the attackers took a full dump of the entire file system which they had access to with those privileges, which included the “tfserver” Linux user, which runs the server, therefore all plugin data, world data and config files for example were retrieved and for a short time published through sites only accessible via TOR.
We are still working through all of the data and working with individuals who took copies of this data for archiving / their own OSINT purposes to evaluate the full data that was included, and at the current time the development server has been suspended and network disconnected to prevent any potential future accesses.
Based on the data we have been able to confirm, we believe it is limited to data such as the users UUID’s for Mojang, and IP’s which they have previously connected to us on as stored in Essentials, and for admins also within TotalFreedomMod’s database.
It is important to stress that this information alone does not pose any direct threat to our end-users, however as with any data like this being made more easily available than before, we would suggest folks use their judgement and are extra vigilant. Where possible it is also recommendable to have your IP address change, for most ISP’s this can be handled through a re-set of the router (Power it off, wait 10 mins, power it back on). Where this is not possible contacting your ISP and notifying them that your home IP Address was included in a breach, often will enable the ISP to change it for you.
Based on our analysis to date, we identified several short falling in our process, many of which resulted in some credentials being accidentality shared between servers (Telnet for example), these were changed as soon as we identified the credentials had been left as identical. Likewise, the Votifier keys were inadvertently left the same, and were promptly changed.
We have also been able to identify that attackers while on the server could (Though we’ve not been able to confirm this) have also had access to the Dev Servers CoreProtect Database, The live NetworkManager Database, and the live Player Plan database. Access to these would only have been possible from the server itself, and from the data which was published there is no indication to suggest access was even attempted, the CoreProtect and Plan databases that we have seen in the data published so far appear to be legacy local SQLite instances, which were replaced by our central MySQL server at a later date.
We are still working through the data and will be working through various remediation steps to better secure our infrastructure that remains, along with making sure our processes for building servers going forward is more secure and that we learn from the mistakes on this occasion.
I also wish to stress that access to sensitive information such as the forums was not possible from the dev server, nor was access to any other databases that weren’t directly related to the dev server. We have engaged with the UK's Information Commissioners Office, and at the current time legally there is no requirement for disclosure given the nature of the information published, though as I've said on other threads I believe it's only right to be transparent as we have information that we know and can share.
I will post an update here in due course when we’ve worked through the remainder of the data, and when we’ve finished remediation action, but wanted to update you with what we know to date.
Thank you again for your understanding.
I hope to be able to provide a further update tonight / tomorrow. I appreciate folks patients and want to confirm that so far no data we have been able to confirm has been accessed or published is believed to be significant enough to normally justify notification of the ICO here in the UK. And again, a lot of this data was included in previous leaks of the server (Mainly the one where Seth accidentally uploaded the full server backup to his web server, which was then downloaded and archived).
I appreciate all of your understanding while we make sure get this right.
Quote@k3das#17553 The issue would've been that there would be even less confirmed information going around. For a lot of companies they're pressured to include responsible disclosure in their breach mitigation plan, because otherwise people hear "there was a breach," and don't understand what really happened.
I don't disagree, my point however does stand, I've disclosed the current information we have and that our investigation has been able to confirm. The investigation is ongoing, and rather than prevent conversation about it or flat out deny it happened (As has previously happened when most of this data was published under previous owners) I wanted to be transparent.
Again I'll remind folks that nearly all of this information was already made public in a leak of server backup files, which was what allowed us to stand up TF when Seth deleted the files... I'm quite sure there wasn't the same level of due-dilligence performed then.
Quote@k3das#17553 Then you have a different explanation to why 7,932 player profiles have appeared in /plugins/Essentials/userdata at exactly 19:50:12 EST on 02/13/2021. All of THIS data shouldn't have been copied, but it was. Yes, there are 12 new profiles that were likely caused by the dev team. About votifier keys, this shouldn't be a problem because they were thankfully changed:
As I said, there is no current confirmed evidence that it is the case, I'm still working with my team to confirm what is accurate and what is not, and to establish what we know was taken, and to establish what could have been.
Quote@k3das#17553 About votifier keys, this shouldn't be a problem because they were thankfully changed:
A number of credentials that were originally overlooked or intentionally shared have been changed, I've been waiting for confirmation of this, the comment by RedEastWood should not have been posted as there was yet to be confirmation of them actually being changed. I hope to update people on the steps taken to date later today.
Quote@k3das#17553 Yet people don't assume things to be false, it should be partially your responsibility to spread real information so people don't need to guess. You did this well with the original announcement, I'm not really blaming you for this part.
I know, which again is why we've been mindful to publish what we know and can confirm when we've been able to do so. It's important we get this right and make sure we're complying with the relevant authorities as our investigation progresses. To date we've done that, and I expect the investigation will take some time yet to fully conclude. We have a number of lessons learnt we need to put into place before publishing any details on them, again to help make sure we're learning from this mistake.
Quote@k3das#17553 I guess a screenshot wont be enough to convince you that there are people who collect data breaches for their own OSINT:
As I say, it's not something we've been able to confirm and until I am able to confirm this as stated above, it should be assumed that speculation around data that might have been published is un-confirmed and potentially false.
Quote@k3das#17553 I'm not well-versed in GDPR, but to my understanding, you'd need T&Cs regarding what data you collect on the forum, and who you share it with, although I might be wrong.
The forums (Fortunately?) pre-date my ownership of the server, we've put steps in place where we can to make sure we're legally compliant, but ultimately it's another driver of moving off Flarum, because we just don't have a lot of info around what Flarum collects / needs to collect. If people have concerns over the data collected on the forums then they can reach out to informationsecurity@atlas-media.co.uk which is the current process for data protection queries around the forums while we work on putting a more formal and robust process in place, again part of moving to the new forums. We've got processes in place to comply with right to be forgotten requests, and subject access requests.
Quote@k3das#17553 Well I guess so, It was my poor judgement in the middle of the night.
It's fine, I appreciate this is a hot topic, and I appreciate there aren't a lot of answers right now, I want to give more answers, but I want to make sure I'm not going to be sat here retracting those statements because I didn't do the level of checking I should have.
Quote@k3das#17546 Ryan, if you hadn't disclosed this breach it would've been a lot worse for you, so stop making it seem like you're being nice to us.
It really wouldn't. It wouldn't have been hard to have denied any breach at all. UK law (which we are primarily governed by) has no requirement for disclosure in these circumstances. That's what the majority of companies do.
Quote@k3das#17546 None of this would've been an issue if the dev server didn't have live data. I completely understand the importance of a good dev server, but real player IPs do not belong there.
The dev server will always have "live" data as players are regularly invited to join the server for testing. While lessons have been learnt and when more information is known and confirmed, we will publish more information and cover off those lessons and the changes to be made going forward.
Quote@k3das#17546 There is way too much general misinformation going around on this thread, and on the discord, so here is my attempt to try to clear things up:
The original announcement has all of the information known to date. Any information stated otherwise should be assumed to be false.
Quote@k3das#17546 The newest player data in the leak was from 03/25/2021, but it looks like almost all player data was added on 02/13/2021. In total, almost 8k
At this time there is no evidence to confirm this. We are still investigating the exact content of the breach and other content which may have been impacted. Right now you are simply adding to the mis information...
Quote@k3das#17546 The leak also contained ban data.
f. Yes there were Votifier keys, but I hope that those were changed since then.
Please again see my previous point. You are currently doing nothing more than spreading mis information.
Quote@k3das#17546 For most players, their IP very likely already changed due to ISPs mainly using dynamic IPs
This isn't entirely true. And should not assume to be true.
Quote@k3das#17546 Ryan, please stop saying things along the lines of "people are stupid." It doesn't inspire professionalism, and a better solution for you would be to try to clear up some misinformation
I haven't claimed anyone to be stupid. I've highlighted where misinformation is being spread or where statements are blatantly false.
Quote@k3das#17546 There has hardly been any transparency before this,
Pretty much everything we do is in the public domain. I don't really see how there is much we could have done to be more transparent. In this case we will only publish information when we've been able to confirm it. Else we're just speculating.
Quote@k3das#17546 and you should have the systems in place to detect somebody using a dev's credentials to download 1.1GB of data off a dev server.
Why? It is entirely reasonable to expect the developers to need to upload / download large files from the Dev server for troubleshooting and debugging... That wouldn't have done anything meaningful beyond plenty of false positives from what I can see.
Quote@k3das#17546 Also, isn't posting banned player's IPs in public chat breaching PPI? I don't think I can find any TOS, T&Cs, etc on your website or forum, and no cookie notice. All of which are a breach of GDPR.
At the current time we have consulted with the information commissioners office and they have confirmed based off the data we have confirmed to be leaked, there is no requirement for disclosure to end users or to them. We've disclosed to end users because I personally believe that the right thing to do.
Quote@k3das#17546 Please acknowledge you and your team's mistakes.
I have and continue to do so... Otherwise I wouldn't have confirmed there was a data leak in the first place.
@videogamesm12#17530 not sure I can think of worse things...
Let's all just use a massive telegram group or matter most server. How bad could it be...
@Telesphoreo#17505 the issue is you missed the point I've made here and on the original panel thread where that quote was taken from out of context...
This (as with all breaches) was a matter of when, not if. Every company / system will at some point be compromised, the difference is that I've been transparent in that, despite having no legal or regulatory requirements to do so, because I'd rather be transparent with data security with the community and while that will make me personally look bad because it happened on my watch, and may even lose us players I believe I'd want to know if a leak happened that contains my data, even if there is not a legal requirement for someone to tell me.
@fionn#17498 Ultimately that's what we are interested in. If someone really wanted to do damage social engineering is a common approach and would give the perception of trust...
