Is there a way to completely delete users including all their data?

Re: Is there a way to completely delete users including all their data?

by stefan weber -
Number of replies: 14
Picture of Plugin developers

tbh, i gave up on the data registry. as far as i'm concerned, it's just another unusable broken moodle feature. 

i'm deleting the users manually with the plugin "delete user by mailing list" now. deleting manually now creates a GDPR delete request, and as of 3.6.2 deleting users this way actually works without breaking moodle or the cron job, even if it still leaves behind data like forum posts, which makes me wonder what the point of all of this is.

the only way I found to remove deletion requests is to directly delete them from the DB table.

In reply to stefan weber

Re: Is there a way to completely delete users including all their data?

by Zoran Jančić -
Picture of Particularly helpful Moodlers

This is just terrible. All that effort for development of GDPR compliance pluggins that actually don't work. 

BTW, I use Moodle Moodle 3.6.3 (Build: 20190311) and can't permanently delete user in any way. Deleting the record from DB table mdl_user would break the database relations so it's not an option for me. After manually deleting a user using Moodle's standard admin interface, the user doesn't appear on deletion request list and his account stays in the database forever. So currently I'm out of options.

In reply to Zoran Jančić

Re: Is there a way to completely delete users including all their data?

by Zoran Jančić -
Picture of Particularly helpful Moodlers

actually, I found the deletion request now, approved it, ran the cron but the user's record is still in the database and it's not even anonimized sad

In reply to Zoran Jančić

Re: Is there a way to completely delete users including all their data?

by stefan weber -
Picture of Plugin developers

wow, you are right - I just checked userids of already deleted users, and their name and e-mail address is still in the database... I highly doubt that this is GDPR compliant...

Together with the fact that not even Forum Posts are deleted, I'm starting to wonder what sense there is in using those GDPR plugins at all...

Andrew (or anyone competent/responsible for Moodle privacy/security) - could you please comment on this and clarify things?

Average of ratings: Useful (1)
In reply to stefan weber

Re: Is there a way to completely delete users including all their data?

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Stefan,

I have replied to your previous comment to try and explain and clarify these points.

We have made every effort to ensure that things are 'compliant', but different interpretations of the legislation make this challenging at best.

Andrew

In reply to Zoran Jančić

Re: Is there a way to completely delete users including all their data?

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Zoran,

Without any further information it is hard to give advice.

Please note that some requests are only created when cron is run. I would recommend making sure that your cron is running correctly and checking the logs.

We are unable to remove the actual database record. Unfortunately while we have made every effort to remove data, we cannot remove the actual DB records in some cases. Moodle is a plugin ecosystem and many plugins make use of core data. In some cases these plugins have not been updated to support the privacy API, or they themselves are unable to remove all data for some reason. As a result we are unable to remove records which that data depends on.

Best wishes,

Andrew

Average of ratings: Useful (1)
In reply to Andrew Lyons

Re: Is there a way to completely delete users including all their data?

by Zoran Jančić -
Picture of Particularly helpful Moodlers

I understand, but what about anonymization od user DB records? 

In reply to Zoran Jančić

Re: Is there a way to completely delete users including all their data?

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Please see my response to Stefan where I've attempted to address that question.

In reply to Andrew Lyons

Re: Is there a way to completely delete users including all their data?

by stefan weber -
Picture of Plugin developers

You did not adress that question at all in that reply...

In reply to stefan weber

Re: Is there a way to completely delete users including all their data?

by Andrew Lyons -
Picture of Core developers Picture of Moodle HQ Picture of Particularly helpful Moodlers Picture of Peer reviewers Picture of Plugin developers Picture of Testers

Hi Stefan,

I'm sorry that this is your experience. We do take this feedback seriously and we have spent a lot of time trying to improve many aspects of the process over the past 12 months since the initial release of the Privacy API and toolset in Moodle.

The Data Registry is a powerful and useful tool, but it can be time-intensive to set up. GDPR legislation requires that you identify the reason for storing data, as well as the lawful basis for doing so, and the intended retention period. This identification of purpose should be as accurate as possible and not applied blanket to all data.

The key difficulty we have faced is that Moodle has not historically kept information about when a users interaction with content ends. Initially we considered any course without and end-date to be in-progress, but we have updated this to allow administrators to determine if open-ended courses should be considered blocking or not.

The other difficulty that we have faced is that ensuring the accuracy of the expiry calculation is an expensive process. In order to determine whether a user has 'expired', we must look to see which courses they are involved in and the activities within those courses, and then whether the retention for any of those has been reached. Whilst a user has ongoing involvement in an activity or course, or still has data in an activity or course which has not expired we cannot remove that user. Therefore we need to calculate the expiry status of every activity, block, and course that a user has content in. As you can imagine this is not a simple task where a user has many interactions.

When a user is deleted we attempt to remove all data for that user. The exceptions for this are for data which has been marked as Protected in the data registry. The protected status should only be used for data which is stored under the legal basis of Public Task, or Legal Obligation - these are explicitly defined as cases where the right to erasure does not apply under GDPR.

Regarding items not being removed from forum this sounds like a bug. It could be one which has been fixed (MDL-63632), or it could be an as-yet unreported bug. It would be great if you can test this with the latest versions of Moodle and provide replication instructions - we can only fix bugs that we know about. When dealing with user deletions and the forum we are only able to remove the content itself, and no the shell that it was in. This is because removal of the post should not affect grades (i.e. via ratings), and should not cause the removal of replies to the original post (i.e. data belonging to another user).  When a course or activity expires then all content in that activity is entirely removed.

In regards the deletion of Data Requests, we deliberately do not remove deletion requests. These are kept to serve as a paper trail that the deletion was actually processed and this is allowed under GDPR. It falls under the basis of Legal Obligation as the record is required to prove that you received, and acted upon the request in accordance with the legislation. Related to this we do not anonymise all data in the user record because it is required to identify the user that the request has been processed for.

As I'm sure you have experienced the GDPR legislation is far from simple. In addition to this some countries have their own addendums to the core legislation, and it is possible to interpret some parts of the legislation differently from person to person.

During our implementation of the Policy API and associated tools we we did seek legal advice from a lawyer specialising in GDPR and believe that our interpretation is correct based on the advice that we have received.

I hope that this clarifies the situation,

Andrew

Average of ratings: Useful (7)
In reply to Andrew Lyons

Re: Is there a way to completely delete users including all their data?

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

Great information, Andrew.  Thanks.  You can probably write a 100-page book about this.  Hmmm, a 100-page document... sound like something that lawyers enjoy doing.smile

I want to add another issue about removing posts (maybe what you were trying to say.)  Right now, if a post is removed and it had replies to it, the replies are also removed.  And this could affect grades for those students who made those replies.  This is why a post cannot be simply removed.  And the solution to this might not be easy.  Should Moodle replace these posts with some "ghost" user?  And how about the words within the post that might somehow identify the original student? And even if the post could be "erased" and the replies remain, if a student whose posts remain questions their rating, how would the instructor know what this student was replying to?

So the idea that a user and that user's content be deleted is really not an easy thing to do, which your post portrays.

In reply to Rick Jerz

Re: Is there a way to completely delete users including all their data?

by stefan weber -
Picture of Plugin developers

Right now, if a post is removed and it had replies to it, the replies are also removed. 

This is not actually true, replies are not removed. Maybe it's a bug in an older version (we are on Moodle 3.6.2+ (Build: 20190201))

And how about the words within the post that might somehow identify the original student?

The words are already deleted, so this is not an issue.


In reply to stefan weber

Re: Is there a way to completely delete users including all their data?

by Rick Jerz -
Picture of Particularly helpful Moodlers Picture of Testers

I have a casual interest in this topic, so I am clearly not knowledgable of all aspects of GDPR.

In my Moodle, I just tested that "replies are also removed" when a post is removed that has replies.  Sure enough, the replies are also removed.  I am running Moodle 3.6.3 (Build: 20190311).  However, I have not installed or enabled any GDPR features or plugins, whatever the default settings are in a standard Moodle for GDPR is what my Moodle has.

In my Moodle, again without any special GDPR plugins, I have also confirmed that a deleted user's posts will still show their posts and their name, even after they are deleted.  But in my case (I am in the US), I would say that this is what I personally would want to happen because the student's post and their reply is needed for proper grading (rating) of other student's posts.  Sure, maybe one could argue that the name should become "anonymous," but then some students would complain that the person who they replied to is now missing and that this is the person who they replied to, in context.

In a physical classroom, if there was an activity that involved students videotaping each other, and posting these videos as a project assignment, and then a student drops the course and wants their data deleted, how does one delete this student's presence from every video that they appeared in? Should all videos be deleted?  Would this be any different than retaining their posts in a forum?

Yes, my novice knowledge about all of this GDPR stuff is showing.  But as an average "Joe", I just don't understand.  (But I am trying to appreciate all of this GDPR stuff.)

In reply to Andrew Lyons

Re: Is there a way to completely delete users including all their data?

by stefan weber -
Picture of Plugin developers

We are using Moodle 3.6.2+ (Build: 20190201), so MDL-63632 should not be an issue.


I know that GDPR legislation is far from simple, I hate pretty much everything about GDPR with a passion, and I know that it is very hard for a community-driven project the size of Moodle to implement all aspects of GDPR.

However, neither me nor Zoran are really talking about minor points or certain aspects here. The very thing that these very complicated plugins are here to do, the single thing that I would measure them by, is deleting the user's information.

Yet, this does not happen when I delete users. This is also why user names still show up for deleted forum posts, since the user information is linked to the user record in the DB, and the name of the user is not deleted from the DB.


I feel kind of silly writing down the steps, but there you go:

  • create a new user
  • post in a forum
  • delete the newly created user
  • view the forum post again -> the user's forum post title and text are still here
  • approve the deletion request that is automatically created
  • wait for cron to run, the deletion request is now on status deleted
  • view the forum post again -> everything except the actual content of the post is still here, title of the post, and the name of the user

  • which is no big surprise, since the user's information never gets deleted from the DB

  • which is the one central thing the GDPR plugin is supposed to do, right?
  • which also is absolutely no problem, since all it has to do is change the information in the DB without deleting the record


  • and voilá - of course the forum no longer shows the information, since now the information has actually been deleted



Do I really need to issue a tracker item for a bug here -  can you confirm that this actually works anywhere

I don't need a 3 page explanation why it is hard to delete DB records, when both me and Zoran explicitly stated that all you need to do is delete the information from the DB without deleting the record!!!

Forget all the overly complicated legal crap and the unusable GDPR plugins without proper documentation, all we want is a way to bulk delete user's information!! 


Average of ratings: Useful (3)
In reply to stefan weber

Re: Is there a way to completely delete users including all their data?

by stefan weber -
Picture of Plugin developers

Found out that this is has already been a tracker issue for almost a year: MDL-62865

I'll try to come up with a plugin if I find the time.


Average of ratings: Useful (2)