You say, it is all moodledata/filedir? How about the other directories like cache, localcache, sessions, temp, trashdir, ...?
I have been running my small moodle, I watch my backup sizes, and I have never seen this kind of growth. But I do know that if I were to load my videos into Moodle, my moodledata folder would dramatically grow.
du -h --max-depth=5 /var/moodledata/filedir | sort -rh | head -5
That gives the amount of space used by the 5 biggest folders sorted by size. My dev moodle shows
Pick the biggest directory to find where the space is going. Lots of small files or a few big ones? When were they created?
$ ls -laShr /var/moodledata/filedir/27/b3
drwxrwxrwx 9 www-data www-data 4.0K Aug 12 14:55 ..
drwxrwxrwx 2 www-data www-data 4.0K Aug 12 12:04 .
-rw-rw-rw- 1 www-data www-data 880M Aug 12 12:04 27b392d76a38ff19979f8aeefa64ae86a31e6b9a
I've also check the biggest files as suggested by James and are annotated PDF files between 15 and 20M each. Aren't they quite big for a PDF?
Let's assume that you can download one of these 20MB PDF. See if it happens to be a scanned document. When I scan documents, it is not odd for a 10-page document to be large, like 100MB. What I do is to open this large scanned document in Acrobat, then pick the Scan & OCR tool, then Enhance - Scanned Document, then Enhance, and save. This process dramatically reduces the size of PDFs (like from 20MB to 300KB. I think that you have to have a copy of the real Acrobat, and not the Acrobat Reader to do this.
From what you have said, my guess is that one or more of your instructors, to quickly go "online," have scanned a bunch of their handouts and uploaded them into your Moodle. Just my guess.
I would check PHP file size upload limitations. Possible someone has increased the limits to allow say a one off backup upload and a teacher has uploaded a full movie. I would think this PHP limit should prevent big uploads.
I wonder if you have recursion somewhere - a backup is added to the system and in turn backed up to double the size of your size. That would show as a file comprising much of the space.
How much space can a ghostscript file take? How can it be uploaded through PHP?
The requirements of the assignment (like multiple files or a 30 page word doc or ?) would lead to conversion to PDF that could be large ... so check the assignment requirements in course.
Annotating conversion will combine multiple files of different mime-types if it can convert submitted files into a single PDF and when viewing in the grading interface it's sometimes difficult to tell where one doc ends and another begins. At any rate, it's a single PDF file.
https://moodle.org/plugins/pluginversions.php?plugin=report_coursesize (works on 3.9)
It's not exactly precise but will give a good picture of your installation.