Hardware and performance

We've been trialling this and, like you have found, it's very difficult. 

Firstly, the amount of space you need to store even a representative sample of these backups is huge. Our (incomplete) solution centred around replicating the Moodle instance and leaving the replicated instance to do the backups. The additional load is horrible and you don't want to be doing it on your live site. We also have 15,000 courses and the initial backup (on pilot hardware) took over a week. 

We've shelved this for the moment mostly for reasons other than the technical complexity, if you know what I mean

Running version 3.3.x, F5 load balanced 3 front ends, glusterfs, memcached for application cache.

We keep 4 semester's worth of courses on our production server with approximately 800 in each. They are all organized by Semester and then Subject.  We only use the course-level backups 3x per year to 'archive' the oldest semester.

I use a minor code hack to direct the automated backup to backup just one category each semester and we mount a windows' share with a lot of storage to our linux box so the backup files get written directly to the share, not the Moodle instance so there are no storage issues.

Once the backups are done we delete that semester's worth of courses from the production server.

This process works very well for us to keep the courses current on our production server while maintaining backups for those instructors that need copies of older courses that they may be re-teaching. Super easy to browse to the windows share to restore backups on request.

If anything you can use the hack to target categories to prevent an entire system backup:

1. moodle/config.php
ADD
// List category id values to be included when backup runs
$CFG->backup_category = array(65);

2. moodle/backup/util/helper/backup_cron_helper.class.php
ADD
// Check for categories to backup
if (!empty($CFG->backup_category) && is_array($CFG->backup_category))
{ $rs = $DB->get_recordset_list('course', 'category', $CFG->backup_category); }
else
{ $rs = $DB->get_recordset('course'); }