Hello Miroslav,
To keep the course from failing its backups for our Moodle instance I increased this setting set_time_limit (3 * 60 * 60) in moodle\backup\controller\backup_controller.class.php to give our moodle instance three hours to backup a single course. It still takes a long time for Moodle to backup the course but I did make some adjustments to the backup settings to reduce it from taking 13 hours to 5 hours on average.
-Do not include logs
-Truncate mdl_backup_controllers if that table is extremely large. I have seen these table get to 20 gigs in size with moodle 2.2 and 2.1. I believe Moodle fixed this table from growing exceptionally large with 2.5 but I do regularly check it.
-Optimize database.
-Set Moodle to only backup a course if it has changed since the last backup. This setting has greatly reduced the amount of time during the backups as Moodle will skip close to 80% of the 500 courses we had.
We are also working on having a dedicated server handle the cron jobs for our instance as well to help speed things up. I am currently testing to see if we will have major added benefit of keeping the server hosting the moodledata/mysql instance on the same server that is running the cron jobs.