This forum post has been removed
Number of replies: 35This error has been illusive in that it appears from time to time.
However, question for investigation ... in your moodledata/temp/backup/ directory (in Windows which I think you are using the slashes lean the other way) ... see if there is a long directory name consisting of a bunch of letters/numbers. Change into that one and see what's there. IF you see a backup.mbz file there along with a moodle_backup.xml you got close. Last action is to **copy** the .mbz file to it's destination ... which by default settings is /moodledata/filedir/SEAOFFILES ... none humanly recognizable.
So check ownerships/permission on moodledata/filedir/ as well as moodledata/temp/backup/ as Moodle needs to destroy what it used to build a backup.
It could also be a time out issue ... see your Moodle's Server -> PHP info and the settings for time for script to run, memory for a script to use, etc. Might need to bump those values upwards a little in your php.ini file. Restart web service if you do that.
Courses have a tendency to grow rather than shrink thus backups/restores need more resources from server.
How about I stop there .... and let's see if any progress is made. IF not, might have to attempt a 'deep dive'. :\
'spirit of sharing', Ken
This forum post has been removed
This forum post has been removed
Am not a coder. ;)
Files in moodledata/temp/backup/ can be manually removed and no harm will come to the system ... Linux or Windows.
Before you do, however, do look at those .log files that are larger than 0 bytes in size for clues. You found one, BTW, the successful completion of a restore ends with the staging getting to 1000 - everytime. You saw one that reached 700 ... incomplete.
Now the question becomes why/what? That I can't really tell you ... many times course dependent.
You say it happens immediately ... so it never gets a chance to 'build a plan' and then execute that plan ... reporting reaching stage numbers. So what else might be going on at the time. Are you attempting during prime time usage by students/teachers? Got anything doing a virus scan or backup of something?
Check your web service logs and db server logs. If running MySQL/Apache, one might see a "MySQL server has gone away' error. Need to increase max_allowed_packet. LIke I may have mentioned before ... courses tend to get larger ... not smaller ... thus require more resources of the server.
Backing up using the backup.php script in code/admin/cli/ is a good idea .... make sure you use the option to save the backup to a directory outside of moodledata/filedir/ ... so you can find it later by a humanly recognizable name.
continuing to share in ... 'spirit of sharing', Ken
This forum post has been removed
This forum post has been removed
Running Windows right?
Is your php 64 bit or 32 bit? You mention what version of Moodle, right?
The zip file - looks like it created an enrollments.xml ... but couldn't populate - 0 byte .xml file.
That's a clue ... enrollments on site/courses are handled how?
'spirit of sharing', Ken
Turn autobackups off.
Manually remove all the contents of moodledata/temp/backup/
After that, do what many Windows server admins do ... reboot the box.
Check the reports for autobackups in Moodle admin. You might have to truncate all tables moodle keeps related to autobackups to 're-gain control' .. perception of 'control', I should say.
'spirit of sharing', Ken
This forum post has been removed
Never allowed means what? Never turned on?
Do you have more than one admin level user?
Has autobackups been turned on without your knowlege?
Other than teachers themselves doing backups, in Moodle code
only the cli script in moodlecode/admin/cli/autobackup.php
provides that function. And to access/run those one has to be logged on and execute via
command line. Unless you've hacked code.
Obvioiusly, factors ... your script/code. You've not shared any info
on that. Have never heard anyone say system doing backups all by itself.
Did you look at reports for automated backups in Moodle Admin? Although nothing would be there on something you've programmed me thinks.
This eror "cannot_finish_ui_if_not_setting_ui" means something interputing a backup process. Seen that in a situation where admin was attempting to restore the same course multiple times by using same browser with 3 active tabs attempt to perfom a restore of the same backup file. That system (a 3.5.3+) was loosing track of stages etc. Errors always showed staging had reached 700.
Now what about your scripts? How are they run? In some sort of timed process? What happens when one fails ... does it log which backup failed and schedule that course fora later time?
Am not capable of Vulcan Mind Meld! :| Encourage you revisit your scripts/system/logs, etc.
'spirit of sharing', Ken
This forum post has been removed
Getting closer ... first ... ad_hoc cron, if I re-call, is an old cron process. Don't re-call in which version of Moodle but there is now a recycle-bin for a course. Deleting module in a course would trigger the re-cyclebin for the course which uses backup to keep the resource/module in a position to be restored.
Best guess!
'spirit of sharing', Ken
Have you looked at your plugins? Try disabling all addons and then run it...
Turn full debugging on and run and then see if you get additional error messages...
This forum post has been removed
This forum post has been removed
Congrats! Indeed ... what happened to those tables? Who knows!
In looking over this thread, at the beginning, you said ...
"We create those courses (with my software of course) by backing up original templated course, then restore into the new term courses, one for every time slot we teach that course in."
Have just finished helping a gentleman do pretty much the same thing with scripts
from command line. Using moosh. OP made one no user backup of his
template course (no user, with whatever theme/addons/blocks, etc.). Saved the backup to a location on the server and renamed it 'restorethisone.mbz'.
In the admin/cli/ area of one of the moodle instances, this bash shell script (called restoretemplate here):
(contents below)
echo 'Template course restored to Misc. category of site. Cat ID 1';
moosh -n course-restore /path/to/restorethisone.mbz 1
Restores the template backup to category ID 1 which is Miscellanous by default. Pretty easy for Op ... just executes the script once ... like ./restoretemplate [ENTER]. When it finishes, 'up arrow' for command line repeat previous command, ENTER ... repeat ... repeat ... repeat ... however many template courses OP desires to restore that session.
Template courses are restored like Template, Template copy 1, Template copy 2, etc.
however many times the op executes that script.
Op then logs onto the site, accesses each course and changes basic info in a course to 'band them'. Template copy 1 might become "Puncuation and Grammer 101 - SMU", Template copy 2 might become "Puncuation and Grammer 101 - Texas Tech", etc. and then moved to an appropriate category via Moodle UI.
And Op ends up with the same thing ... only no programming required.
'spirit of sharing', Ken
This forum post has been removed
"A lot of work' ... well considering your resources (multiple people doing jobs related) then yes, my 'sharing' might seem ancient ... for a single OP who wears all hats in the entire setup of his system - includes Paypal, etc. could have expanded that simple script to do a lot more + setup a Webmin that would provide much more 'automation' as well. Single OP ... mind ya ... wears all hats.
But ... more power to ya ... you need all the automation you can muster sounds like.
Suggest filling out your profile with a description of your system(s). Other folks would be interested!
Good luck finding out why those tables were missing!
'spirit of sharing', Ken
Hi Perry,
great investigative work!
If I were you I'd file an issue - maybe w/ low priority - into the Tracker since tracing that exception in the log would be much better, regardless your issue was having a "broken system" from a DB perspective - wondering how LTI activities could work when accessed.
TIA,
Matteo
Happen to be working with a Moodler who does same process ... backup no user, restore to new site.
On his site/hosting, mod_security is installed and active.
The error you are seeing indicates, obviously, cannot finish ... something interrupted the process ... not serious enough to crash Moodle, but serious enough to either kill a backup or complete it with missing parts/pieces. But what? I know not!
There is a command line utility (swiss army knife type like that of drush for Drupal sites) called moosh.
Works well ... on Linux.
The file have attached indicates that restore was successful ... it reached stage 1000.
There shouldn't be a 'matching' directory by "b26146b9868f832de5b5fdef60c81395" in there ... the last two actions is to ***copy*** (note that is copy NOT move) the backup file to configured destination directory ***AND*** then destroy the temp directory being used to build the backup file.
So, sorry, no clue in that .log file. As I tried to explain, the log file means something to coders ... hard core coders ... but not to me ... I asked one time in these forums what the numbers meant and if they were tied to a process for a part/piece of making the backup file. Don't recall getting any response, but then again, if there was, I might not have understood it.
It's only by observing the process that I've concluded that a stage ID number IS associated with making a backup of oh say 'activites and resources' (mods) ... you will see an xml file ... road map to activities directory. A files.xml ... road map to files in files directory but not by humanly recognizable names .. but, rather, built the same way files in moodledata/filedir are stored.
Now if you've created your own program, suggest maybe comparing what you've done to what Moosh does.
Too bad moodle doesn't have a 'restore.php' script in admin/cli/ that works like backup.php.
But, that's where moosh comes in.
There are 44 contributors to moosh
https://github.com/tmuras/moosh/graphs/contributors
Maybe by reaching out to one of them ... the lead ... you could shave off hours and hours ... and get this done!
'spirit of sharing', Ken
Hello
i have the same error. Please i need your help, with a course lees than 1gb work well but if is more than 1Gb aprox apears this error
server: Linux centox
Moodle: 3.2.2
var max:Allow_pocket: 536000010
max_execution_time: 1000
Warning: Error while sending QUERY packet. PID=10658 in /home/****/public_html/lib/dml/mysqli_native_moodle_database.phpon line 1174
Error al leer de la base de datos
SELECT count(r.repositoryid)
FROM mdl_files f
LEFT JOIN mdl_files_reference r
ON r.id = f.referencefileid
JOIN mdl_backup_ids_temp bi
ON f.id = bi.itemid
WHERE bi.backupid = ?
AND bi.itemname = 'filefinal'
[array (
0 => 'fa039f76d89dc6152d9ba2ef521edaa8',
)]
Error code: dmlreadexception
- line 479 of /lib/dml/moodle_database.php: dml_read_exception thrown
- line 1175 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
- line 1551 of /lib/dml/moodle_database.php: call to mysqli_native_moodle_database->get_records_sql()
- line 1624 of /lib/dml/moodle_database.php: call to moodle_database->get_record_sql()
- line 1834 of /lib/dml/moodle_database.php: call to moodle_database->get_field_sql()
- line 522 of /backup/util/dbops/backup_controller_dbops.class.php: call to moodle_database->count_records_sql()
- line 1981 of /backup/moodle2/backup_stepslib.php: call to backup_controller_dbops::backup_includes_file_references()
- line 34 of /backup/util/plan/backup_execution_step.class.php: call to backup_store_backup_file->define_execution()
- line 181 of /backup/util/plan/base_task.class.php: call to backup_execution_step->execute()
- line 177 of /backup/util/plan/base_plan.class.php: call to base_task->execute()
- line 120 of /backup/util/plan/backup_plan.class.php: call to base_plan->execute()
- line 322 of /backup/controller/backup_controller.class.php: call to backup_plan->execute()
- line 136 of /backup/util/ui/backup_ui.class.php: call to backup_controller->execute_plan()
- line 131 of /backup/backup.php: call to backup_ui->execute()
@Wilder Ferney Quintero Vanegas
Might have the same error, but for different reasons ...
MySQL server has gone away
and repository are clues for you.
See /home/x/public_html in the paths which could indicate a shared server with shared DB servers.
Are you on shared hosting?
Typically, those on shared hosting cannot make tweaks to shared DB server.
Even if you have attempted to increase values, system is setup to ignore going over your account maxes.
Gone away is related to MySQL server and I see you have shared:
var max:Allow_pocket: 536000010
Is that from your my.cnf file or some interface you have for the DB for your Moodle?
Default value for MySQL is 16M
https://dev.mysql.com/doc/refman/5.6/en/program-variables.html
Using the mysql client on your moodle server or whatever tool you have to work with DB's, what does the following query show:
mysql> show variables like 'max_allowed_packet';
Fact that you've noticed 1G or below ok but anything over 1G no go - wondering how you know a course backup is over 1G?
In the error it shows repository ...
Make the backup deselecting any link you have in course that goes to
a file from a repository. Those could be videos ... large files.
See what happens.
At this stage ... it's a guessing game.
'spirit of sharing', Ken
hello ken
thanks a lot
It is not a shared server, it is a dedicated server.
Is that from your my.cnf file or from an interface that you have for your Moodle database?
is the value that I assign in the global variables of the mysql.
only happens with some courses, this has 5 modules, if I make individual copies or of two modules, the backup is generated without problems, that is,
Module 1 200 ---- M
Module 2 300M
module 3 200M
Module 4 400M
Module 5 300M
mosulo 1 -2 600 M
Module 3-4 800M
Module 1-3 ... 700M
approximate values
all generate good, but if I generate the full backup it generates the error'public_html' even on a dedicated server is in a 'user jail' ... trust me when I say that!
So you do have ssh into server and can call up the mysql client app on the server.
So again;
mysql> show variables like 'max_allowed_packet';
what does the above show.
Assigning globals via command last only as long as the DB server runs that session. When/if DB server is rebooted or restarted, that global setting is lost. To be permanent, has to be in config file on start up of DB server.
You could also do a full backup of that course from the command line ... takes apache out of the loop leaving only php and mysql server then.
cd /path/to/your/moodle/code/admin/cli/
php backup.php --courseid=[idnumberofcourse] --destination=/path/to/where/you'd/like/to/save/it
Let's try that to see if it errors or finishes ... but only after making sure max_allowed_packet is set the way you think it is.
If it does error, the command line backup.php execution should show errors ... come back here an post the text of that error.
'spirit of sharing', Ken
Hello Ken
I verified and like i told you
| Server Load | 0.279297 (8 CPUs) | |
| Memory Used | 7.79% (2,550,792 of 32,752,996) |
mysql> show variables like 'max_allowed_packet';
+--------------------+-----------+
| Variable_name | Value |
+--------------------+-----------+
| max_allowed_packet | 535999488 |
+--------------------+-----------+
1 row in set (0,00 sec)
i tried to do backup to this course but do not work
root@server [/home/pruebaseducacion/public_html/admin/cli]# php backup.php --courseid=141 --destination=/home/pruebaseducacion/prueba_backup/
== Performing backup... ==
!!! Error al leer de la base de datos !!!
i did a test with other courseid end worked well
root@server [/home/pruebaseducacion/public_html/admin/cli]# php backup.php --courseid=260 --destination=/home/pruebaseducacion/prueba_backup/
== Performing backup... ==
Writing /home/pruebaseducacion/prueba_backup/copia_de_seguridad-moodle2-course-260-prueba1-20190221-1228.mbz
Backup completed.
root@server [/home/pruebaseducacion/public_html/admin/cli]# php backup.php --courseid=24 --destination=/home/pruebaseducacion/prueba_backup/
== Performing backup... ==
Writing /home/pruebaseducacion/prueba_backup/copia_de_seguridad-moodle2-course-24-codificacionexternos-20190221-1229.mbz
Backup completed.
Install moosh
https://moodle.org/plugins/view.php?id=522
It's not really a moodle plugin but a command line utility for moodle.
Once it's installed and checked, let's see what's in course ID 141
From inside the moodle code directory anywhere, issue:
moosh -n course-info 141
The -n allows root user to execute rather than web service user.
You'll see something like this (my example done on course ID 10 in a Moodle):
Course ID: 10
No of contexts: 16
Context by level:
CONTEXT_MODULE (70): 16
Context by module:
assign (1): 2
lti (14): 1
page (15): 3
resource (17): 10
Number of role capability overwrites: 0
Role capability overwrites by context:
Number of local role assignments: 0
Locally assigned roles by context:
Number of enrolled users: 0
Users enrolled by role:
Number of groups: 0
Group statistics:
Min number of members in a group: 0
Max number of members in a group: 0
Avg number of members in a group: 0
Course modinfo size: 33733
Number of sections: 2
Section statistics:
Sections visible: 2
Sections hidden: 0
Min number of modules in a section: 0
Max number of modules in a section: 16
Avg number of modules in a section: 8
Number of grades: 0
Number of log entries: 65
Number of files: 10
Total file size: 8833144
Cache build time: 0.078015089035034
Dunno what we might see for your 141.
Also run:
moosh file-list course=141Still the error is DB related ... I wonder ...
How about also installing MySQLTuner.pl and run it.
https://github.com/major/MySQLTuner-perl
We might find some suggestions to tuning the DB that will help with the backup of this one course.
'spirit of sharing', Ken
Hello,
first thanks a lot for your help
moosh -n course-info 141
Course ID: 141
No of contexts: 38
Context by level:
CONTEXT_MODULE (70): 38
Context by module:
label (12): 4
resource (17): 15
scorm (18): 18
certificate (23): 1
Number of role capability overwrites: 0
Role capability overwrites by context:
Number of local role assignments: 0
Locally assigned roles by context:
Number of enrolled users: 22
Users enrolled by role:
student (5): 22
Number of groups: 0
Group statistics:
Min number of members in a group: 0
Max number of members in a group: 0
Avg number of members in a group: 0
Course modinfo size: 73833
Number of sections: 6
Section statistics:
Sections visible: 6
Sections hidden: 0
Min number of modules in a section: 0
Max number of modules in a section: 11
Avg number of modules in a section: 6
Number of grades: 418
Number of log entries: 8092
Number of files: 1851
Total file size: 1252162926
Cache build time: 0.071723937988281
moosh -n file-list course=141
!!! Error al leer de la base de datos !!!
it is the same issue
I see some things in course info worthy of note:
scorm (18): 18
SCORM installs usually had one upload a zip and unzip then link to the manifest.xml. Those SCORM's could have been uploaded to a file system repo and then from some reason, course creator, check the box for 'alias'. Have seen courses with SCORMs in courses that have a lot more files than the next bit of info.
Number of files: 1851
Total file size: 1252162926 - which indicates a backup would be around 1.2 Gig - probably higher. SCORM's with videos (mp4's) could be very large.
moosh wasn't gonna fix anything ... just tell us where we might need to look.
And it sure did tell us with the command to check files in the course ... there was an issue ... hmmmm, same one Error writing to the DB.
In the backup process, there are temp tables created which are erased when the session finishes ... sucessfully or in failure.
So when the command to list files error'd about how long in time did it take to error. Almost right away?
In moodledata/temp/backup/ there might be some clues in .log files that are larger then 0 bytes. If there are directories in there by a contenthash name, check them out also. A failed backup won't have a moodle_backup.xml file in there ... nor an .mbz file.
Do logs still say mysql has gone away? Might need to increase max packets even larger ... and look at increasing default value for number of open files.
And mysqltuner said? What recommendations did it make?
What if you made a backup but deselected a SCORM or two when backing up?
'spirit of sharing', Ken
Good afternoon:
the error "error reading in the database" is in a matter of 2 seconds if I make a backup of 3 of the modules it is generated correctly, and if I then backup the module 4 it is also generated without problems. In the backup / temp / folder there are no log files higher than 0k allusive to that backup.
the variable I increased to 1073741824, however I see something strange in the backup folder and it is that a folder is generated 207978a8a7bccca6fb8b29c172f3f303
where you have the content of the course support, including the file moodle_backup.xml and a .mbz of 1.9Gb but both in the course and with moosf the error continues.
thanks a lot for your help
The attached moodle_backup.xml is complete.
And if you have a .mbz file in that temp directory am almost willing to bet it's a valid/good backup file.
The last things moodle attempts to do is *copy* the .mbz file to destination ... which could be /moodledata/filedir/ and clean up those temp tables.
Back to the original error posting ... see:
moodle_temptables.php
moodle_temptables->dispose()
drop_table(Object(xmldb_table))
Those indicate trying to remove a temp table.
So suggest changing the db user and db password in config.php to the superuser and superuser password.
The user currently in config.php might not have priv's to do things like dropping temp tables.
BTW, your use of the term 'module' really doesn't equate directly to the way moodle makes backups
Take for example this entry in moodle_backup.xml
<setting>
<level>activity</level>
<activity>scorm_1940</activity>
<name>scorm_1940_included</name>
<value>1</value>
</setting>
If you put your mouse over the SCORM link it should show an ID number.
If you had your mouse over the SCORM linked to the 1940 SCORM it would show:
yoursite/mod/scorm/view.php?id=1940
So if using superuser creds don't fix this, you need to backup the course multiple times noting which SCORM link (module in your terms, I guess) caused the backup to fail. There's your culprit.
'spirit of sharing', Ken
Good afternoon
When I speak of module I mean a tab, or a topic (ontopic), within each of them are several resources (scorm, assessments, forums ...).
This error only happens with some courses in specific since with others it ends without problems.
As for several copies, I already did the individual procedure by tabs, individually all backups end up successful.
The user of the database has all the permissions
thanks a lot
Does the DB user in config.php of the site have ***FULL PRIVILEGES*** to the DB?
'spirit of sharing', Ken
Hello Ken
I configure the program backups and the courses that generated issues do not the backup.
Total 8 courses with issue and many ok (84)
Have you run MySQLtuner with super user creds?
Are you sure the DB user in moodle config actually has all priv's?
Do you know the superuser creds? If so, use them in the config.php file and try the command line backup of the troubled course.
From descriptions sounds like something messed up in that one course ... difficult to guess what!
Don't have an answer for ya!
'spirit of sharing', Ken