Hi all,
We are using the the fileprocess.php to import xml file for course creation apart from the Banner/Luminis Message Broker that is scheduled normally. the problem we are facing is that the import from the command:
"sudo -u apache php /var/www/html/moodle/enrol/lmb/cli/fileprocess.php"
is slow. for a 216K xml file is takes around 10 min, but i have a 14MB file and it is very slow more than 2 hours now.
the config i have on PHP.INI is:
max_execution_time = 300
; Maximum amount of time each script may spend parsing request data. It's a good
; idea to limit this time on productions servers in order to eliminate unexpectedly
; long running scripts.
; Note: This directive is hardcoded to -1 for the CLI SAPI
; Default Value: -1 (Unlimited)
; Development Value: 60 (60 seconds)
; Production Value: 60 (60 seconds)
; http://php.net/max-input-time
max_input_time = 60
; Maximum input variable nesting level
; http://php.net/max-input-nesting-level
;max_input_nesting_level = 64
; How many GET/POST/COOKIE input variables may be accepted
; max_input_vars = 1000
max_input_vars = 1000
; Maximum amount of memory a script may consume (128MB)
; http://php.net/memory-limit
memory_limit = 1024M
=======================================================================
and opcache:
; Determines if Zend OPCache is enabled
opcache.enable=1
; Determines if Zend OPCache is enabled for the CLI version of PHP
opcache.enable_cli=1
; The OPcache shared memory storage size.
opcache.memory_consumption=512
; The amount of memory for interned strings in Mbytes.
opcache.interned_strings_buffer=16
; The maximum number of keys (scripts) in the OPcache hash table.
; Only numbers between 200 and 100000 are allowed.
opcache.max_accelerated_files=32531
opcache.use_cwd=1
; When disabled, you must reset the OPcache manually or restart the
; webserver for changes to the filesystem to take effect.
opcache.validate_timestamps=1
; How often (in seconds) to check file timestamps for changes to the shared
; memory storage allocation. ("1" means validate once per second, but only
; once per request. "0" means always validate)
opcache.revalidate_freq=60
the mysql is configured to handle a lot of abuse and the server has 16Gb of memory mysql as per mysqltuner is setup to use a max of 7.6GB of memory.
innodb_file_format = Barracuda
innodb_file_per_table = 1
innodb_large_prefix
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
skip-character-set-client-handshake
default_storage_engine=InnoDB
sql-mode="STRICT_TRANS_TABLES,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"
log-output=FILE
general-log=0
general_log_file="/var/log/MariaDB/VSAPPDEV07-general.log"
slow-query-log=1
slow_query_log_file="/var/log/MariaDB/VSAPPDEV07-slow.log"
long_query_time=10
log-error="/var/log/MariaDB/VSAPPDEV07-error.log"
server-id=1
max_connections=1000
query_cache_size=200M
table_open_cache=2000
tmp_table_size=747M
thread_cache_size=40
myisam_max_sort_file_size=100G
myisam_sort_buffer_size=2G
key_buffer_size=128M
read_buffer_size=64K
read_rnd_buffer_size=256K
innodb_flush_log_at_trx_commit=1
innodb_log_buffer_size=16M
innodb_buffer_pool_size=5G
innodb_log_file_size=750M
innodb_thread_concurrency=8
innodb_autoextend_increment=64
innodb_buffer_pool_instances=5
innodb_concurrency_tickets=5000
innodb_old_blocks_time=1000
innodb_open_files=300
innodb_stats_on_metadata=0
innodb_checksum_algorithm=0
back_log=80
flush_time=0
join_buffer_size=256K
max_allowed_packet=100M
max_connect_errors=100
open_files_limit=4161
query_cache_type=0
sort_buffer_size=256K
table_definition_cache=1400
binlog_row_event_max_size=8K
sync_master_info=10000
sync_relay_log=10000
sync_relay_log_info=10000
i am using Moodle 3.5.2 (Build: 20180910)