Understanding course/category sortorder and fix_course_sortorder()

Understanding course/category sortorder and fix_course_sortorder()

by Dan Poltawski -
Number of replies: 17

Over the past few months i've been encountering numerous sites which have course reodering ordering crippled by completely crazy[1] course sortorder values.

I'm trying to understand what is causing this problem and how to fix it robustly (aware that fix_course_sortorder() function has been rewritten in HEAD).

So I am wondering if anyone can enlighten me a bit:

  • Describing a high-level overview of the purpose of fix_course_sortorder()?
  • How do course_category sortorder and course sortorder interact? Is this the source of massive sortorder values?
  • Can the crazy high course sortorder values not be fixed by renumering the sortorder incrementally from 1?

Any help/history on this would be appreciated!

[1] An example from today:

mysql> select sortorder from course order by sortorder limit 6;
+--------------------+
| sortorder          |
+--------------------+
|               1000 |
|               8011 |
|               8012 |
|               8013 |
| 136511462030056106 |
| 136511462030060000 |
+--------------------+


mysql> select max(sortorder) from course;                        
+--------------------+
| max(sortorder)     |
+--------------------+
| 136511462030240100 |
+--------------------+

mysql> select count(*) from course;
+----------+
| count(*) |
+----------+
|      451 | 
+----------+
Average of ratings: -
In reply to Dan Poltawski

Re: Understanding course/category sortorder and fix_course_sortorder()

by Jeffrey Silverman -
We've seen this. Is there a Moodle Tracker report for this problem?

Don't have a solution yet, though...
In reply to Jeffrey Silverman

Re: Understanding course/category sortorder and fix_course_sortorder()

by Rainer Herbst -
We have encountered the same problem on our production system (Moodle 1.8.9, MySQL). "sort by names" did not work, using the arrows did not change the sortorder.

As mentioned in previous answers, the reason is an overflow in the "sortorder" field of the mdl_course table. Even if SQL can handle the large numbers, PHP can not. In our case, the sortorder reached values over 13666533340000000000, which is really large!

At first, we fixed the problem with an SQL-Statement:
update mdl_course set sortorder=sortorder -  13666533340000000000 where sortorder >=  13666533340000000000;

Then, we searched the source code and stumbled over the "fix_course_sortorder()" function in lib/datalib.php, which is called every time the courses of a category are sorted. To be honest, I do not very like the way this function works. E.g., there is a "catgap" value of 1000 wired into the source code. When every category has its own sortorder, why do we need a catgap?

But the main thing is the following:
<snip>
      // $mustshift indicates whether the sequence must be shifted to
    // meet its range
    $mustshift = false;
    if ($min < $n+$tolerance || $min > $n+$tolerance+$catgap ) {
        $mustshift = true;
    }
<snip>

$n is normally set to $min some lines above, so $min will be less than $n+$tolerance, wouldn`t it? So every time the function is called the sortorder wull be shiftet by that $catgap of 1000. It is only a matter of time when it will overflow!

We changed that part, here is the diff:

    // $mustshift indicates whether the sequence must be shifted to
    // meet its range
    $mustshift = false;
-    if ($min < $n+$tolerance || $min > $n+$tolerance+$catgap ) {
+    if ($min < $n-$tolerance || $max > $n+$tolerance+$catgap ) {
        $mustshift = true;
    }

There is another minor problem with the calculation of the course order in course/category, here is the diff:

@@ -77,9 +77,16 @@
        if ($resort and confirm_sesskey()) {
            if ($courses = get_courses($category->id, "fullname ASC", 'c.id,c.fullname,c.sortorder')) {
                // move it off the range
-                $count = get_record_sql('SELECT MAX(sortorder) AS max, 1
+                # rherbst - must there be a shift of 100 for every sorting?
+                # old
+                # $count = get_record_sql('SELECT MAX(sortorder) AS max, 1
+                #                          FROM ' . $CFG->prefix . 'course WHERE category=' . $category->id);
+                #  $count = $count->max + 100;
+
+                # new
+                $count = get_record_sql('SELECT MIN(sortorder) AS min, 1
                                          FROM ' . $CFG->prefix . 'course WHERE category=' . $category->id);
-                $count = $count->max + 100;
+                $count = $count->min;
                begin_sql();
                foreach ($courses as $course) {
                    set_field('course', 'sortorder', $count, 'id', $course->id);


In the old version, every time you clicked "resort by names"-button, the sortorder will increase in steps of 100. In the new version, we will start with the minimum sortorder value and increase the value by one for the next course.

Untill yet, everything seems to work correctly on our test and production system.

As far as I know, version 1.8.x will not receive bug fixes, so I will not bother to create a bug tracker entry. Hope the diffs show you what do do!
Average of ratings: Useful (1)
In reply to Rainer Herbst

Re: Understanding course/category sortorder and fix_course_sortorder()

by Ângelo Rigo -
We have the same problem here if you create a entry in the tracker i can vote for it.

Also i will test your solution.
In reply to Rainer Herbst

Re: Understanding course/category sortorder and fix_course_sortorder()

by Paolo Oprandi -
I have just discovered a similar problem in 1.95. The sort works up to a point, but for some categories is failing. I haven't worked out the conditions, which break it yet.

Although I can identify the same lines of code, your fix doesn't work for our problem though.
In reply to Rainer Herbst

Re: Understanding course/category sortorder and fix_course_sortorder()

by Paolo Oprandi -
Rainer,

Okay - sorry maybe this is the solution. But how do we reset our sort orders? As you say the catgap is unneccessary. I have been looking at taking each category lowest sortorder number and running:

update mdl_course set sortorder=sortorder - [ lowest category sortorder ] where sortorder >= [ lowest category sortorder ] and category = [ cat id ];

is that what you would advise? if so I'll make a script and post it here.

If not, our sortorder field ranges from:

2925439 to 82918385555801000

...so have you any advice how I should update it?

Btw, it is worth adding this to the bug tracker as its still a problem in 1.9.

thanks,

Paolo
In reply to Paolo Oprandi

Re: Understanding course/category sortorder and fix_course_sortorder()

by Rainer Herbst -
Hi!

is [lowest category sortorder] the lowest sortorder within that category or the lowest sortorder of all categories? In the first case, you might get multiple identical sortorder values for different categories.

This sortorder mess is not really clear to me! sad

In your case, 82918385555801000 is the max. sortorder and it is probably close to the integer overflow. Simply substracting something like 82918385500000000 should do the job.

Regards!
Rainer
In reply to Rainer Herbst

Re: Understanding course/category sortorder and fix_course_sortorder()

by Paolo Oprandi -
Hi Rainer,

Thanks for your reply.

as you say, I will get identical sortorder values for different values, but as every category has its own sortorder they should be able to be the same, I think? The DB column doesn't specify it should be UNIQUE or anything.

Also, by just deleting from some of the records you could also get multiple identical sort orders. Probably it would be safer to delete the same value from all records if duplicate values is a problem.

Cheers!
Paolo
In reply to Paolo Oprandi

Re: Understanding course/category sortorder and fix_course_sortorder()

by Paolo Oprandi -
If my theory is correct (that it doesn't matter if sortorders are the same in different categories) then this script will update your site:

include('config.php');
$cats=get_records('course_categories');
foreach ($cats as $cat) {
     $get_min_sortorder_for_cat_sql = 'SELECT MIN( sortorder ) as lowest_sort FROM {$CFG->PREFIX}course where category = {$cat->id}';
     $min_sortorder_for_cat=get_record_sql($get_min_sortorder_for_cat_sql);
     $sql = "update {$CFG->PREFIX}course set sortorder=sortorder - {$min_sortorder_for_cat->lowest_sort} where sortorder >= {$min_sortorder_for_cat->lowest_sort} and category = {$cat->id}";
     execute_sql($sql);
}

I suggest making the changes Rainer advises then running it.
In reply to Dan Poltawski

Re: Understanding course/category sortorder and fix_course_sortorder()

by Rosario Carcò -
I migrated my production server from a physical one to a virtual one. So I copied Moodle's data and htdocs directories to the new server. Dumped the databases and imported them on the new one.

Everything worked fine, but today I can not sort the courses and not even change the order in Edit Mode.

A look into the database with phpMyAdmin showed that a lot of courses have the same very high sort oder number.

I will check on the old production server to see whether the numbers were messed up there or after I imported the dump on the new server.

I already knew there is a function to fix the sort numbers but at the moment I do not see how to call it.

In the screenshot I sorted DESCENDING (from the highest to the lowest number), but the messed numbers could mean that there was an overflow in that field, which could be the bug we are looking for.

Rosario
Attachment messedSortOrder.gif
In reply to Dan Poltawski

Re: Understanding course/category sortorder and fix_course_sortorder()

by Rosario Carcò -
If I correctly understand the following code snippets from fix_course_sortorder(), the numbers are not checked to avoid an overflow:

$info = get_record_sql('SELECT MIN(sortorder) AS min,
 MAX(sortorder) AS max,
 COUNT(sortorder) AS count
 FROM ' . $CFG->prefix . 'course
 WHERE category=' . $categoryid);
:
:
// UPDATE course SET sortorder=sortorder+$shift
execute_sql("UPDATE {$CFG->prefix}course
 SET sortorder=sortorder+$shift
 WHERE category=$categoryid", 0);
$n = $n + $catgap + $count;
} else { // do it slowly
$n = $n + $catgap;
// if the new sequence overlaps the current sequence, lack of transactions
// will stop us -- shift things aside for a moment...
if ($safe || ($n >= $min && $n+$count+1 < $min && $CFG->dbfamily==='mysql')) {
 $shift = $max + $n + 1000;
 execute_sql("UPDATE {$CFG->prefix}course
 SET sortorder=sortorder+$shift
 WHERE category=$categoryid", 0);
}

Rosario
In reply to Dan Poltawski

Re: Understanding course/category sortorder and fix_course_sortorder()

by Rosario Carcò -
I fixed the sortorder with sql-queries in phpMyAdmin last friday:

UPDATE mdl_course SET sortorder = sortorder - 1772075502951000000 WHERE sortorder > 1772075502951000000

As I told, I had big numbers like: 1772075502951432704

I repeated the procedure until I had sortorders in the range 1000..10'000

For a moment everything seemed to be Ok and today I have again sortorder numbers like: 149392843888854943, which prevents the course-lists to be sorted alphabetically or arranged in any other order.

So IS THERE SOMEBODY OUT, willing to help us track this down and corrct the behaviour of fix_course_sortorder()

Any help would be appreciated, Rosario

In reply to Rosario Carcò

Re: Understanding course/category sortorder and fix_course_sortorder()

by Anne Garnavault Remy -
hello,
I have the same problem on my moodle and no solutions!!! no help on the french forum diabolique
In reply to Anne Garnavault Remy

Re: Understanding course/category sortorder and fix_course_sortorder()

by Ed Cetera -

We also had this problem and found a hint on this forum post:

http://moodle.org/mod/forum/discuss.php?d=126896

Checked sortorder with:

SELECT MIN(sortorder) AS min, MAX(sortorder) AS max, COUNT(sortorder) AS count from mdl_course

We implemented and are testing this simple fix (Win2k, Apache2, MySQL51,PHP523).

UPDATE mdl_course SET sortorder=ID +1000

In reply to Ed Cetera

Re: Understanding course/category sortorder and fix_course_sortorder()

by Robert Russo -
WE HAVE NOT TESTED THESE CHANGES WITH ANYTHING EXCEPT FOR 1.9.2.

We dealt with this problem a long time ago. Here is our fix for 1.9.2. It is by no means perfect, but it got us from 3 minute course creations down to instant course creations and deletions.

Replace your fix_course_sortorder function with the one below and then update everything that calls fix_course_sortorder to pass in the courseid or categoryid of the course in question in order to update category counts or update sort order appropriately.

WE HAVE NOT TESTED THESE CHANGES WITH ANYTHING EXCEPT FOR 1.9.2.
Please test accordingly.
function fix_course_sortorder($categoryid=0, $n=0, $safe=0, $depth=0, $path='', $courseid=0) {

 global $CFG;

 // adamzap

 // We only want to sort things for specific requests.
 // We are not concerned with cleaning up the system all the time.
 if ($categoryid == 0 or $courseid < 3) {
 return 0;
 }

 // Update the course count for this category and its parents (if it has any)
 do {
 $course_count_sql = 'SELECT count(id) FROM ' . $CFG->prefix . 'course ' .
 'WHERE category = ' . $categoryid;
 $course_count = get_field_sql($course_count_sql);

 set_field('course_categories', 'coursecount', $course_count, 'id', $categoryid);

 $categoryid = get_field('course_categories', 'parent', 'id', $categoryid);
 } while ($categoryid != 0);

 // Update the sort order of the passed in courseid
 $max_sortorder_sql = "SELECT MAX(sortorder) from mdl_course";
 $max_sortorder = get_field_sql($max_sortorder_sql);

 $sortorder_sql = 'UPDATE ' . $CFG->prefix . 'course c1 ' .
 'SET c1.sortorder = ' . $max_sortorder . ' + ' . rand(1, 100) .
 ' WHERE c1.id = ' . $courseid;

 execute_sql($sortorder_sql, 0);

 return 1;
}



We reset our sort-orders in a similar way as above.
UPDATE mdl_course SET sortorder=ID +1000

With the new function and changes to about a 1/2 dozen other files everything works great, just much faster and no db limit issues.

WE HAVE NOT TESTED THESE CHANGES WITH ANYTHING EXCEPT FOR 1.9.2.
In reply to Robert Russo

Re: Understanding course/category sortorder and fix_course_sortorder()

by Julien Cros -

Hi,

Does this fix may work for a 1.9.4+ instance of Moodle..?

As we're heavily modifying the tree of categories, we have the sortorder issue about every 2/3 days... mixed

Thanks in advance for any insight or updated fix..

Julien

In reply to Ed Cetera

Re: Understanding course/category sortorder and fix_course_sortorder()

by Rosario Carcò -
The problem with this solution (UPDATE mdl_course SET sortorder=ID +1000) is, that as soon as new courses are created, everything gets messed up again.

Thus a patch to the
fix_course_sortorder
function is the best way to go. But why is this patch not yet implemented in 1.9.9+ ?

Rosario