Upgrade: mod-quiz duplicate key row error

Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -
Number of replies: 48

 

I'm upgrading from 1.9.19 to 2.2.5. (using a duplicate of the database/data).

Get the error below on mod_quiz ...

I can find and remove the duplicate but then it just finds another, and another.  I gave up after four, I could be here for weeks. 

What's causing this (I'm guessing this doesn't happen very often, I can only find one other thread about this and that didn't help except to tell me how to clear it for the one error)?

Is there a way I can clean up the whole thing in one go? 

Many thanks for any help.

Debug info: SQLState: 23000<br>
Error Code: 2601<br>
Message: [Microsoft][SQL Server Native Client 10.0][SQL Server]Cannot insert duplicate key row in object 'dbo.mdl_question_attempt_steps' with unique index 'mdl_quesattestep_queseq_uix'.<br>
SQLState: 01000<br>
Error Code: 3621<br>
Message: [Microsoft][SQL Server Native Client 10.0][SQL Server]The statement has been terminated.<br>

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1315316991','1047',N'todo','0','21166')
[array (
0 => NULL,
1 => '1315316991',
2 => '1047',
3 => 'todo',
4 => 0,
5 => 21166,
)]

Stack trace:
  • line 397 of \lib\dml\moodle_database.php: dml_write_exception thrown
  • line 256 of \lib\dml\sqlsrv_native_moodle_database.php: call to moodle_database->query_end()
  • line 372 of \lib\dml\sqlsrv_native_moodle_database.php: call to sqlsrv_native_moodle_database->query_end()
  • line 885 of \lib\dml\sqlsrv_native_moodle_database.php: call to sqlsrv_native_moodle_database->do_query()
  • line 963 of \lib\dml\sqlsrv_native_moodle_database.php: call to sqlsrv_native_moodle_database->insert_record_raw()
  • line 300 of \question\engine\upgrade\upgradelib.php: call to sqlsrv_native_moodle_database->insert_record()
  • line 259 of \question\engine\upgrade\upgradelib.php: call to question_engine_attempt_upgrader->insert_record()
  • line 222 of \question\engine\upgrade\upgradelib.php: call to question_engine_attempt_upgrader->save_usage()
  • line 177 of \question\engine\upgrade\upgradelib.php: call to question_engine_attempt_upgrader->convert_quiz_attempt()
  • line 132 of \question\engine\upgrade\upgradelib.php: call to question_engine_attempt_upgrader->update_all_attempts_at_quiz()
  • line 1112 of \mod\quiz\db\upgrade.php: call to question_engine_attempt_upgrader->convert_all_quiz_attempts()
  • line 540 of \lib\upgradelib.php: call to xmldb_quiz_upgrade()
  • line 271 of \lib\upgradelib.php: call to upgrade_plugins_modules()
  • line 1437 of \lib\upgradelib.php: call to upgrade_plugins()
  • line 269 of \admin\index.php: call to upgrade_noncore()
تێکرایى نمرەپێدراوەکان: -
In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

I am sure someone else hit this problem, but I cannot find the tracker issue or forum thread now.

Basically, what is going on is that due to old bugs, some bits of quiz attempt data in your database are meaningless, and that confuses the upgrade code.

In this case, for one particular 'question_attempt' (in Moodle 2.1+ language. In Moodle 1.9/2.0 these were called question_sessions) there are two 'start' states. Obviously the a given attempt at a question should only be started once.

To find problem data in your Moodle 1.9 database, try a query like this:

SELECT question, attempt, seq_number, COUNT(1) AS numduplicates
FROM mdl_question_states
GROUP BY question, attempt, seq_number
HAVING numduplicates > 1

That query should return zero rows. (There should logically have been a unique index on (question, attempt, seq_number) in Moodle 1.9, but there wasn't.

تێکرایى نمرەپێدراوەکان:Useful (2)
In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Thank you, my SQL is *very* rusty, but that gives an SQL error:

Invalid column name 'numduplicates'.

I'm using Microsoft SQL 2008 if that makes any difference?  (Wasn't my choice.)

Edit: This was the other thread.  it helped me remove the individual rows but I was hoping for something to clean the whole thing.

https://moodle.org/mod/forum/discuss.php?d=196599

Edit2: I tried this, got 465 rows.

SELECT question, attempt, seq_number,
COUNT (seq_number)AS NumOccurrences
FROM mdl_question_states
GROUP BY question, attempt, seq_number
HAVING (COUNT(seq_number)> 1 )

No idea if it's right, or what to do now.

تێکرایى نمرەپێدراوەکان:Useful (1)
In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

That was the right way to fix the HTML. Sorry I got it wrong. Well done for finding the other forum thread.

What you need to do now is to look at those problem attempts, and try to work out if they have any value. Try this query: http://docs.moodle.org/dev/Question_database_structure#Following_what_happens_when_questions_are_attempted edited to have a sutitable WHERE clause. For example

WHERE quiza.uniqueid = {attempt value from the other query}

If you do that, or just find quiz_attempts.id for quiz_attempts.uniqueid matching one of the problem attempt numbers, then you can got to .../mod/quiz/review.php?attempt={quiz_attempts.id} in your Moodle 1.9 site to see what that quiz attempt is.

If you are lucky, you will find all those attempts are reall old, no longer need to be kept, and you can just delete them using the Moodle quiz reports.

If you find those attempts are still relevant, then you may need to try to sort out the data manually in your database before you upgrade. If that is necessary, and if you can give us more information about what the data looks like, then we may be able to help you sort it out.

تێکرایى نمرەپێدراوەکان:Useful (2)
In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Excellent, thank you.  I'm sure I'll be back with more questions once I've dug a bit deeper.

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Okay, I'm struggling a bit to get my head around this if I'm honest.  I need to go back a few steps and try and understand Moodle a bit better (not really on this thread) and probably brush up my SQL.

However, before I go that far, is it significant that so far, every one of these I get has a NULL in the fraction field?

Does this mean

a) The row can be safely removed without worry? (One can only hope)

b) I can automatically find and delete all of these entries?

 

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

I am afraid that you are asking me a detail about the Moodle 1.9 database structure that I can no longer remember.

But having had a quick look at the Moodle 1.9 code, that is very suspicious. Moodle 1.9 did initialise the grade to 0 when you started a question, so it should never be blank.

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Thanks for the reply anyway.

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن María José Blanes -

Hi!

We had the same problem upgrading 2.2.3 to 2.2.5 using a duplicate of the database.
In our case, the sequence that 'mdl_question_attempt_steps' uses for generating the primary key was out of date.

For instance, the next sequence number for the primary key in 'mdl_question_attempt_steps' was 10 whereas the last ID in the mdl_question_attempt_steps table was 20.
When moodle was trying to insert a new row, the value 10 was just used in the primary key, and that caused the error.

We solved this problem increasing the sequence 'mdl_quesattestep_id_seq'. Its value must be bigger than the last value in the ID field of the mdl_question_attempt_steps table (bigger than 20 in the previous example).
Alfter that, we must solve the same problem with the sequence 'mdl_quesattestepdata_id_seq' used for the 'mdl_question_attempt_steps_data' table.

We are almost sure that the origin of this problem was the duplicate of the database. Our moodle was working while we was doing the backup. So, some data could have desynchronized during the process.

I hope it will be helpful

تێکرایى نمرەپێدراوەکان:Useful (2)
In reply to María José Blanes

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Hey María,

This sounds very promising!

You say you solved it by "increasing the sequence", how exactly did you do that?

Many thanks.

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن María José Blanes -

Ok,
I'm not an expert with databases but I've asked my colleague to answer your question.

Well, there is not a direct way to say 'next value of the sequence will be 35'.
So, the best solution we could find was to increase the sequence until the value you need.
The good new is that you can increase the sequence 20 by 20, for example, in order to 1 by 1.

So, these are the steps you can follow for increasing the sequence:

1. Find the last ID value in the 'mdl_question_attempt_steps' table.
      SELECT MAX(ID) FROM mdl_question_attempt_steps;
   Imagine that it is 3520.

2. Find the last value in the sequence 'mdl_quesattestep_id_seq'. The instruction you can use is:
      SELECT LAST_NUMBER FROM USER_SEQUENCES WHERE SEQUENCE_NAME = mdl_quesattestep_id_seq';
   Imagine that it is 3515.

3. In this case you would need to increment the sequence in 5. To be really sure, we are going to increment the sequence in 6.
   So, now, modify the increment of the sequence in 6 with the intruction:
      alter sequence mdl_quesattestep_id_seq increment by 6;

4. Increment the sequence with the instruction:
      select mdl_quesattestep_id_seq.nextval from dual;
   Now, the sequence next value is 6 times bigger.

5. Restore the increment of the sequence to 1:
      alter sequence mdl_quesattestep_id_seq increment by 1;


And, that's all.
Remember you must do something similar with the 'mdl_question_attempt_steps_data' table and the 'mdl_quesattestepdata_id_seq' sequence, too.

تێکرایى نمرەپێدراوەکان:Useful (1)
In reply to María José Blanes

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

Thanks.

How did you get all this "sequencing" to work with the Moodle installer? Did you have to change any Moodle code?

Is your Moodle database an Oracle database? As far as I'm aware, Sequences are only available on the Oracle database platform.

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن María José Blanes -

Oh, yes, our database is Oracle. I though I had mentioned. Sorry!

We haven't changed any Moodle code. I think it's only a problem of the database duplication.

I'm not sure how this "sequencing" works in other databases. Maybe our solution can give you an idea of what is happening but, right now, I don't know how to fix it in other databases.

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

Steve,

What do you see when you go to Site administration / Development / XMLDB editor and run the Check foreign keys option?

On my 2.0.4+ (Build: 20110907) Moodle I get a list of 19 violated foreign keys:

Violated foreign keys found:

  • Foreign key question on table question_answers is violated by 721 out of 219726 rows.
    SELECT * FROM {question_answers} t1 LEFT JOIN {question} t2 ON t1.question = t2.id WHERE t1.question <> ? AND t2.id IS NULL; (0)
  • Foreign key question on table question_states is violated by 6511 out of 6374615 rows.
    SELECT * FROM {question_states} t1 LEFT JOIN {question} t2 ON t1.question = t2.id WHERE t1.question <> ? AND t2.id IS NULL; (0)
  • Foreign key questionid on table question_sessions is violated by 3325 out of 2600065 rows.
    SELECT * FROM {question_sessions} t1 LEFT JOIN {question} t2 ON t1.questionid = t2.id WHERE t1.questionid <> ? AND t2.id IS NULL; (0)
  • Foreign key newest on table question_sessions is violated by 25 out of 2600065 rows.
    SELECT * FROM {question_sessions} t1 LEFT JOIN {question_states} t2 ON t1.newest = t2.id WHERE t1.newest <> ? AND t2.id IS NULL; (0)
  • Foreign key newgraded on table question_sessions is violated by 25 out of 2600065 rows.
    SELECT * FROM {question_sessions} t1 LEFT JOIN {question_states} t2 ON t1.newgraded = t2.id WHERE t1.newgraded <> ? AND t2.id IS NULL; (0)
  • Foreign key parent on table grade_categories is violated by 2 out of 411 rows.
    SELECT * FROM {grade_categories} t1 LEFT JOIN {grade_categories} t2 ON t1.parent = t2.id WHERE t1.parent IS NOT NULL AND t2.id IS NULL;
  • Foreign key rawscaleid on table grade_grades is violated by 36 out of 597362 rows.
    SELECT * FROM {grade_grades} t1 LEFT JOIN {scale} t2 ON t1.rawscaleid = t2.id WHERE t1.rawscaleid IS NOT NULL AND t2.id IS NULL;
  • Foreign key parent on table grade_categories_history is violated by 1115 out of 6975 rows.
    SELECT * FROM {grade_categories_history} t1 LEFT JOIN {grade_categories} t2 ON t1.parent = t2.id WHERE t1.parent IS NOT NULL AND t2.id IS NULL;
  • Foreign key courseid on table grade_items_history is violated by 7896 out of 42279 rows.
    SELECT * FROM {grade_items_history} t1 LEFT JOIN {course} t2 ON t1.courseid = t2.id WHERE t1.courseid IS NOT NULL AND t2.id IS NULL;
  • Foreign key categoryid on table grade_items_history is violated by 10187 out of 42279 rows.
    SELECT * FROM {grade_items_history} t1 LEFT JOIN {grade_categories} t2 ON t1.categoryid = t2.id WHERE t1.categoryid IS NOT NULL AND t2.id IS NULL;
  • Foreign key rawscaleid on table grade_grades_history is violated by 875 out of 1132667 rows.
    SELECT * FROM {grade_grades_history} t1 LEFT JOIN {scale} t2 ON t1.rawscaleid = t2.id WHERE t1.rawscaleid IS NOT NULL AND t2.id IS NULL;
  • Foreign key usermodified on table grade_grades_history is violated by 6 out of 1132667 rows.
    SELECT * FROM {grade_grades_history} t1 LEFT JOIN {user} t2 ON t1.usermodified = t2.id WHERE t1.usermodified IS NOT NULL AND t2.id IS NULL;
  • Foreign key blockid on table block_instance_old is violated by 193 out of 3651 rows.
    SELECT * FROM {block_instance_old} t1 LEFT JOIN {block} t2 ON t1.blockid = t2.id WHERE t1.blockid <> ? AND t2.id IS NULL; (0)
  • Foreign key quiz on table quiz_attempts is violated by 56 out of 89026 rows.
    SELECT * FROM {quiz_attempts} t1 LEFT JOIN {quiz} t2 ON t1.quiz = t2.id WHERE t1.quiz <> ? AND t2.id IS NULL; (0)
  • Foreign key scorm_scoes_data_scoid on table scorm_scoes_data is violated by 6 out of 54 rows.
    SELECT * FROM {scorm_scoes_data} t1 LEFT JOIN {scorm_scoes} t2 ON t1.scoid = t2.id WHERE t1.scoid <> ? AND t2.id IS NULL; (0)
  • Foreign key scoid on table scorm_scoes_track is violated by 1 out of 347 rows.
    SELECT * FROM {scorm_scoes_track} t1 LEFT JOIN {scorm_scoes} t2 ON t1.scoid = t2.id WHERE t1.scoid <> ? AND t2.id IS NULL; (0)
  • Foreign key question on table question_multichoice is violated by 179 out of 50118 rows.
    SELECT * FROM {question_multichoice} t1 LEFT JOIN {question} t2 ON t1.question = t2.id WHERE t1.question <> ? AND t2.id IS NULL; (0)
  • Foreign key question on table question_numerical_options is violated by 1 out of 5 rows.
    SELECT * FROM {question_numerical_options} t1 LEFT JOIN {question} t2 ON t1.question = t2.id WHERE t1.question <> ? AND t2.id IS NULL; (0)
  • Foreign key question on table question_truefalse is violated by 72 out of 15123 rows.
    SELECT * FROM {question_truefalse} t1 LEFT JOIN {question} t2 ON t1.question = t2.id WHERE t1.question <> ? AND t2.id IS NULL; (0)

Tim, what does this mean? Can it be related to the "Cannot insert duplicate key row in object dbo.mdl_question_attempt_steps" error?

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

I don't know how to do that Luis.  I only have a working 1.9 install, can I do that with 1.9?

Thanks for the info Maria, I only wish I could do something with it.

I've more or less given up on the upgrade path and am now looking at starting a clean install of 2 but even that isn't working.

https://moodle.org/mod/forum/discuss.php?d=215150

 

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

I'm not sure if it will work in 1.9 but go to:

http://yourschool.com/admin/xmldb/

What do you see? You should see some links like:

[Reserved words] [Doc] [Check indexes] [Check defaults] [Check foreign keys]

If you do, click the Check indexes an Check foreign keys links as see if you key any violation messages.

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

Thanks Tim.

So the query will be:

SELECT
 mdl_question_states.id,
 mdl_question_states.attempt,
 mdl_question_states.question,
 mdl_question_states.seq_number,
 CASE WHEN mdl_question_states.id = mdl_question_sessions.newest THEN 'newest' END AS isnewest,
 CASE WHEN mdl_question_states.id = mdl_question_sessions.newgraded THEN 'newgraded' END AS isnewgraded,
 CASE mdl_question_states.event
  WHEN 0 THEN '0 (open)'
  WHEN 2 THEN '2 (save)'
  WHEN 3 THEN '3 (grade)'
  WHEN 5 THEN '5 (validate)'
  WHEN 6 THEN '6 (close&grade)'
  WHEN 7 THEN '7 (submit)'
  WHEN 8 THEN '8 (close)'
  WHEN 9 THEN '9 (manualgrade)'
  ELSE CAST(mdl_question_states.event AS varchar)
 END AS event,
 mdl_question_states.answer,
 mdl_question_states.raw_grade,
 mdl_question_states.grade,
 mdl_question_states.penalty,
 mdl_question_sessions.sumpenalty,
 mdl_question_sessions.manualcomment,
 mdl_question_states.timestamp,
 mdl_quiz_attempts.id as quiz_attempts_id,  -- extra column, see last question below
 mdl_quiz_attempts.uniqueid as quiz_attempts_uniqueid  -- extra column, see last question below
FROM mdl_question_states
JOIN mdl_question_sessions ON mdl_question_states.question = mdl_question_sessions.questionid
AND mdl_question_states.attempt = mdl_question_sessions.attemptid
JOIN mdl_quiz_attempts ON mdl_quiz_attempts.uniqueid = mdl_question_sessions.attemptid
WHERE mdl_quiz_attempts.uniqueid = 7250  -- this is one of the 'attempt' values from the other query
ORDER BY mdl_question_states.attempt, mdl_question_states.question, mdl_question_states.seq_number;

Is that correct?

How do you interpret the data that this query returns? I assume it shows you the questions (and the students' answers) for quiz attempt 7250.

But what if 7250 is a valid quiz attempt? You said that "if you find those attempts are still relevant, then you may need to try to sort out the data manually in your database before you upgrade". How do you do that? What do you have to fix if this is a valid quiz attempt?

And does it mean anything if the values in mdl_quiz_attempts.id and mdl_quiz_attempts.uniqueid are different, example, mdl_quiz_attempts.id =  60964 and mdl_quiz_attempts.uniqueid = 60965?

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

"this is one of the 'attempt' values from the other query"

Which other query Luis?

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

The one that you origionally posted and shows the duplicates:

SELECT question, attempt, seq_number,
COUNT (seq_number) AS NumOccurrences
FROM mdl_question_states
GROUP BY question, attempt, seq_number
HAVING (COUNT(seq_number) > 1)

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Hmm.

When I run the first query based on the error message I get 465 results. When I put the first attempt value into the second query I get 42 results.

Every one of those 42 results has the same quiz_attmepts_id and quiz_attempts_uniqueid (the same value that I took from the first query). The id column is unique for each row.

What's going on? Is my Moodle totally broken? (It all seems to be running okay (1.9).)

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Okay, so that attempt number (from the first query) does resolve to an attempt in Moodle. (So why do I need the second query?)

But I repeat Luis' question:

What do I do with it?  It looks fine to me, and it looks like it needs to stay, so how do I actually resolve the issue?

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

As was stated above this query gives you the records that "have two start dates":

SELECT
 question,
 attempt,
 seq_number,
 COUNT (seq_number)AS NumOccurrences
FROM mdl_question_states
GROUP BY
 question,
 attempt,
 seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

How do I get the mdl_question_states.id value for each of the records that the above query returns (the PK for each of the records)?

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

Having run the above query, do something like

SELECT * FROM mdl_question_states WHERE question = ? AND attempt = ? ORDER BY id

Replace the two ? by appropriate vaues.

تێکرایى نمرەپێدراوەکان:Useful (1)
In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

Thanks Tim.

Applying one of the id's to:

SELECT *
FROM mdl_question_states
WHERE question = 360
AND attempt = 8418
ORDER BY id;

gives me this data:

id attempt question seq_number answer timestamp event grade raw_grade penalty
476526 8418 360 0  1245680767 0 0.0000000 0.0000000 0.0000000
476541 8418 360 1 929 1245680768 3 0.0000000 0.0000000 1.0000000
476633 8418 360 1 929 1245680768 3 0.0000000 0.0000000 1.0000000

The id = 476541 and 476633 records are identical which, based on all the info that you've provided, I suspect is where the problem is. There should only be one record, so I need to remove one of them.

The only thing is that the origional query you posted:

SELECT question, attempt, seq_number, COUNT (seq_number)AS NumOccurrences
FROM mdl_question_states
GROUP BY question, attempt, seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

gives me 6300 records, which I suspect are all the same kind of bad duplicates. And removing them individually one by one will take a while!

So an easier way, I think, is to get the mdl_question_states.id for each of those duplicate rows and then remove them using a DELETE FROM mdl_question_states WHERE id IN (all,those,6300,record,id's) command. (Perhaps I'll do it in smaller batches instead of all 6300 at once!)

Then I can try the upgrade to 2.2 and see if Moodle successfully manages to upgrade all those quiz attempts. The problem now is how do I get all those 6300 mdl_question_states.id values?

FYI, I installed the qeupgradehelper and if I exclude all the quizzes on the site, except for a few that I know don't have any duplicate records on mdl_question_states, then the Moodle 2.2 upgrade completes successfully. Then when I upgrade the quizzes individually the ones containing the duplicates in the mdl_question_states table continue to crash with the "Cannot insert duplicate key row in object 'dbo.mdl_question_attempt_steps' with unique index 'mdl_quesattestep_queseq_uix'" message. And those quizzes without the duplicates appear to upgrade successfully.

So, the current issue is how to get those 6300 mdl_question_states.id values.

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

Well, you just need to write the right SQL

SELECT
 MAX(id)
 question,
 attempt,
 seq_number,
 COUNT (seq_number) AS NumOccurrences
FROM mdl_question_states
GROUP BY
 question,
 attempt,
 seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

Will get you the higher numbered duplicate of each set. I guess those are the ones you want ot delete. If you have any cases where there are three duplciated rows, not just two, this wll not be enough.

If you are brave (and have a backup) you can try soemthing like

DELETE FROM mdl_question_states WHERE id IN (
    SELECT MAX(id) FROM mdl_question_states
    GROUP BY question, attempt, seq_number
    HAVING (COUNT(seq_number) > 1 )
)

Note, I have not tested this. Use at your own risk.

تێکرایى نمرەپێدراوەکان:Useful (1)
In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

Aaah! Thanks!

The "HAVING (COUNT(seq_number) > 1))" is where I was going wrong! تەریق دەبیتەوە

The first query gives me 6287 rows:

SELECT
 question,
 attempt,
 seq_number,
 COUNT (seq_number)AS NumOccurrences
FROM mdl_question_states
GROUP BY
 question,
 attempt,
 seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

And:

SELECT
 MAX(id),
 question,
 attempt,
 seq_number,
 COUNT (seq_number) AS NumOccurrences
FROM mdl_question_states
GROUP BY question, attempt, seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

also gives me 6287 rows which, I think, is a good thing.

So I'm going to execute that DELETE command to remove those 6287 rows. Then I'll try doing the 2.2 upgrade again and see if that solves the problem. And yes, it will most certainly be on a test machine!

THANKS for all your help so far.

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

I deleted the 6287 rows that the queries below returned:

SELECT
 question,
 attempt,
 seq_number,
 COUNT (seq_number)AS NumOccurrences
FROM mdl_question_states
GROUP BY
 question,
 attempt,
 seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

And:

SELECT
 MAX(id),
 question,
 attempt,
 seq_number,
 COUNT (seq_number) AS NumOccurrences
FROM mdl_question_states
GROUP BY question, attempt, seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

Now they both show 0 records.

So then I tried upgrading the quiz attempts via the qeupgradehelper. The quiz upgrades continue to crash with errors like:

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES (NULL,'1283418267','9774',N'todo','0','0')

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1362825781','50525',N'todo','0','0')

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1348582545','48077',N'todo','0','0')

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1367325199','45265',N'todo','0','0')

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1332487368','43666',N'todo','0','0')

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1303827253','11889',N'todo','0','0')

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1345118985','46810',N'todo','0','0')

It's interesting that sequencenumber and questionattemptid always have the value '0'. But, this query

SELECT
 question,
 attempt,
 seq_number
FROM mdl_question_states
WHERE seq_number = '0'
AND question = '0'

returns 0 rows so I'm not sure where Moodle is getting those '0' values that it is using in the INSERT INTO mdl_question_attempt_steps command. 

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

You are right. That makes no sense. questionattemptid should never be 0. Which version of qeupgradehelper (& moodle) are you using? Could there be a bug in the PHP code?

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

I upgraded Moodle:

$release = '2.0.4+ (Build: 20110907)';
to: 
$release = '2.2.10 (Build: 20130513)';

using qeupgradehelper $plugin->version = 2011092500;

So I'm now trying to upgrade the 2.0.4+ quiz attempts to 2.2.10 using version 2011092500 of the qeupgradehelper.

Perhaps I need to go from 2.0.4+ to 2.1.10 to 2.2.10 instead...

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

I've tried upgrading from 2.0.4+ to 2.1.10 and then to 2.2.10 and also directly from 2.0.4+ to 2.2.10 (on MSSQL 2008 R2). Both attempts produce the same results - the upgrade crashes with the "Cannot insert duplicate key row in object 'dbo.mdl_question_attempt_steps' with unique index 'mdl_quesattestep_queseq_uix'" error. And this is AFTER I have already DELETED all the duplicate records returned by the original query that Tim suggested earlier in this thread (SELECT question,  attempt,  seq_number, COUNT (seq_number)AS NumOccurrences FROM mdl_question_states GROUP BY  question,  attempt, seq_number HAVING (COUNT(seq_number) > 1) ) .

If I now run that SELECT query I get zero results implying, I presume, that there should be no more "duplicates" in mdl_question_states. But if there are supposedly no more duplicates in that table why does the above duplicate row error still occur?

The ONLY way I've been able to upgrade from version 2.0.4+ to version 2.2.10 without getting this error is to DISABLE the mdl_quesattestep_queseq_uix index on the new mdl_question_attempt_steps table that is created in 2.1 or 2.2.

But then when I try to rebuild that mdl_quesattestep_queseq_uix index after the 2.2.10 (or 2.3 or 2.5) upgrade is complete I get a "Rebuild failed for Index mdl_quesattestep_queseq_uix" error:

"The CREATE UNIQUE INDEX statement terminated because a duplicate key was found for the object name 'dbo.mdl_question_attempt_steps' and the index name 'mdl_quesattestep_queseq_uix'. The duplicate key value is (0, 0)."

Thus disabling that index, just to get past the upgrade, is not an option.

So what are my other options for cleaning out the supposed duplicates? How do I determine what these remaining duplicates are?

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

That SELECT gives me the ID's for the duplicate rows, example: attempt = 16971. Great...

Checking the data using:

SELECT *
FROM mdl_question_states
WHERE attempt = 16971
ORDER BY question,seq_number,id

gives me two duplicate rows:

id attempt question seq_number answer timestamp event grade raw_grade penalty
945279 16971 9214 0 29785,29784,29783: 1257752688 0 0.0000000 0.0000000 0.0000000
945301 16971 9214 1 29785,29784,29783:29785 1257752688 2 0.0000000 1.0000000 0.0000000
945303 16971 9214 1 29785,29784,29783:29785 1257752701 2 0.0000000 1.0000000 0.0000000
945329 16971 9214 2 29785,29784,29783:29785 1257752716 6 1.0000000 1.0000000 0.0000000

Those two duplicate rows are identical, except for the timestamp. Which of those two records do I delete? Or rather, how do I determine which one I should keep?

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

If they are identical, delete either. 

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

The timestamp is different so, technically, they are not identical. Which one do I delete?

Is there an easier way to cleanup mdl_question_states, considering that there are 8 million rows in that table (as described above)?

What are the implications of disabling the mdl_quesattestep_queseq_uix index?

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

It does not matter which of those two rows you delete. Flick a coin or something چاوداگرتن

The point of the mdl_quesattestep_queseq_uix is to prevent a mess like this ever happening again. What would happen if you deleted it? I don't know. It might do no real harm, but you would have to test. Better to clean up the mess.

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

I flicked the coin. It disappeared into the drain! I'm now broke! چاوداگرتن

Seriously, the query:

SELECT question, attempt, seq_number, COUNT (seq_number) AS NumOccurrences FROM mdl_question_states GROUP BY question, attempt, seq_number HAVING (COUNT(seq_number) > 1 )

returns about 6300 rows on my Moodle database, so going through each one and verifying it against .../mod/quiz/review.php?attempt={quiz_attempts.id} as you suggested earlier in this thread isn't really an option. It will take forever. The ones that I have checked are all valid attempts that I want to keep.

So I need more help sorting this mess out! As I've described elsewhere in this thread, I've tried deleting all those 6300+ rows that the above query returns but I still crash with the "Cannot insert duplicate key row in object 'dbo.mdl_question_attempt_steps' with unique index 'mdl_quesattestep_queseq_uix'" error when I try to finish the upgrade to 2.1 or 2.2, so something else is still wrong.

I did try disabling the mdl_quesattestep_queseq_uix index and the upgrade from 2.0.4 to 2.1 or 2.2 completed "successfully" - although I haven't tested the results thoroughly yet. But I really wouldn't like to do that on the production database. Especially since I can't rebuild the index after the upgrade because that also fails with duplicate messages (see 213578#p1002572). And disabling that index is probably a dumb thing to do anyway...

So, what else do I need to do to clean up that mdl_question_states table? I really don't know what is still wrong with it and what the remaining problem data is.

Should I create a Tracker issue for this?

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

Sorry, I assumed that you had already deleted the obvious 6300 duplicate rows, and that there were only a few left causing some other problem.

Anyway we need to work out why it is failing after you have cleaned up those rows.

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

Yes, I did delete the 6300 records before attempting the 2.1 or 2.2 (I tried both) upgrade.

But I still get the "Cannot insert duplicate key row in object 'dbo.mdl_question_attempt_steps' with unique index 'mdl_quesattestep_queseq_uix'" error when attempting the upgrade AFTER having deleted those 6300+ records.

What actually happened was:

I ran the query:

SELECT question, attempt, seq_number, COUNT (seq_number) AS NumOccurrences FROM mdl_question_states GROUP BY question, attempt, seq_number HAVING (COUNT(seq_number) > 1 )

It returned the 6300 rows (ok it was about 6280). I deleted those.

I ran the above query again. It returned about another 1000 rows.

I deleted those.

I ran the query again. It returned about 600 rows.

I deleted those and ran the query again. It returned about 250 rows.

I kept on re-running the query over and over again and deleting the rows that it returned each time until the query finally returned 0 rows. I had to run the delete statement about 7 or 8 times before 0 rows were returned.

So it's actually more than 6300 records. I've got the exact numbers written down at the office, which I can post here if you want.

But after doing all this the upgrade to 2.1 or 2.2 still fails with the "duplicate key row" error. So there's still some "bad" data that the above query isn't catching.

In reply to Luis de Vasconcelos

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Tim Hunt -
وێنەی Core developers وێنەی Documentation writers وێنەی Particularly helpful Moodlers وێنەی Peer reviewers وێنەی Plugin developers

Right. I understand that. What we have to do now is to use the kind of techniques we used before, to work out exactly what other sort of bad data is causing the duplicate key row error after we have removed all of the types of duplicate we identified before.

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

I ran this SELECT on a fresh "uncleaned" copy of the production database:

SELECT *
FROM mdl_question_states
WHERE question = 9091
AND attempt = 22062

It returns:

id attempt question seq_number answer timestamp event grade raw_grade penalty
1296505 22062 9091 0 29409,29408,29407: 1265011137 0 0.0000000 0.0000000 0.0000000
1296744 22062 9091 1 29409,29408,29407:29409 1265011152 6 1.0000000 1.0000000 0.1000000
1296748 22062 9091 1 29409,29408,29407:29409 1265011156 6 1.0000000 1.0000000 0.1000000
1296752 22062 9091 1 29409,29408,29407:29409 1265011156 6 1.0000000 1.0000000 0.1000000
1296753 22062 9091 1 29409,29408,29407:29409 1265011157 6 1.0000000 1.0000000 0.1000000
1296755 22062 9091 1 29409,29408,29407:29409 1265011155 6 1.0000000 1.0000000 0.1000000
1296838 22062 9091 1 29409,29408,29407:29409 1265011136 6 1.0000000 1.0000000 0.1000000

Rows 2 to 7 are identical except for the id and slightly different timestamp values.

Is it correct that 5 of those rows can be deleted so that there is only ONE row WHERE seq_number = 1?

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

Tim,

Using /mod/quiz/review.php?attempt=22062, I checked one of the 2.0.4 quiz attempts that supposedly has those duplicate records on mdl_question_states. As the screenshot shows there are SIX "Close&grade" rows for this question:

There should only be one.

And these 6 rows correlate to the rows returned by: SELECT * FROM mdl_question_states WHERE question = 9091 AND attempt = 22062

Is that correct?

In reply to Tim Hunt

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

The example in https://moodle.org/mod/forum/discuss.php?d=213578#p1003462 shows one case where there are more than two duplicated rows, so Tim's query:

SELECT
 MAX(id),
 question,
 attempt,
 seq_number,
 COUNT (seq_number) AS NumOccurrences
FROM
 mdl_question_states
GROUP BY
 question,
 attempt,
 seq_number
HAVING (COUNT(seq_number) > 1 )
ORDER BY attempt, question;

does not return ALL the duplicate rows.

And the DELETE code that Tim posted also does not remove all the duplicates. I still crash with the "Cannot insert duplicate key row in object 'dbo.mdl_question_attempt_steps' with unique index 'mdl_quesattestep_queseq_uix'" error AFTER running that DELETE statement.

Considering that I have 8 million rows in the mdl_question_states table, how can I get a list of all the duplicate rows that are causing this error? I'm completely unable to upgrade Moodle because of this problem...

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

Okay, this is getting extremely frustrating now.  Can we go back to basics here?

If I'm honest I'm really not sure what I am supposed to be looking at.  The two queries do not seem to be given me what I think you think they are Tim.

Can somebody give me a query, or set of queries, that takes me from the initial error, e.g.

INSERT INTO mdl_question_attempt_steps (fraction,timecreated,userid,state,sequencenumber,questionattemptid) VALUES(NULL,'1318857822','744',N'todo','0','1414')
[array (
0 => NULL,
1 => '1318857822',
2 => '744',
3 => 'todo',
4 => 0,
5 => 1414,
)]

To a quiz_attempts.id that I can paste into an address .../mod/quiz/review.php?attempt={quiz_attempts.id}  in Moodle 1.9 to look at the entry.

 Thank you.

In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Steve Lawrence -

I have news.

Upgrading from 1.9.19 -> 2.0.x failed.

Upgrading from 1.9.19 -> 2.2.x failed.

Upgrading from 19.19 -> 2.1.8 -> 2.2.5 -> 2.3.2 ... WORKED!

Might need to review something.  The documentation at the very least.

تێکرایى نمرەپێدراوەکان:Useful (1)
In reply to Steve Lawrence

Re: Upgrade: mod-quiz duplicate key row error

لە لایەن Luis de Vasconcelos -
وێنەی Particularly helpful Moodlers

FYI... I'm trying to upgrade from 2.0.4+ to 2.5. I can't get past 2.2.

All the following upgrade paths failed for me:

  • 2.0.4 -> 2.0.10 -> 2.1.10 -> 2.2.11 : failed
  • 2.0.4 -> 2.1.10 -> 2.2.11 : failed
  • 2.0.4 -> 2.2.11 : failed
  • 2.0.4 -> 2.2.11 -> 2.3 : failed
  • 2.0.4 -> 2.2.11 -> 2.4 : failed
  • 2.0.4 -> 2.2.11 -> 2.5 : failed

I'm not as fortunate as Steve: 2.0.4 -> 2.1.8 -> 2.2.5 -> 2.3.2 did NOT work for me.