At the moment, we fully initialise each question when the question_attempt is loaded (https://github.com/moodle/moodle/blob/master/question/engine/questionattempt.php#L1557).
We could change that to a lazy-load. So that the question is only initialised using $question->apply_attempt_state the first time that a method that requires the question to be fully initialised is called.
It would just be a matter of analysing all the methods of question_attempt to work out which need an initialised question. And then hoping that our automated tests are good enough to catch any mistakes made.
I am not sure if this would help the standard quiz module. I wonder if rendering the Quiz navigation UI needs enough details about the question that it has to be initialised? But, for other activity modules, this could be a useful change.