I like your suggestion, Shirley. However, "start a new attempt" might only be valid if that particular quiz can allow more attempts. If a quiz allows only one attempt, then "start a new attempt" should probably not be an option. So it might be that "Start a new attempt" is a bit more complex logic, especially if the quiz is using interactive questions (the student does 9 of 10, knows where they are, and wants to abort.)
Likewise, the "cancel" could be challenging. If a quiz contains allows one attempt with 10 randomly picked questions, the student looks at all and doesn't like 8 of them, what should "cancel" do? Should it allow students to redo the quiz with a new set of questions?
Getting back to Tim's question, yes, "continue" should be there. I think if a quiz allows only one attempt, then "continue your attempt" is better than "continue your last attempt." Perhaps this is what Tim has discovered, that "last" is also "first" when there is a quiz with only one attempt.