When a quiz is deleted, the entry in the mdl_quiz table should also be deleted (specifically, by this line of code: https://github.com/moodle/moodle/blob/master/mod/quiz/lib.php#L205 ).
If that is not happening, then there is a bug on your system that needs investigating.