That surprised me, but I went to my test site to play with settings. I was able to remove the button by:
- "Require students to click submit button" = "Yes"
- "Attempts reopened" = "Never"
When I do that, I no longer see the "Edit Submission" button when I log in as a student.