I tried to delete an incorrect feedback file uploaded for a student in her first attempt and saved changes. And, when I returned to check if the feedback file was deleted and guess what?! It was still there!!!!
If you've found a bug in Moodle, please take a few minutes to double-check the problem (can you reproduce it consistently?). You should also make sure you are running the latest version of the Moodle 2.9 code (2.9.4 is out).
If you can reproduce the problem, then head over to tracker.moodle.org (you may need to sign up for an account there), create a new issue and add step-by-step instructions for reproducing the problem.
Once that is done, someone may well be able to develop a fix for the problem you have identified.