I most likely have found the cause:
We had a problem with our nfs file storage, so the requested file was not created in the moodledata dir when the file was uploaded.
Workaround:
In the grading overview ("View all submissions") click on the arraw next to "Edit" and then on "Allow submission changes".
Click on the arrow again and choose "Edit submission". Next click on "save changes" (no need to do anything else).
When you click on "Grade" in the submission overview again, the pdf is loaded correctly.