Found the answer partially at https://tracker.moodle.org/browse/MDL-51515 but had to add an additional parameter to get it to work.
The answer is:
If you are not the user looking at your own report then these parameters should be added to the link ~/assign/view.php?id={cmidfor assignment}&action=grade&rownum=0&userid={theuserid}