A lot would depend on your version of Moodle and of the block. Some early versions displayed ALL ungraded work. The latest Moodle version (for Moodle 2 only) creates a cache and allows you to reduce the types of modules being used, which should allow you to fine tune to your needs.
https://github.com/dakotaduff/moodle-block_grade_me
All of that said… it can be taxing on a large database to have to search for ALL ungraded work that a user would have permission to grade, so if this is causing problems please use only at the course level.