Trick (that might work) ... go directly to via URL the page that has the uninstall button for mods/blocks/etc.
Example: http://site/moodle29/admin/plugins.php
In newer versions of Moodle it warns that it will remove files when un-installing. In older versions of Moodle it warns you should manually remove the directories related or the next time one run notifiications, Moodle will see the directory and assume it's supposed to install - which will re-create the tables in the DB. I don't re-call now which versions did what ... but you'll see it and can then act appropriately.
There is also a table that keeps track of the mods/blocks/etc. that are installed. That's cleaned up via Moodle UI (don't forget that as you march versioning information becomes more important).
On a Moodle march via git ... git will acquire updates to core code ... not addons/plugins, etc.. That's why it's important to stop at steps and check the addons/plugins to see what Moodle would want to update. This is where your prior research comes into play. Does a mod/block/theme/etc. have a compatible version for your next step?
As far as leaving them ... no apparent ill affect ... well, that could be true, for now. But is one leaving a ticking time bomb? Do we really know that when 4.0 becomes available that bomb won't be triggered? Always the choice of op ... roll the dice. But one shouldn't be surprised if it happens. And, then the quest begins for how to fix. :\
Always best to use the Moodle UI rather than editing the DB directly, but I've had to do that with plugins/addons that weren't programmed correctly to drop the tables. After having the need to do that on sites that have been ignored for years (by inexperienced server admins) it's the reason I always caution folks about going nuts with addons/plugins.
'spirit of sharing', Ken