I haven't, but that's a good plan
The issue, for me, is that I develop on more than one machine so I need to check/remember if plugins are stand-alone git repositories or not (I assume your plan won't make the .git/ directory survive a push/pull to another localtion?). It's best (for me) to have a fixed regime - even if it is a bit clunky.