A lot of the "pain" depends upon how complex your moodle is, and your own knowledge and skills.
Just the other day, I upgraded my moodle (yes, my VPS's moodle) in about 20 minutes. I always upgrade my "experimental" moodle first, then my "production" moodle. So of the 20 minutes, 18 minutes was preparing, and in the remaining 2 minutes, I upgraded my production moodle. I have about 8 extra plugins in my Moodle.