Your upgrade to latest and greatest Moodle should go smoothly without removing Library module. Wouldn't hurt anything to manually remove the table from the database though. Are you installing 1.2.1, or going to try latest 1.3? The latest seems pretty stable and has a lot of great new features. I really like it.