Hi Paul,
You remove it just as you added it; Login as admin, go to the administration block and click "Appearance" -> "Sticky Blocks", choose the page you added it to (MyMoodle or Course Page) and click the x in the title bar of the block you want to remove.
Hope that helps,
Rich