Sorry - I thought you were wanting to extend a the plugin's global settings (defined in settings.php), not the per-instance settings when you add an activity to a course.
There is no way to directly extend the instance settings without making a change to the original plugin code - you could use the code that Vitaly has outlined below to add a new navigation item that links to a page that defines your custom settings (or you could do something really nasty like use some javascript to insert the settings into the page after it has loaded, but please, please don't do that). If you do go down the modifying the plugin code route, you'd also need to extend the X_add_instance and X_update_instance function in the plugin's lib.php in order to save your custom settings somewhere (and then make sure they were loaded where you wanted to use them).
