With flv, you could just remove video and the remaining audio would play (as flv). I don't know how that would work with mov files though. There is the ability to specify encoding options for ffmpeg, there on the PoodLL filter settings page, but it does assume the end format will be mp3. In my own experience the conversion to mp3 is not a significant drag on the server, but it depends on how many conversions you have going on simultaneously.
Related to this, I hope to implement background encoding sometime soon, so that the user doesn't have to wait for conversions to finish.