I made a test and removed my old Wiris files from folder filter and lib/editor/tinymath/tiny_mce/3.4.9/plugins
I downloaded the latest Wiris files from http://www.wiris.com/en/plugins/moodle/download and placed the files of Wiris filter to folder filter and plugin files to the place of old tinymath plugin files. I went to site administration and moodle upgraded the filter and plugin. Because there was no Wiris plugin in tinymce plugins folder Wiris gave a note that Wiris plugin might be missing - otherwise the Wiris plugins worked ok with tinymath editor ( I had enabled Wiris plugin from Math filter settings and enabled Wiris filter )
Next I copied the Wiris plugin to tinymce plugins folder and I did not get any errors - and no note about missing Wiris plugin.
So according to my test you can use Wiris plugins both with core tinymce editor and with modified tinymath editor.
I still suspect that there might be some "mixture" of old and new files on your site - there can't be two Wiris filters but there can be two plugins in different editors - maybe some old file is in cache, non upgraded or some new file is missing, corrupted or in a wrong folder (people sometimes unzip files to wrong subfolders like filter/filter... have you tried to purge all caches from administration menu?