it seems that, what is happening, is indeed this:
"Custom auth plugins may be failing in the last step,not redirecting back to local/mobile, this will happen only if the plugin does not support correctly the $SESSION->wantsurl."
You should check the app code to see if after doing the final redirection back to Moodle it checks the $SESSION->wantsurl variable. This variable indicated the plugin that it should redirect to an enpoint in local/mobile (or admin/tool/mobile in latest Moodle versions) to correctly perform the SSO
Maybe you could reach the original plugin developers