it used to use md5 on the database table and there is a salt in config.php.
What you are attempting is not a good approach, you should look at writing your own authentication plugin for moodle. This may give you a starting point:
https://moodle.org/plugins/view.php?plugin=auth_onelogin_saml