In Moodle core, there is an enrol_paypal module. And, in the plugins database, there are plugins like https://moodle.org/plugins/enrol_stripepayment. I suggested you look at those, to get a feel for how they work, then do something similar with the API for your payment gateway.