If you did anything beyond what appears below, you used the instructions that were identified as NOT recommended for using code that is NOT provided in the Moodle CVS.
SInce dragmath is working, the problem is with the integration.... which means you don't have the correct files in the correct places..... either because you did not copy a file to the right directory or because after you put the files in the right places you made some changes.....
Do you have shell access to your server (command line?) if you don't, how did you put the files in the correct locations and if you do, what specific steps did you take (the material you quote, as I mention, identifies two different methods and multiple different sources.)
In any event, you may want to go through your /lib/editor directories and provide details of which files were placed where.... I am sure this is just a matter of files being in the wrong place, and there are really less than a handful of files in issue....... we just need to make sure that the files in the source directory treee are placed in the corresponding directory tree in your moodle.....
Installing DragMath From the Moodle CVS - Recommended!
This could not be easier. If you are running verion 1.9.+ click here or if you are running Moodle version 1.8.4 or greater but not 1.9.x click here and you will start the download of the zip file (at present there are no differences between the two versions. At this point you can unzip on your workstation and then you can upload the included files to their appropriate locations. This version retains the smileys and includes all integration files and all DragMath files (demos, docs, and all export formats) Warning: At present the zip file includes a version of adminlib.php that should NOT be used with Moodle 1.8.2 If you are using 1.8 DO NOT over write your adminlib.php. Install the rest of the files (other than adminlib.php) and DragMath will work without a problem.
Apparently the Moodle automagically creates the zip with a parent directory of "dragmath". Should you wish you can unzip the files and then rezip with the top level directory being /lib, and this will allow you to simply upload the zip you just created to your moodle root and unzip there, and the files will be placed in their appropriate locations. However, please make sure all files are backed up, and in as much as there are only a few files you may want to manually install even if a zip with /lib as top directory were provided here just to ensure each file is backed up before installation. Upload the zip file to your Moodle root and unzip.