Solved, although solution is not satisfactory.
Basically it is a PHP memory issue and I needed to increase the max memory setting to 1024M and it worked. So for 1 user but with about 440 grade items (the import file is only 46 KB) I don't understand why Moodle PHP script needs 1G . This seems really bad or I am missing something.
2nd problem but not related to memory issue is that there is no attempt to match imported items with existing items, default is ignore. So I need to go to each of the 440 grade items and set the mapping even if my import is 1:1 w.r.t existing grade items. This is highly inconvenient.