I think this could be caused by misconfigured Grade mapping table. There is a longstanding usability issue with this setting reported as MDL-49779.
Basically, go to "Edit assessment form" and make sure that your "Grade mapping table" has fields filled like this:
