Hi,
open the edit_from.php file in course folder and increase the maxlength in below mentioned code.
$mform->addElement('text', 'shortname', get_string('shortnamecourse'), 'maxlength="100" size="20"');
If you don't want to change in core files then you can use the moodle customscripts feature to modify the course form.
Thanks.