Found out this was turned off under Site Administration, Appearance, Navigation:
Always link course section
Default: Yes
Always try to provide a link for course sections. Course sections are usually only shown as links if the course format displays a single section per page. If this setting is enabled a link will always be provided.