All you need to do is create groups, no groupings. On the Edit settings page for the activity, set Group mode to Visible Groups or Separate Groups. Set Grouping to none. Click Add group/grouping access restriction. For access restrictions, select Student must match the following. Select the Group that you want to see the LL. Click the eye so that no other group will see the activity.
With this setup, only the group you select for the access restriction will see the LL in the course. However, you as the teacher are STILL going to see ALL the students enrolled in the course, listed in the Recordings section. I guess that the plugin doesn't have groups fully coded like most plugins do.