It solved my problem.
If I choose to use Aggregate only non-empty grades to false, it will force to give 0 to all other students who are excepted from taking a quiz. I have two levels of groups.
Then under Common module settings, I set Grouping to a group which I want to give the quiz and tick "Available for group members only"
And as you suggested setting the Aggregate only non-empty grades to false in Grades > Edit > Categories and items.
Wow, it getting more complicated and I am a bit worried for my colleagues who are not so IT oriented to follow this.