Well there's no better learning experience then reply to your own questions. So here it is.. just in case anyone has the same issue in the future.
When you manually edit the grade of a face-to-face activity it breaks the link between the attendance and the grade item. So any changes you make to the attendance will never be kept.
If you want to re-establish that link, the only solution I've found, is accessing your database and finding the field overrideen and putting zero's everywhere. This will tell face-to-face and grade to stay in synch.
Regards,
Pierre