Hi Rory
You should be able to do a Aggregate query to obtain this. If you want to get the number of votes a specific user has given:
where 'xxxxxx' is the user's id..
select count(ParentId) from Vote where CreatedById = 'xxxxxxx' Parent.Type = 'yyyyyyyy'
where 'xxxxxx' is the user's id..