It's definitely NOT yes's. After all, we're not saying that the plural of yes belongs to yes! Actually, in my opinion, that form looks wrong even when pluralizing numbers. So, for instance, I usually write "the 1970s" and not "the 1970's". But I know the accepted form with numbers includes the apostrophe. It just seems unnecessary, unless where a 5 could be mistaken for an s in a particular typeface.

Anyway, "yeses" is apparently correct. However, someone mentioned "ayes" which is an interesting alternative.