The phrase "to spoil the fun" means to ruin or take away the enjoyment, excitement, or happiness from a situation or an activity.