The phrase "mustard sweater" refers to a sweater that is a yellowish color, similar to the color of mustard condiment.