We've talked about fiber formality before, and while I don't have a problem with anyone wearing cashmere with twill I do think that certain fabrics won't be conservative enough for many offices.
In my experience, even people who say they don't know or care much about fashion have very definite opinions about levels of fiberformality and how they can contribute to fiber clashes.