Labor Day signaled the end of summer and so the rich would «return home» from vacations / traveling, etc. and
stop wearing white after Labor Day because they'd be back in the cities / towns, back at school, maybe back to work... (source, this TIME article was really helpful!)