The phrase "difference matters" means that the distinctions or variations between things are important and should be considered or acknowledged. Full definition
Strange though that the older people get, the less age difference matters because both become adults. (myintimcity.com)
Thus, whether ideological differences matter more than modestly in individual districts does not appear to have been clearly addressed. (herculesandtheumpire.com)
However, other differences matter a lot more, like those who love the single life versus those who prefer to stay coupled up. (cupidspulse.com)