It isn't very noticable in most games at all. (3dsblessed.com)
With the settings turned down to medium, there's a more noticable difference, but not as much as you'd expect. (pcgamesn.com)
from what you've said it sounds like an issue with the rear tyres, mismatching different makes of tyre's can throw the handling out as they all have different tread patterns (even more noticable in the rain because they aqua plane differently) (mechanics.stackexchange.com)