"To haunt the memory" means that something sticks in your mind and continues to emotionally or mentally disturb you, usually in a negative way. It keeps coming back and affecting your thoughts or emotions, making it difficult to forget or get rid of.
Full definition