This is the same reason why you shouldn't use fabric softeners with CD - s. (backtocalley.com)
Occasionally you will find that too much detergent or the accidental use of fabric softener sheets can cause excessive odor or repelling. (the-cloth-diaper-connection.com)
The sickly sweet floral scent of fabric softener used by the previous owners was so strong that I had to rewash the clothing multiple times before I felt safe using it to dress my son! (thehealthyhomeeconomist.com)