I generally put this down to very religious people (who have been raised with the concept that God is personally invested in them and is a central force in their life) experiencing the thought of a person without a religious belief system as being close to someone soul-less: without morals and without any fear of punishment (hell), so obviously less
trustworthy than religious people who have a spiritual Big Brother and religious community watching their every move.