In the Package, Widmore said, that "Obviously you are not John Locke. Everything else I know is a combination of myth, ghost stories and jungle noises in the middle of the night".
That explains everything. By "ghost stories" we can mean the appearence of Yemi, Christian, Ben's mother, Isabella, etc.
Jungle noises = the whispers. Ben told Rousseau: when she hears whispers, she runs the other way. He was warning her of the smoke monster.