I am rewatching season 4 and am confused with what happens with Desmond after his flashes to and from the past. When he first goes back, it is like his mind resets to 1996 (thats what year he thinks it is, he doesn't know about the island, doesn't know Sayid, etc). But when he talks to Penny, we as the audience think he is finally able to speak with her after all his time stuck on the island. But if his mind is still thinking its 1996, he actually has no concept of how long it has been since he talked to her last (chronologically anyways). Granted he is so emotional because he knows she saved his life by answering and it also proves that she still cares for him after 8 years (1996-2004).
What I'm wondering is does his memory come back in season 4 at all? Because when he is done talking to Penny, he still talks to Sayid as if he doesn't really know him. However we know that in season 5 he recognizes everyone from 815 and Ben and names his kid Charlie, so his memory did come back, but when? They are unclear about this and don't really address it in season 4 or 5. Anyone have any insight to this?