"I'm talking about the fact that I never saw ghosts in Mystic Falls, because I was in Chicago during that time. A rift, which is a break in space and time and universes, kidnapped me and Elena and a bunch of people. So the Damon that was there then isn't me."

Damon doesn't know how the science of it all works really, and the whole world ended up being destroyed in the end anyway. He just knows he has not been in Mystic Falls in a long time, and he didn't realize they'd added ghosts to the supernatural beings wandering around in it.