He doesn't have to ask how terrible. If Damon considers it terrible enough to use the word, Stefan can imagine. And he's not terribly inclined to let his brother go back there either. He studies Damon's face carefully for a moment, before glancing out to the edge of the ship.

Unlike Damon, Stefan doesn't think throwing himself overboard is preferable.

"So you don't go back." Because that's as simple as it is. "You can come with me and Rebekah, or go with Elena. Just because it's our door doesn't mean you can't go through it." That's not how the rifts worked, anyway.