Bucky. When he's on the ship fighting Steve, and he's telling him to SHUT UP, I think he felt something starting to click in his mind, that he did know this guy from before. But he was scared, he didn't want to hurt anyone he was close to, so he forced himself to shut the feeling out. He COULDN'T hurt anyone he was close to. Which is why he stopped punching Steve after he heard him say "I'm with you till the end of the line" and everything fell apart.