The One Direction heartthrob - who enjoyed a two month romance with the country singer late last year - reportedly told friends that he wishes he never went out with the 23-year-old star after she took a swipe at him in her new music video 22.
A source close to the 19-year-old British boy band member told RadarOnline.com: "Harry might have taken Taylor's swipes with an air of grace, but he now wishes he never ever dated her.
"I mean, the fact she's aiming a jab at him in the 22 video has got Harry wondering what's coming next! He just doesn't know what to expect.
"It wasn't the most amicable of splits, but Harry said all along that he wanted to take things slow and easy with Taylor.
"There was certainly no commitment made, despite Taylor's protestations and the fact that they were so busy with their careers and lived in different countries meant that it could never have really got that serious.
"What for Harry was 'a bit of fun' at the time has slowly turned into a personal nightmare.
"But he won't stoop to Taylor's level, he's just going to continue to be the more grown-up person and get on with his life. Perhaps something Taylor should consider too."
Styles was previously said to have demanded that all footage of his ex-girlfriend be removed from the upcoming One Direction film.
Swift admits writing song about Styles
Meanwhile Swift has confirmed that her song I Knew You Were Trouble was written about ex-boyfriend Styles.
The 23-year-old pop star gave a scintillating rendition of the song at the BRITs in February as the One Direction heartthrob looked on.
In a new interview, Swift says that she found it easy to connect with her emotions as the person she wrote it about was so close to her.
"Well, it's not hard to access that emotion when the person the song is directed at is standing by the side of the stage watching.
"You're balancing the analytical side of your brain which is telling you where to go and how to go there, with the other side of your brain, which is saying, 'Feel everything you're singing, and show it on your face. Feel everything exactly as you felt it when you wrote the song," said Swift.
Swift, who has written tracks about several of her famous former lovers, including John Mayer and Jake Gyllenhaal, insists that she no longer has time for guys who are scared of commitment.