The One Direction heartthrob - who enjoyed a two month romance with the country singer late last year - reportedly told friends that he wishes he never went out with the 23-year-old star after she took a swipe at him in her new music video '22'.
A source close to the 19-year-old British boy band member told RadarOnline.com: "Harry might have taken Taylor's swipes with an air of grace, but he now wishes he never ever dated her.
"I mean, the fact she's aiming a jab at him in the '22' video has got Harry wondering what's coming next! He just doesn't know what to expect.
"It wasn't the most amicable of splits, but Harry said all along that he wanted to take things slow and easy with Taylor.
"There was certainly no commitment made, despite Taylor's protestations and the fact that they were so busy with their careers and lived in different countries meant that it could never have really got that serious.
"What for Harry was 'a bit of fun' at the time has slowly turned into a personal nightmare. But he won't stoop to Taylor's level, he's just going to continue to be the more grown-up person and get on with his life. Perhaps something Taylor should consider too."
Harry was previously said to have demanded that all footage of his ex-girlfriend be removed from the upcoming One Direction film.
A source recently revealed: "During their relationship producers shot Taylor and Harry in Los Angeles and New York City but Harry has ordered that none of it makes the cut. He wants nothing more to do with Taylor, she was a fling and they're done."
The film, which is called '1D3D' and is due to be released in August, will show the group's meteoric rise to global stardom, their backstage antics and life on the road.