Gossip about the One Direction hunk's womanising ways features appears daily and the 19-year-old star admits the constant speculation affected his past relationships with the likes of Taylor Swift and British TV presenter Caroline Flack.
He told the new UK issue of GQ magazine: "Some of them are funny. Some of them are ridiculous. Some of them are annoying. I don't want to be one of those people that complains about the rumors.
"I never like it when a celebrity goes on Twitter and says, 'This isn't true!' It is what it is; I tend not to do that.
"The only time it gets really annoying is that if you get into a relationship and you get into a place where you really like someone and then things are being written in the papers that affect them and how they see you. Then it can get annoying."
Harry also laughed off rumours he may be bisexual.
When quizzed about his sexuality by the publication, he scoffed: "Bisexual? Me? I don't think so. I'm pretty sure I'm not."
Meanwhile, Harry's bandmates - Niall Horan, Louis Tomlinson, Liam Payne and Zayn Malik - are adamant they aren't interested in groupies and don't have time to seduce their fans.
Irish singer Niall insisted: "We go back on the road straight after a show and we're gone. I wish we did have time for girls and parties but we don't."
Louis - who is dating long-term girlfriend Eleanor Caldwell - added: "The type of girls who would sleep with you in a heartbeat aren't the type of girls I'd want to take home."
However, Niall, 19, would make an exception for singer Ellie Goulding, who he has been shamelessly flirting with on Twitter for the past month.
He blushed: "I'll say one thing: Ellie Goulding - wow. She's absolutely amazing looking."