The 22-year-old singer doesn't believe he is a sex symbol and thinks women are more interested in looking after him than making passionate love to him.
Speaking at the MTV Video Music Awards (VMAs), he told People magazine: "I find the whole concept of women screaming at me so odd. It's very flattering, but I don't think I will ever consider myself to be a sex symbol.
"If you speak to most women, they want to take me home and mother me rather than take me to bed. I just have that impression."
The ginger-haired star - who is currently single - was spotted cosying up to Ellie Goulding at the VMAs in Brooklyn, New York City, on Sunday night, with the pair pictured holding hands while watching the performances.
However, Ellie was quick to dispel rumours, as she has just started dating One Direction star Niall Horan - who is friends with Ed.
In a post on her Twitter account, Ellie said: "I love that holding hands with my friends means we're an item. In that case I am in many relationships. Lover not a hater. #whatthef**k (sic)"