The 24-year-old, who recently split from co-star Louise Thompson, said the 20-year-old pop star knew they were an item when she spent the night at his house.
Andy told Now Magazine: "I'm pretty annoyed with Niall. He sends a car to take my girlfriend back to his place? What am I supposed to do?
"I wonder how many girls he sends that car to. I'd love to ask Niall exactly what happened and why he did it.
"I feel that he's snaked his way into my girlfriend's life."
Andy continued: "I believe he knew we were together. When we won the BAFTA we were photographed together and afterwards he messaged her and said, 'I see you have a boyfriend'.
"I don't know what she said back. And I guess I never will."
But Andy believes 23-year-old Louise may have told the 'Story of my Life' singer that she was single, and she has since admitted she slept with him but claims it wasn't when she was dating Andy.
He said: "I wouldn't be surprised if Louise said we weren't together because everyone I've met who knows Niall says he's a really nice guy."
The couple's eight month romance ended when Andy found out she'd stayed the night at another man's house and the star has never been named on the show.
Andy said: "I pushed her on it and said, 'Why are you lying to me? I heard you've been sleeping with Niall'. She just denied it.
"She admitted she'd gone to his for a house party, but swore nothing had happened.
"Then after about an hour she broke down and confessed she'd slept with him."
Niall previously brushed off the "meeting" and said it caused him to look at who he can trust.
He said: "One meeting turned into a big story, which is ridiculous, so that's the end of that.
"It does make me wary. You struggle with who to trust and who not to trust."