He told the Daily Mirror newspaper: "If you only ever heard Lady Gaga, she's the most boring singer in the world. You've got to see her – you need TV.
"Same with Beyonce. Or Katy Perry. We're all about image. You have to have that X Factor."
Simon, 51, also poked fun at 'American Idol' host Ryan Seacrest, describing him as the "complete opposite" to "intelligent" US 'X Factor' presenter Steve Jones.
He added: "He's tall, dark, British – he's basically completely the opposite of Ryan Seacrest. You've got to add in intelligent as well."
Simon quit his role as a judge on 'American Idol' to launch the new version of his pop star search show.
Lady Gaga - 'The Edge of Glory' video: