While the singer has now taken up a role on his former show 'American Idol', Simon has revealed she was first interested in being a member of this year's new look panel on his talent program.
However, the 52-year-old music mogul didn't want her on the series because he didn't think she could be nasty enough to the pop hopefuls.
He told In Touch Weekly magazine: "Mariah called me about 'X Factor' and I met with her but my main concern was that she might have found it harder on 'The X Factor' than on 'Idol'. 'Idol' is a more gentle show and Mariah is not someone who wants to say no to people."
Judges Nicole Scherzinger and Paula Abdul left 'The X Factor' after its' debut season in America and were replaced by Britney Spears and Demi Lovato for the new series. They are joined by Simon and L.A. Reid.
Simon also revealed he made the decision to quit 'American Idol' after so many successful years because he was "bored".
He explained: "I was bored on the show. I didn't think it was fair to the network that they were paying me to be bored.
"Over time on 'The X Factor' the weak will go and the strong will survive. I believe with all my heart that 'The X Factor' is the best show."