Savan Kotecha claims the Tesco worker, 50, felt out of place at the contestants' London house.
'It was very difficult for her in the beginning,' he explains.
'As a middle-aged woman, she didn't feel like she fitted in with all these young kids who were bouncing off the walls. She kept saying: "I don't feel like I belong here."'
But Mary - who is being mentored by Louis Walsh, 58 - is now determined to succeed on The X Factor after having a pep talk with Savan, 31.
'I sat her down and we talked,' he tells the News Of The World.
'She's a very valuable commodity in the music industry. People her age are the only ones back buying albums.'
Where to next?- Test your X Factor knowledge with our quiz
- X Factor final 16
- Frock factor: Who wins in the style stakes?