The news was broken when a 'source' spilled the beans but Rita has since shared the news herself on Twitter.
With rumours rife that Rita wants to concentrate more on her singing and acting career the pop princess simply told fans, 'I had a ball on @TheXFactor last year and will miss the team.'
I had a ball on @TheXFactor last year and will miss the team.— Rita Ora (@RitaOra) May 10, 2016
This news comes just weeks after a whole string of X Factor announcements. Cheryl Cole, Nick Grimshaw, Olly Murs and Caroline Flack have also confirmed they won't be returning for a new series either - so it looks like Simon's the last man standing from last year's line up.
The most recent departure, aside from Rita, was Cheryl. Despite her decision to quit, a source told The Sun that Cheryl loved working on The X Factor in 2015; 'Last year was one of her favourite series, she had a lot of fun mentoring her groups and enjoyed working closely with the production team on the show.'
Then it was the turn of cheeky Essex lad Olly. The former X Factor contestant told The Sun he made the decision a few months back, telling them, 'My heart belongs to music.'
The 31 year old continued, 'This was an incredibly hard decision to make and one I didn't take lightly as I've really enjoyed co-hosting The X Factor.'
If rumours are to be believed then it could have been a close call for Olly in the axing stakes, with the Daily Mail reporting that 'sources' said he was given the chance by Simon Cowell to 'walk before he was pushed'.
Others have suggested that the singer and presenter wouldn't work on the show without his close friend and co-star Caroline, who was already rumoured to be leaving.
It looks like Caroline Flack and Olly Murs couldn't be apart!
Both stars then confirmed the news that they planed to leave. Caroline, 36, said, 'I have had a brilliant time working on The X Factor over the last few years, and hosting the main show was just fantastic, I made some amazing friends.'
The presenter continued, 'I'm looking forward to carrying on working with the team over at ITV.'
To round off the recent X Factor news, Nick Grimshaw announced he had also quit a few months back, in an apparent shake-up for the show after ratings dropped last season. While some have said that Nick was also 'pushed' from his role as a judge, a source for the DJ confirmed that he left of his own accord: 'After careful consideration, Nick has decided to rule himself out as a judge on this year's X Factor.
'He loved working on last year's series alongside Rita, Cheryl and Simon but it was very much a once in a lifetime experience.
'He remains a massive fan of the show and is looking forward to watching it this year.'
It seems the radio star hasn't left on bad terms, with a spokesperson for the show confirming: 'Nick was a great mentor on last year's series. He did a fantastic job with the boys' category and we'd like to thank him for all his hard work. We are sad to see him go but wish him all the best.'
The radio presenter had to deal with quite a bit of criticism during his one season on the show with many calling him boring, but we loved him!
Caroline and Nick were all smiles together during their time on the X Factor
So now the big question is, who will replace Rita, Cheryl, Caroline, Olly and Nick? Bets are on new dad and One Direction star Louis Tomlinson as a judge - as a well-known favourite of Simon.