A representative for the couple confirmed to PEOPLE magazine that Auden James Ellis Grammer arrived 14th November, weighing 7lbs and measuring 21 inches long.
'Both mother and son are doing great,' the spokesperson said.
Kelsey, 61, and Kayte later explained that their son's name was a combination of their favourite performers and writers, and a little family history.
'The name was chosen to honor the great poet W.H. Auden, and James is after another great poet and singer James Taylor — two of our favorites,' the couple said in a statement, adding that they chose Ellis as it was the name of Kelsey's maternal great-grandfather.
'He was a colourful character, to say the least, and we wanted to give the name a second chance'.
Auden is Kelsey and Kayte's third child together - they are already parents to daughter Faith, born July 2012, and son Kelsey, born July 2014. Kelsey also has three daughters, Spencer, Greer and Mason, and one son, Jude, from previous relationships.
The pair kept their latest pregnancy under wraps for quite some time, only revealing the news when Kayte appeared at LAX airport in September, sporting a noticeable baby bump.
Speaking to Boisdale Life in 2015, Kelsey, who lost his own dad when he was just thirteen, opened up about fatherhood, admitting that he has 'no real blueprint' for parenting.
'I had to improvise this dad thing the first couple of times,' he said. 'A lot of my children were born into relationships that were destined to fail – there was that quality in some of my earlier marriages.'
'But I'm still a young man – I think there is great value in being a more fleshed out person – men are pretty useless until they are 40 at least. I’m very happy where I am now.'