King Charles III, Prince Harry, Duke of Sussex and Meghan Duchess of Sussex

Queen Elizabethand her heirPrince Charlesdidn’t always see eye-to-eye, but in the final years of the record-breaking monarch’s life, they bonded over a difficult decision.

WhenPrince HarryandMeghan Markleannounced their desire to step back from royal duties, they proposed a scenario where they would combine public duties with their work outside the royal family. As negotiations continued over the couple’s future, Harry’s father, Charles, 75, and grandmother, Queen Elizabeth,stood firm. The Duke and Duchess of Sussex had to fully commit when it came to being working royals.

“Harry and Meghan’s departure brought Charles and his mother closer together,” Robert Hardman, author of the new bookThe Making of a King: King Charles III and the Modern Monarchy,tells PEOPLE.

Prince Charles, Prince of Wales, Camilla, Duchess of Cornwall, Queen Elizabeth II, Meghan, Duchess of Sussex, Prince Harry, Duke of Sussex, Prince William, Duke of Cambridge and Catherine, Duchess of Cambridge watch a flypast to mark the centenary of the Royal Air Force from the balcony of Buckingham Palace on July 10, 2018 in London, England.

King Charles and Prince Harry in April 2019.Samir Hussein/Samir Hussein/WireImage

Prince Charles, Prince of Wales and Prince Harry, Duke of Sussex attend the “Our Planet” global premiere

Samir Hussein/Samir Hussein/WireImage

The author, who has spoken to many past and present royal aides and friends of the King for his book, says there was “sadness” for Charles at the time, but that he remained “pragmatic.”

Hardman says, “There was a sort of weary resignation, but also a sense that ‘look, I’ve got so much to worry about now that I don’t have the luxury of dwelling on this.' If that’s what they want to do. I mean there’s, there’s only so much I as a father can do. And the door is always open.”

Meghan Markle and Prince Harry on Sept. 16, 2023.Chris Jackson/Getty Images

Prince Harry, Duke of Sussex, and Meghan, Duchess of Sussex attend the closing ceremony of the Invictus Games

Chris Jackson/Getty Images

Can’t get enough of PEOPLE’s Royals coverage?Sign up for our free Royals newsletterto get the latest updates on Kate Middleton, Meghan Markle and more!

Hardman adds that the Duke and Duchess of Sussex’s departure is a loss for the monarchy, saying there is the “occasional wistful thought" at the palace “of if they were still part of the team and of all the things they could have done and could now be doing.”

He adds, “There is no question that Harry and Meghan are a great loss to the institution, and that is still appreciated and understood. There’s absolutely no sense of good riddance or anything like that. It’s fundamentally a source of deep regret.”

source: people.com