Prince Harrysays that his older brother,Prince William, broke a vow between them about never turning their offices against one another.

In episode 4 of his and wifeMeghan Markle’s newNetflixdocuseriesHarry & Meghan, the Duke of Sussex touches on the 2020 report that “bullying” led to the couple’s decision tostep back from royal lifeand the subsequentstatement that was releasedwithout his permission, he claims.

About the press,Prince Harrysaid, “There’s leaking, but there’s also planting, of sorts. So if the comms team want to be able to remove a negative story about their principal, they will trade and give you something about someone else’s principal. So the offices end up working against each other.”

For more on the Royals, listen below to our daily podcast PEOPLE Every Day.

According to Harry, there’s a “weird understanding, or acceptance, that happens,” in which “you can always say, ‘I didn’t know about this,’ or, ‘Don’t be ridiculous. This would never happen. Are you suggesting that I condone this?’ It’s like, ‘No, but what Iamasking is, ‘Have you done anything to stop it?’ And the answer is no.”

“William and I both saw what happened in our dad [King Charles III]’s office, and we made an agreement that we would never let that happen to our office,” Harry said.

Prince Harry.Netflix

Harry & Meghan Netflix Documentary

Prince Harrywent on to say he “would far rather get destroyed in the press than play along with this game, or this business of trading.”

“To see my brother’s office copy — the very same thing that we promised the two of us would never, ever do — that was heartbreaking,” he added.

Prince Harryalso said there was “this wedge created between myself and my brother so he is now on the institution side. Part of that, I get. That is his inheritance. To some extent that is already engrained in him, that part of his responsibility is the survivability and continuation of this institution.”

Can’t get enough ofPEOPLE’s Royals coverage?Sign up for our free Royals newsletterto get the latest updates onKate Middleton,Meghan Markleand more!

Prince William (L); Prince Harry.Daniel Leavl-Olivas-Pool Getty Images; Chris Jackson/Getty Images

Prince William, Duke of Cambridge and Prince Harry

“Despite clear denials, a false story ran in a U.K. newspaper today speculating about the relationship between The Duke of Sussex and The Duke of Cambridge,“read the denial statementthat was presented as a joint one from Harry, 38, and William, 40. “For brothers who care so deeply about the issues surrounding mental health, the use of inflammatory language in this way is offensive and potentially harmful.”

In episode 5 ofHarry & Meghan, Harry recalled, “A story came out saying that part of the reason why Meg and I were leaving wasbecause William had bulliedus out. And once I got in the car after the meeting, I was told about a joint statement that had been put out, in my name and my brother’s name, squashing the story about him bullying us out of the family.”

“We couldn’t believe it,” he went on. “No one had asked me permission to putmy name to a statement like that. And I rang M and I told her and she burst into floods of tears because within four hours, they were happy to lie to protect my brother. And yet, for three years, they were never willing to tell the truth to protect us.”

“Suddenly, what clicked in my head was, ‘It’s never gonna stop,’ " said Meghan, 41. “Every rumor, every negative thing, every lie, everything that I knew wasn’t true and that the palace knew wasn’t true and internally they knew wasn’t true, that was just being allowed to fester.”

“So there was no other option at this point,” Harry added. “I said, ‘We need to get out of here.’ "

Prince Harry and Prince William.Samir Hussein/WireImage

Prince William, Duke of Cambridge and Prince Harry walk through a trench during the commemorations for the 100th anniversary

During the second volume ofHarry & Meghan, the royal also said that his brother “scream(ed) and shout(ed) at” him over his and Meghan’s choice at theSandringham Summitto be “half in, half out” of their royal roles, meaning that the couple would “have [their] own jobs but also work in support of"Queen Elizabeth.

“It became very clear very quickly that that goal was not up for discussion or debate,” said Harry. “It was terrifying tohave my brother scream and shout at me, and my father say things that simply weren’t true and my grandmother quietly sit there and sort of take it all in.”

All episodes ofHarry & Meghanare now streaming on Netflix.

source: people.com