Donald Trump and Maryanne Barry during the opening of Trump’s Taj Mahal Casino in Atlantic City on April 5, 1990.Photo:Ron Galella/Ron Galella Collection/Getty

Donald Trump and Maryanne Trump during the opening of Donald Trump’s Taj Mahal Casino on April 5, 1990.

Ron Galella/Ron Galella Collection/Getty

Donald Trumpand his older sisterMaryanne Trump Barry, a former federal judge and prosecutor, had no real relationship in the years leading up toher death at age 86on Monday.

Sources tell PEOPLE that at least some of the friction stemmed from a secretly recorded conversation Barry had with her niece,Mary Trump, who later released the audio. Though Barry didn’t often speak publicly about Trump,she criticized her brother’s presidencyin the conversation, calling him “cruel” and a “phony.”

“As is the way it often is with thin-skinned Donald, when he heard about some of the things his older sister said about him to Mary Trump, he got bent out of shape and they cooled the relationship,” a source close to the Trump family tells PEOPLE.

The source adds that, in the years prior to the audio being released in 2020, the two had been closer.

“Before that they had talked quite a bit over the years, as he admired and trusted her on many legal and business issues. He was proud of where she went in her career and boasted about it, often taking some credit for her success," the source said.

The source adds that “the Trumps have always been a close family and a lot of this was kept quiet and not discussed publicly. They didn’t hate each other, so it was sad.”

Donald Trump with sister Maryanne Trump Barry.Andrew Milligan - PA Images/PA Images/Getty

Donald Trump with sister Maryanne Trump Barry.

Andrew Milligan - PA Images/PA Images/Getty

Barry’s leaked private conversation came in contrast to how she had spoken about her brother nearly two decades prior, when shetoldNew Yorkmagazinein 2002 that he consistently showed up for her.

“Six years ago, I had an operation. Donald came to the hospital every single day,” she said at the time. “Once would have been enough — the duty call. That’s how love shows, when you go that extra yard.”

“She thought he was born without empathy for others, and was basically a ruthless and selfish businessman without character,” the source continues. “She basically didn’t trust much of anything he did in business or politically.”

“Maryanne loved her brother but basically gave up on him,” that source says. “It was like sending your kids away to military school because you can’t discipline them at home. She didn’t talk trash about him in public but she’d had enough of his lies and fabulism, and overdosing on social media.”

Donald Trump and Maryanne Trump Barry in Aberdeen, Scotland, on June 10, 2008, where Trump was defending his plans to build a golf resort.ED Jones/AFP/Getty

Donald Trump is pictured with his sister Maryanne Trump Barry at the Aberdeen Exhibition & Conference centre, Scotland, on June 10, 2008.

ED Jones/AFP/Getty

The New York source says Barry thought her brother was too self-centered to have a healthy outlook on life in general.

“She believed that he only thought of himself which was self destructive and basically ruined his life,” the source says. “But she maintained her decorum and didn’t talk about him much in later years.”

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer.

A political source says that the former president would have liked to lean more on his sister, but was hurt and angry over what she said to Mary Trump.

“Once you have stabbed Donald in the back it is hard to get back in his good graces but for a family member who can offer him something, or who has something on him, he is more forgiving,” the political source tells PEOPLE.

Despite the strained relationship, another source based in Florida tells PEOPLE that “the family is upset about her death and will do whatever is needed to close out her affairs.”

PEOPLE reached out to former President Trump for comment on the sources' allegations.

source: people.com