In 2006 Winfrey called out the author when she learned that his memoir, which she chose for Oprah’s Book Club, was partly fabricated

James Frey and Oprah Winfrey.Credit : Steve Granitz/WireImage; Stewart Cook/Getty for Warner Bros.

NEED TO KNOW

James Frey is speaking out about his infamous 2006 interview with Oprah Winfrey in a new interview with The New York Times
The author became the subject of controversy when it was revealed that parts of his 2003 memoir A Million Little Pieces, which was chosen for Winfrey’s book club, were fabricated
Winfrey confronted Frey on her show, later stating she did so because she was “defending every single viewer who had bought that book”

James Frey is speaking out about his infamous publishing controversy.

In an interview with The New York Times published on June 8, the author looked back on the publication of his 2003 book A Million Little Pieces. Originally marketed as a memoir, the book detailed the author’s experience with drug and alcohol addiction, as well as his subsequent rehabilitation. Oprah Winfrey chose the memoir for her book club in 2005, and it soon became a bestseller.

However, in January 2006, the website The Smoking Gun published an exposé, which claimed that police reports and other sourcing showed that Frey had fabricated parts of the book, including the severity of his criminal record.

On Jan. 26, 2006, Winfrey brought Frey onto her show, where the author admitted to embellishing parts of the book (notably, the amount of time that he spent in prison). In what would become one of Winfrey’s most controversial interviews, the media personality called out Frey, later saying that she was “defending every single viewer who had bought that book.”

The cover of ‘A Million Little Pieces’ by James Frey.Vintage

“It’s the brutal hypocrisy of it,” Frey told The New York Times of the controversy. “She told more lies to the public times a thousand than I ever have. And I’ll leave it at that.”

The PEOPLE App is now available in the Apple App Store! Download it now for the most binge-worthy celeb content, exclusive video clips, astrology updates and more!

Frey described the scandal as a “tsunami” to The New York Times. On top of negative press and being dropped by his literary agent, the author recalled reporters sneaking into his apartment, and the paparazzi taking photographs of his family. In 2011, Winfrey invited Frey back onto her show to apologize, but the author still holds resentment.

“You might be the most influential lady in this world, you won’t stop me,” he told the Times. “I will lower my head and I will walk forward and I’ll keep throwing punches until I die. You can’t stop me.”

“When Picasso makes a self-portrait, if it’s not photorealist, is it invalid?” Frey asked. “When Rembrandt painted self-portraits, is he allowed to manipulate the paint to make himself look however he wants himself to look?”

James Frey on ‘The Oprah Winfrey Show’ in 2011.OWN

In the years following the scandal, Frey published several novels for adults and young readers. But reflecting on being an author today, Frey told the Times that it’s different than it was decades ago.

“For a long time, writers were fearless sorts of people who held mirrors up to society and showed us what was up. And that’s not the case anymore, right?” Frey said.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

“Writers are scared of getting canceled,” he continued. “Writers are scared of making work that makes people uncomfortable. Everybody wants a hug and a Pulitzer. I don’t. I don’t need either one.”

Despite this, Frey is set to publish a new novel, Next to Heaven, on June 17, and says that he wrote “every sentence” of the thriller.

“I’m a writer, I create books, but I’m also something else. And I don’t know what that is, but I do things with art,” the author said. “I do whatever I want.”