Exploring NotebookLM: Can AI Decode the Depths of a Novel?

I learned about NotebookLM about a weeks ago when listening to The Hark Fork, a New York Times podcast hosted by Kevin Roose and Casey Newton. They invited Steven Johnson, one on the NotebookLM team to talk about the application together. I was absolutely amused by the podcast generated from one of the host's monthly credit card bill by NotebookLM, where the card owner's habit of taking Uber rather than a more environmental friendly transportation was teased.

While it was called out that NotebookLM is still expanding to support more languages and additional features, when I get to play with it, I decided to assume no limits and throw something unexpected to it: a novel I wrote 10 years ago that contains about 800K Chinese characters (roughly 1000 pages long)

TL;DR:

• The length of the input resource does not seem to be any problem. Later I created another notebook with PDF, Youtube video, etc., and it seems multi-modal are also well-handled.
• The overview and introductory podcast generated out of my novel is really stunning at first glance. However, the big model behind the scene is still not fully capable to peer off all the layers of complexities in the non-English text. Literature majors are temporarily "safe", I guess.??

• The RAG / reference feature seems unbalanced: a lot in the earlier sections of the text are referenced than those appear later in the text. Also, the text are taken with their superficial meanings, not analyzed with depth as it supposed to for a novel.
• The big model is really struggling to decode suspense in the plot, especially ones that are very distant apart in the novel. It seems the model is not yet capable to establish and maintain a "time series world view" for the novel.

Stunning Introductions

Here is the summary generated without any prompt by NotebookLM. As the author of the novel, I would say while it is definitely not comprehensive and not cover the best (as thought by the author) of the novel, it does manage to get all the juicy, marketing-facing stuff in a short paragraph. I mean, from 800K Chinese characters to something below, I cannot ask more.

This is a story told from the perspective of a woman named Cao Jie who is a modern-day graduate student who accidentally travels back in time to the Qing Dynasty during the reign of the Kangxi Emperor. After being mistaken for a deceased daughter of a powerful family, she is forced to enter the palace and serve as a court lady. Cao Jie navigates the intricacies of palace life, witnessing the power struggles and personal dramas that play out within the imperial court. She is caught in a love triangle with the eighth and fourteenth princes while trying to survive the treacherous world of the royal court. The story also explores themes of identity, loyalty, and the complexities of human relationships in a world ruled by power and politics.

The introductory podcast is more amused and intriguing. Also, it does a better job to cover the main characters of the story. It is a bit long and definitely not high pitches all the time, but it does make me smile at a few places.

Missing Subtleties

The major missing subtlety is that neither the text nor the podcast summary get it right in terms of the romance relationship among the main characters. This is partly due to the fact that the story is a series of complicated plots that involve many characters and since it is written in first-person perspective, some of the suspense and people's motivations were revealed in very distant part of the novels. This is also partly due to the fact that the language used in the novel are most of the time conveying super nuance details of character's emotions, motivations, and thoughts, which make it really hard to peel off the onion. A thousand people see a thousand Hamlet. When it is all straightforward, there is no beauty left.

Meanwhile, it is also noticeable that the big model behind the scene seems not fully capable to comprehend the novel in its entirety. I tried one of the NotebookLM generated chat question as shown below.

It is a good question and I have to admit the bullet points summaries are roughly on track. However, when looking into the references and quotes, they are really superficial. The model is taking chunk of words literally rather than really comprehending and analyzing the character's behavior and change of thoughts over time, which is one of the most important aspect in answering "how ... relate to the protagonist's journey" question in my view.

The World View

It seems to me that the big model behind the scene is good at synthesizing information but not establishing and maintaining a world view, where it involves understanding a lot of autonomous agents, i.e. characters in the novel, their changing minds, the cause of their changing minds, their characteristics and beliefs, the limited and constantly changing information they have, the conundrum they face, and the consequence of the actions they take for other characters in a kind of chain / graph effect. Therefore, when it comes to fundamental literature analysis questions involving what, why, how, the model becomes struggling. It is currently a STEM college student doing literature analysis, not a literature major college student. ??

Find the Right Use

Before NotebookLM becomes more sophisticated in novel analysis, I still find it quite useful in many other situations. In fact, I also set up another notebook in which I threw a bunch of materials of a company (Youtube of CEO keynote speech, investor report, company website, etc.) and it did a fantastic job to extract main study points. I am considering to experiment letting it digest 10K filings as well. If anything fun, I will definitely share. Stay tuned.