NotebookLM - Early Thoughts

First impressions of Google's NotebookLM+AutoPodcast ( notebooklm.google.com ):?

• The Notebook part: works well. Great info management tool. Feels inevitable.
• The auto-generated podcast part: jaw-dropping, but … addictive, useful, and awful.

I based my initial exploration on 3 different sets of data:

1. A big set of my own messy jottings about strategy, capabilities, etc
2. A maths paper that is way beyond my comprehension
3. A large collection of jokes from a variety of sources

The Notebook part

The basic idea of the NotebookLM is great, and works well. Feels like a good, sensible, useful, and inevitable use of LLMs. You get accustomed to the ask, pause, response approach. Each response can persist as a note in the set.

I can imagine a power user running with this. The initial set of “you could do XYZ” buttons nudges the user towards its strengths in info management and analysis. I favour the taxonomy route, so I would want a hot key for that.?

The UI is a bit confusing. e.g., I keep losing the podcast button. But experimental, so totally fine.?

I was able to interrogate the datasets, and its comprehension was generally good.?

By dataset

1. Messy notes. Added useful structure and overviews to the messy strategy notes. Was the best on this.
2. Hard maths. Provided an ‘in’ to the complicated maths paper. The generated analogies were helpful, and it did not matter that a real mathematician would have laughed (and did) at their wrongness. They helped me.
3. Messy jokes. It needed help with the jokes. The inputs were messy, and included some jokes labelled as coming from the Edinburgh Fringe, but the system seemed to assume the jokes were all from there. Via the chat dialog, I was able to phrase questions that corrected this misunderstanding, but it needed re-correcting. There seemed to be no way to amend high level misunderstandings like this. It was very judgemental about the jokes, combined with a lack of understanding of the nuance. And sometimes simply wrong about interpretations. But it was able to create a useful taxonomy and classify the jokes according to that taxonomy, so not useless. Except, on closer inspection, while the taxonomy seemed ok, the classification of jokes into that taxonomy had numerous mistakes. Hm. Needs more digging.

Questions / Thoughts?

• Will be interesting to see if the generated notes themselves start distorting its view of the original content.
• I wonder how this will scale up to having 100s or 1000s of notes in a set. Note management will presumably become an issue.
• The length of pause waiting for a response to a meaty question is significant. Will that become increasingly frustrating? Perhaps ensuring the GUI allows the user to be getting on with something else in the meanwhile will be sufficient.
• Need to be wary of high-level misunderstandings of the data that distort all subsequent interpretations of it. I would guess the system could benefit from some user-supplied context before it takes in the various sources. Or, assume there would be two steps to working on a new set of data: 1. read in the data, generate an overview, 2. User sense-checks the overview, corrects the context, go back to 1.
• The “create a taxonomy, then classify all the pieces according to the taxonomy” seems like a generally useful step.
• I may have missed it in the UI, but I couldn’t find the fallback option of a simple keyword search.

Overall, big thumbs up. Some questions about the reliability of classifying large datasets using a taxonomy, and correcting high level misunderstandings. NotebookLM will keep getting better, faster, more useful.

The auto-generated podcast part

Jaw dropping. After a few minutes of the spinny thing, it was expressing “concentrated essence of smug podcasters”. If you overheard little snippets, there is no way to tell this is not real. And when you listen more closely, they are talking about your info. Jaw drops again.

But when you listen more closely still, there are cracks. Again, experimental, so totally fine. But I noticed the following:

• The speaking heads seem to alternate being The One Who Understands The Info, which can feel odd, and increasingly fake/glitchy.
• The relentless positivity and smugness is nauseating - face-punchingly annoying in fact. Feels like a satire. Cannot listen to this for long. I guess being a non-American that may be more on me than them, but FFS, tone it down a bit.

By dataset

I had very different experiences with the podcasts for each of the 3 datasets:

1. Messy notes: This was scary. Properly worrying. Confirmation bias on steroids. My own notes being discussed and agreed with enthusiastically, with nuances and phrasing that did not come directly from me, but oh how very much I agreed with. I was “yes, yes, at last, I’m right, ha, see, they love it”. I could listen to people agreeing with me all day. I felt like I was on a slippery slope into some psychological issues. Thoroughly and instantly addictive. Specific benefits included hearing my own points rephrased into a better, more relatable form, so I’ll take those. But, yes. This needs care, psychologically-speaking.?
2. Hard maths: This was fascinating and useful. Whilst being wrong about some of the nuances in the maths paper (so a disgruntled expert informs me), the overall approach did help me wrap my head around the paper. I can now begin to ask less stupid questions. It means subsequent explanations have a better chance to stick. The podcast pacing helped with this. So, a net positive.?
3. Messy jokes: Oh dear. Fell apart on this one. The podcast personas came across as pompous, judgemental, wrong-headed ignoramuses. They just did not get it, but were going to smug their way through 10mins of talking anyway. Their opinionated lack of understanding was painful. When a chat response indicates a misunderstanding, it is fine. You can restate the question, perhaps re-structure the data, change the context, etc, and ask again. But the podcast keeps on trucking. Felt like a brutal satire of polished podcasting ignorance. Speaking for the sake of speaking. Funny though.

Questions / Thoughts

• I wonder if it will become easy (and annoying) to spot re-use of mannerisms in the podcast speech. TBF, it is anyway with normal human podcasters...
• The option of non-American accents and mannerisms would be nice. Might help me be less judgemental ;-)
• I can imagine a dashboard of multiple sliders to configure the podcast personas and their respective ‘understanding’ of the data.?
• I wonder what would happen if one of the personas was explicitly contrarian. Or a positive persona who does not fully understand the data debating with a contrarian who really does.
• Make it easy (perhaps the default) to have a sceptical persona.
• Make it easy to configure the podcast with a slider from objective to subjective.
• Being able to ‘tune’ the podcast in these subtle but profound ways would be fascinating.
• When the basic NotebookLM gets an answer ‘wrong’, it does not matter so much. Just rephrase the question and try again. But the podcast being ‘wrong’ was a horror show. It takes the 'wrong' to a whole new level.
• The addictive aspect was properly worrying.

Overall, wow. But…